Spatial Video experience on iPhone 15 Pro — beautiful and realistic, but requires Vision Pro to play it – CNET Japan

I’m currently watching a plate of sushi floating in front of me in 3D. The chef finished by placing the toppings on the junkanmaki and yellowfin tuna, then moved his hands as he spoke to me. It looks clear and realistic. Surprisingly, I shot this video with the iPhone 15 Pro a short while ago. And this is the virtual reality (VR) experience I’m watching in beautiful 3D using Apple’s Vision Pro headset.

Swipe your finger to watch other great videos from Apple. Different families relax inside their homes, walk in the meadow, and huddle together. Everything is 3D and looks just like the real thing. It’s like I’m getting a glimpse into that family’s life, and it feels strange but familiar. However, the clarity cannot be denied.

I’m currently trying out Apple’s new “spatial video” feature which uses the iPhone camera. This feature is currently available in the new public beta of iOS 17.2. The official release of iOS 17.2 is expected in 2023. iOS 17.2 lets you record 3D video, but requires Apple’s high-end iPhone 15 Pro, which costs $999 (starting at 159,800 yen in Japan). Additionally, to watch 3D video, you’ll need the $3,499 Vision Pro headset, which will be released in 2024.

The videos are great and the 3D is immersive and realistic. It’s also easy to record and can be saved as a video for 2D playback using standard video formats. However, unfortunately, this feature is intended for the Vision Pro, which will be released in 2024, so it is not something that many people can afford right away. However, the experience is impressive.

Reaffirming the wonders of Vision Pro

I used the Vision Pro for the first time at Apple’s annual Worldwide Developers Conference (WWDC) in June, and was once again reminded of the display’s exceptional performance. Photos and videos look great with this headset. It’s no exaggeration to say that the Vision Pro is better than any other monitor I currently have at home. Of course, if you own a Vision Pro, you’ll want to watch videos and photos, whether they’re 3D or not.

When I tried the Vision Pro again with prescription lenses that suited my needs, I realized that the headset was much smaller than I expected. I was again impressed by the ease of use of the interface.

Setting up eye tracking is easy, just look at the dots surrounding the circle and tap on them. I also tried out the new zoom gesture, which lets you pinch and extend your fingers to zoom in on an image. The gesture was linked to eye tracking, so the image was zoomed in no matter where you looked. It’s like telepathy. I looked at some sample photos in the Photos app and played a slideshow of photos in Memory. I recently tried out Meta Quest 3, and the Apple camera and display resolution were on a whole other level.

The panoramic photos were amazing. It felt like a window had spread around me, enveloping me and allowing me to see other places out there. To be honest, it’s almost 3D. Spatial video is great too. The playback quality is almost fantastic, beyond realistic, and seems to be aimed at immersive memory rather than regular video playback. However, even during this short trial run, I felt there were some limitations.

Unfortunately, I cannot publish here what I saw or recorded. All images are courtesy of Apple, so you can just read my experience.

Spatial video on iPhone 15 Pro

To shoot spatial videos, you have to hold the device horizontally. This is because by setting the iPhone 15 Pro to landscape mode, it uses the main camera and the ultra-wide-angle camera side by side to create 3D videos. It shoots two 1080p, 30fps videos at the same time, and uses computational photography technology to address differences in lens and distance. The files are saved in HEVC format like all Apple video files, and the file size is around 130MB per minute, which isn’t bad. This is probably because the file size is relatively low, with a resolution of 1080p and a frame rate of 30fps.

Provided by: Apple

Spatial videos can be shared via Messages or AirDrop, but other apps may compress files and cause 3D data loss. In this case, the video remains available everywhere like a regular video file, which is good news for those who wondered whether they should choose a “special” 3D video format (I was one of them).

Spatial video mode must be turned on in the Camera app settings on your iPhone, or via the Vision Pro mask-shaped toggle in the Camera app itself. This limits the video resolution and frame rate, and forces you to shoot in landscape mode. Unfortunately, 3D videos cannot be shot in portrait orientation.

The camera app will remind you to hold the camera sideways and maintain a certain distance from the subject. When I was shooting, I was also required to maintain a distance of 3 to 8 feet from the subject to get good spatial video. However, when I took a test shot of someone holding sushi on a table, I had no problem getting closer. I’ve taken photos indoors with plenty of light, but when I set it to spatial video shooting mode, I can’t seem to adjust the brightness and contrast. This means that when shooting in poor lighting, photos may be grainier than normal videos.

Provided by: Apple

Spatial video processing and editing

There is no API available for Apple’s 3D HEVC video format. This means that files in this format are not yet designed to be recognized by third-party applications. However, it is possible that app developers will come up with a solution. It’s also disappointing that there’s no way to watch videos in 3D on anything other than the Vision Pro.

Speaking of recording stereoscopic 3D images, it seems to be theoretically compatible with other VR headsets, such as the $499.99 (74,800 yen in Japan) Meta Quest 3. This is because it is simple 3D video, and no LiDAR or depth sensors are used anywhere in spatial capture to capture all the action. Unfortunately, Apple currently has no plans to do so.

There is no maximum length of time that a video can be shot, so it is theoretically possible to shoot a 3D feature film using spatial video. However, on both the iPhone and Vision Pro, the only editing feature available for clips is cropping. Spatial video editing will be supported in Apple’s Final Cut Pro video editing software, but we’ll have to wait until sometime in 2024. In the meantime, if you try to edit a 3D video file using a Mac or iOS video editor, it will be converted to a binary video file Dimensions.

It’s also a bit disappointing that you can’t shoot 4K videos. The image quality and resolution of the Vision Pro headset is impressive. When I looked at an iPhone photo with Vision Pro and zoomed in, I realized that again, and when I looked at a panorama in full-circle mode, it was a vivid, immersive reproduction of the scene, just like 360 ​​degrees — a photograph to the point where I felt like I was there. Spatial video is really nice, but I’d like to see it at 60fps and at a smoother 4K resolution. Maybe it will come true one day.

“Memory” for the 3D future

It seems that Apple is trying to become a company that values ​​memories as well as lifestyle. Automated photo collections are actually called “memories.” This is exactly what it feels like to watch 3D spatial videos with Vision Pro. The edges are blurred, like the futuristic digital memories depicted in the sci-fi films Minority Report and Blade Runner.

Apple’s choice to surround spatial video with a blurred frame makes the video look like a 3D projection of the room, with the edges blending slightly into the air. But I also like to watch videos surrounded by normal frames.

Apple’s spatial video formats appear as a new category in the Photos app and don’t appear in Memory yet. Perhaps the new Spatial Video tab in the Photos app is the place to store new 3D experiences. Currently, the iPhone 15 Pro can only shoot videos and cannot take 3D spatial photos, but the Vision Pro itself can take photos. But Apple clearly wants everyone to start recording in spatial video formats and build a library of great videos before launching Vision Pro.

This means that you need to remember to turn on spatial video and use it consciously. Future Vision Pro owners will likely have this feature turned on by default on their iPhone 15 Pro, but only for a small percentage of iPhone users. I plan to play it. I’m interested to see what this video will look like on the Vision Pro in 2024. But unless you’re planning to buy the Vision Pro, there’s little reason to shoot spatial video now. Who knows, maybe one day in the distant future you’ll regret not filming your birthday party in 3D. I recently shot a video for my niece, and I’m already wondering what it would be like to watch it in 3D. The record button on this new spatial camcorder really attracts the “FOMO” (fear of being left behind) that I will feel in the future.