Envisioning the Future of Digital Media with the Apple Vision Pro
by Megan Botti
Apple’s upcoming Vision Pro headset is being compared to the iPhone and the iPad in terms of its potential to revolutionize human interaction with computers, but it seems like users and even the company itself aren’t sure what possibilities their new tech will enable.
When Apple unveiled the headset at its developer conference earlier this year, it didn’t pay much lip service to augmented and virtual reality, the technology buzzwords most highly associated with wearing a screen on your face. Instead, Apple is referring to the Vision Pro in broader terms, as the world’s first ‘spatial computing’ device, and touting its productivity benefits for work.
If the Vision Pro can reach the same or similar market penetration as the iPhone, there’s no doubt that it will change our relationship with screens, and enable efficiencies by revolutionizing the form and function of the digital workspace. The Vision Pro also will likely change the way we experience meetings and presentations.
The Vision Pro’s potential impacts on arts and entertainment are less clear. Consensus belief among Apple observers is that the company unveiled the product prototype two or three quarters ahead of its actual release so that developers could get a head start on imagining and building new use cases for the headset.
We explore some of the ways we may be seeing the headset put to use when it goes on sale in early 2024. While some art forms, such as interactive storytelling, may simply become a more popular entertainment product, hopes are high for Apple and its supporters that the Vision Pro will unlock innovative new forms of digital media.
Immersive Media and TV Experiences
One of the Vision Pro’s more likely impacts will be driving the growth of interactive storytelling and branching narratives, like the Bandersnatch movie, released by Netflix in 2018. The choose-your-own-adventure film gives viewers the power to direct the narrative through multiple possible paths by making decisions at key junctures in the plot.
The movie was generally well-received, earning a 74% on Rotten Tomatoes, despite the challenges that writers faced in managing multiple interwoven narrative arcs. Netflix even had to develop brand new software to help the writers keep track of the narratives. Producers originally intended the film to be an interactive episode of Black Mirror, but were forced to pivot due to the demands of production, which one show creator said was equivalent to making four episodes of Black Mirror.
These challenges mean that the risk-reward for producing interactive entertainment might not make sense when the audience is overwhelmingly consuming that content on a 2D screen and interacting by clicking buttons. Unlike a laptop screen or TV, the Vision Pro offers true immersion. Instead of breaking up the flow of the narrative to make the viewer click on a decision, the Vision Pro’s outward-facing cameras, gesture recognition, and voice recognition capabilities would allow viewers to remain fully immersed and make their decision by hand gesture or voice. Combined with A.I.-enabled natural language processing, one could imagine watching a film where the characters, faced with a decision, turn to the viewer to ask what they should do, and the viewer is able to provide their input without breaking the illusion of being ‘in’ the film.
Some studios are already working with A.I. to produce interactive storylines. PBS is working on a pilot of a children’s show where the characters actually talk back. Bandersnatch showed, however, that existing technological infrastructure isn’t optimized well enough yet to make big bets on interactive storylines for adult viewers that may or may not perform well enough to justify the upfront production costs.
Additionally, interactive stories have to get over the obstacle of being perceived as gimmicky. Interactive and fully immersive content on the Apple Vision Pro, combined with Apple’s brand loyalty, may be the shifts needed to change that perception. By the time (and if) the Vision Pro makes immersive viewing more mainstream, progress on interactive storytelling for children may lead to better production technology that makes the creation of such content more feasible on a larger scale.
Revolutionized Gaming Experiences
While AR and VR gaming has been around for years, popularized by Pokemon Go and Beat Saber, the Vision Pro’s hardware features will enable a different kind of gaming experience, focusing on XR, or extended reality which blends AR, VR, and whatever immersive experience is best suited for the application.
The Vision Pro users can adjust the immersion level of their experience by twisting a digital crown on the side of the display casing. Turning the crown all the way down allows the wearer to see and interact with their physical surroundings through cameras that simultaneously display the outside world to the wearer and display the wearer’s eyes to others in real time. Turning the crown all the way up fully immerses the wearer in virtual reality. The ability to easily transition between reality and virtual reality is a critical component of the Vision Pro that is not possible with current VR headsets or AR applications.
Extended reality will allow developers to make games that integrate the player’s physical environment, overlaying digital elements onto the real world. This creates a more engaging and realistic experience, especially in genres like adventure, mystery, or horror.
There are games on the market in this category already. A popular example is the piano learning platform, PianoVision, on the Meta Quest 3. However, that program teaches the player on a virtual piano. The Apple Vision Pro would allow a player to sit in front of a real piano, and adjust the digital crown to be able to see both the piano and the virtual overlay.
With the Vision Pro, any backyard could become a jungle maze, any city could become a quest map, and almost any experience could be gamified, from gardening to grocery shopping.
The Vision Pro will also enable the reverse: social games where players share a virtual space that feels as real as physical interactions. This could foster a deeper sense of community and player engagement. These games already exist on platforms like the Meta Quest, but users have to interact through characters rather than as themselves. According to Apple, the Vision Pro will allow wearers to scan their faces to create a digital ‘persona’ (Apple’s terminology) that they can then use to participate in virtual spaces. Of course, there’s no obligation in virtual reality to appear exactly as you appear in real life, however, the appearance of real faces in virtual reality will increase the immersiveness of the experience.
Games may become more social in general, as the vast clusters of gamers distributed across discord servers, subreddits, clans, and guilds can establish a presence in virtual reality, like a meeting hall complete with a sign outside.
The Apple Vision Pro could represent a big step toward blending the best of physical and virtual reality experiences.
Digital Galleries and Performances
The Apple Vision Pro will have a significant impact on expanding access and exhibition possibilities within the world of fine art.
Millennial readers may remember the Eyewitness nature education series from seeing it in elementary school. The show’s intro sequence is a camera panning through a computer-generated museum-like scene where animals are swimming and flying out of picture frames, and turning a corner leads into a jungle exhibit or volcanic terrain. Twenty years after the last episode of Eyewitness aired, we’re pretty close to being able to walk through such a museum ourselves in virtual reality.
If the Vision Pro gains widespread adoption, nobody will ever have to travel to see an art gallery or a live performance. The Apple Vision Pro could allow for a live audience in virtual reality, filled by personas of viewers from anywhere in the world. Importantly, the performer would be able to see the audience and hear their reactions instead of just performing in front of a camera.
Creativity Squared listeners will remember Episode 26 with Sergiu Ardelean, whose company, Artivive, empowers artists to bring their works to life through augmented reality that patrons can experience through their phones. The Vision Pro will not only eliminate the need for the phone, but also the need to even be standing in front of the actual work in order to activate the AR component. Works of art can be displayed in a virtual gallery, patronized by anyone with the Vision Pro. In the virtual gallery, viewers can not only see the art but experience the art. Viewers could step through the frame into an immersive artwork or interact to alter the art without the worry of damaging it permanently.
The Apple Vision Pro could also unleash a new level of artistic collaboration never seen before. For example, a virtual art space could include an installation where multiple users contribute to a piece of art in real time, creating collaborative works that evolve with each interaction. Think about an artwork like Mesoamérica Resiste by the Beehive Design Collective, an absolutely massive drawing filled top-to-bottom with deeply granular detail of 400 years of Central American history. Using mixed reality and the natural input methods available on the Apple Vision Pro, Beehive collaborators from all around the world could all work on the project simultaneously and share feedback in a central location in real time. Schools struggling with funding for art programs could consolidate classes into an online platform attended by countless students from different districts. The possibilities for new ways to experience and create art are practically endless.
We Don’t Know What We Don’t Know
Although AR and VR have been around for some time now, a quality product can quickly change how quickly technology catches on and what that tech is used for. Online dating was widely considered a last resort for older people before Tinder came along and almost single-handedly changed modern dating. Similarly, the Apple Vision Pro has the potential to kick-start wider engagement with the metaverse and redefine our relationship with screens.
Once the masses start playing around with it, there’s no telling what new possibilities will emerge from the collective creative consciousness. To be sure, creatives will be leading the way in building new entertainment and cultural experiences. We’re excited to see what 2024 will bring.