Apple Vision Pro & Augmented Reality

Now in 2024, we are on the cusp of another revolution with the launch of the Vision Pro, Apple’s first augmented reality glasses. Much like the iPhone provided the core functionality allowing startups to connect people and digitize the world around us, the Vision Pro offers the potential to overlay digital information onto our analog surroundings and integrate it directly into our daily lives in a more personal way than ever before.

If you’d like to learn more, watch our webinar. 

REPORT AUTHORS

We believe this wave of “spatial computing” is different from the augmented reality/virtual reality buzz of the past. This is due to the Vision Pro’s technological breakthroughs and seamless user experience – Apple’s demo and guided tour demonstrate this in action. The Vision Pro’s “passthrough cameras” see the world for you and display them within your headset real-time. You can also easily control how immersive your display is within your field of view – this helps you stay connected with the world around you even while you are wearing the device. Additionally, the user experience is executed seamlessly with user-friendly eye tracking so that the device knows what you are looking at and can select it for you, as well as the device’s ability to read your hand gestures even when your hands are comfortably situated on your lap. We are excited by the fact that the technology is only going to get better. The breakthrough in usability coupled with the power of Apple’s robust app ecosystem will usher in a whole new set of applications across many industries.

While the first iterations of Apple’s glasses may focus more on entertainment and gaming, we expect that later versions will enable more immersive augmented reality experiences in both consumer and business use cases, across education, remote work, manufacturing, healthcare, real estate and more.

As the hardware improves and the price point for the product decreases, we believe there will be a gamut of new visionary startups digitizing the physical world much like Uber and Airbnb did via the iPhone. Tim Cook has described the Vision Pro ‘Tomorrow’s technology today’, and we explore what that tomorrow could look like across different verticals in more detail.

Gaming

Imagine if you could battle mythical creatures in your backyard with friends. The Vision Pro would blend digital worlds into our physical reality, making games come alive all around us. You could turn your neighborhood park into a virtual battlefield, dodging behind trees and benches as you outmaneuver trolls and orcs. A child’s bedroom could be transformed into Hogwarts castle. Structures around you could become part of the video game landscape. Gaming is no longer contained to screens – it will be tactile, social and integrated into your surroundings – a new paradigm.

The potential of augmented reality in gaming was hinted at earlier with the phenomenon of Pokemon Go in 2016, which surpassed $5 billion in lifetime revenue and attracted over 250 million monthly users at its peak. We expect the Vision Pro to represent a significant evolution in the gaming landscape, akin to the transformative impact of console and mobile gaming. The innovation could also pave the way for further collaboration with other sectors such as education or brand marketing, broadening the scope of gamified experience.

Entertainment

We live at an extraordinary time witnessing two groundbreaking technologies collide – augmented reality and artificial intelligence. OpenAI has recently introduced Sora, a groundbreaking platform where everyone has the opportunity to become a creator. Envision a world where individuals can craft their own narratives and share them with both friends and a wider audience. Vision Pro offers not just consumption of traditional content like Netflix and YouTube, but also unlocks new media experiences tailored to 3D and more interactive formats.

Imagine attending a concert and having access to real-time lyrics. Your friends could appear as avatars visible only to you, dancing alongside you in the crowd. From there, you could message each other reactions and commentary on each song. Exclusive backstage content and real time interaction with celebrities and fans would further immerse you in the event. Or imagine watching a basketball game as statistics show up real-time on the screen along with game analytics and play predictions overlaid with an immersive in-game betting experience. You could even virtually invite friends from around the world to share the experience with you as if they were there in person. Your Vision Pro glasses would connect you and your virtual guests to a private chatroom during the show. By blending physical and digital environments, Vision Pro allows you to share experiences with remote friends as if you were together, no matter where you are in the world.

Education

Imagine medical students performing virtual surgery, gaining first-hand experience by operating on detailed 3D digital models. Or history coming alive for students as they walk through interactive recreations of historical sites and events. Physics would no longer be abstract and could easily be visualized – at the atom or molecule level, as well as planetary scale – all through spatial computing. The Vision Pro makes experiential learning more hands-on, immersive, and collaborative than ever before. Driving and flight simulators would be more realistic and effective than ever before. Language learners could access instant translations of menus, signs, and overheard conversations while immersed in foreign locales.

Additionally, AR and VR can bridge geographical gaps in education, enabling students from different parts of the world to participate in the same virtual classrooms, thus fostering global collaboration and diverse perspectives. By overlaying digital enhancements onto tangible environments, the Vision Pro could revolutionize education across every field.

Remote Work

The pandemic created a new paradigm in the remote work revolution, but many companies have reverted to their traditional in-person work routines. With augmented reality, remote work could see a resurgence. Imagine turning your kitchen table into a virtual conference room, where you are seated next to your remote colleagues from around the world. The Vision Pro bridges geographical divides, bringing distributed teams together through shared immersive workspaces in a way that a 2D Zoom window does not. Detailed 3D renderings of prototypes stand in the center, which you can pick up, examine from all angles, and tweak through intuitive hand gestures. Documents and web pages open and hang in midair for discussion, visuals coming to life all around you. Whiteboards cover every wall to capture brainstorms. The Vision Pro erases physical boundaries, fostering productivity and connection regardless of location. Furthermore, any environment can turn into a virtual, collaborative workspace. For digital nomads working abroad or while traveling, AR provides ultimate flexibility. A coffee meeting can transform into a collaborative, private workspace, only for the eyes of those at the table. As a result, we expect commercial real estate prices will experience another drop.

As more businesses embrace work-from-anywhere policies, AR promises that “anywhere” really can mean anywhere without teams sacrificing productivity or a sense of unity.

Manufacturing

Designing and visualizing physical products will no longer be the same. Imagine engineers tweaking 3D models of cars, spacecrafts or homes simply with their hands and seeing the changes reflected in real-time in their headsets. Or maintenance crews accessing remote experts who can visually assess equipment issues and walk them through repairs. Meanwhile, a technician troubleshooting an issue on the assembly line can connect with a remote expert via Vision Pro. As they describe the problem out loud, custom markers and annotations appear overlaid on the malfunctioning equipment to guide the repair. The expert sees everything the technician sees and can draw digital arrows and instructions in real-time. Logistics supervisors can map item flows, track deliveries, inventory levels, and supply bottlenecks all projected into the warehouse itself.

AI and computer vision algorithms along with spatial computing will enable real-time diagnosis of equipment failures. Maintenance crews will be able to access experts on demand and show them their surroundings in real-time and hands-free.

In essence, data will flow out of isolated monitors and into the real world – the future of how we make things is inexorably connected to augmented interfaces.

Healthcare

The healthcare industry is ripe for augmentation through AR apps on the Vision Pro. Imagine your doctors securely accessing your patient data and scans overlayed directly in their field of view during a routine checkup, or real-time during surgery without taking their eyes off the operation. Surgeons can monitor heart rate and oxygen levels during complex procedures without glancing away from the incision site, eliminating critical seconds of delay. Surgical residents can also access projections of best practice techniques specific to each operation they undertake. On the mental health front, therapists can immerse patients in tranquil natural environments, blanketed with calming ambient sounds only they can hear via Vision Pro. They can also confront patients with photographic projections of phobia triggers, slowly adjusting exposure levels and monitoring biofeedback indicators through the glasses. By overlaying digital elements onto real situations, Vision Pro enables better informed, more personalized patient care.

Real Estate

Augmented reality and spatial computing hold promise for the real estate industry across home builders, decorators, brokers as well as buyers. Imagine buyers walking through homes before they’re even built, customizing layouts and finishes with the swipe of a finger. Architects and designers can showcase interactive scale models, letting buyers adjust floor plans specific to their lifestyle needs. Contractors and clients can visualize and experiment with design choices like countertops and tile patterns mapped onto their actual spaces, eliminating guesswork. For existing homes, interactive AR remodeling apps can overlay proposed kitchen or bathroom upgrades mapped directly into the home’s current layout and finishes—allowing buyers and sellers to literally see major renovation potential before investing. Urban planners can project the impact of proposed developments, allowing citizens to toggle views and provide input. The Vision Pro allows us to envision future spaces more dynamically and collaboratively.

Conclusions

We believe the Vision Pro is a groundbreaking innovation poised to transform a multitude of industries, including gaming, film, live events, education, manufacturing, healthcare, and real estate, through the lens of augmented reality. It holds promise to birth the next generation of computing platforms, potentially fostering the creation of numerous unicorn startups that will redefine the contours of innovation and progress.

The path to unlocking the full potential of spatial computing and the Vision Pro is complex and may face some challenges, including privacy concerns, health and safety considerations, and social isolation. History has shown us that with innovation comes the responsibility to address and overcome such challenges. However, we believe that these risks will be systematically mitigated as the Vision Pro continues to evolve and gain more traction and as the technology inevitably gets better.

The promise of augmented reality extends beyond mere technological advancement; it heralds a paradigm shift in enhancing human experiences. With conscientious development and a forward-thinking approach, AR devices have the potential to usher in a new era of interaction, where the digital and physical realms converge in harmony. This integration will not only elevate our day-to-day lives but also unlock untapped avenues for creativity, learning, and connectivity.

As we look to the future with anticipation, we are energized by the prospects that Vision Pro introduces, and eager to contribute to the unfolding story of how spatial computing will shape the future of human experiences.