Augmented & Virtual Reality Trends: Conversations from OrlandoiX 2017

Virtual and augmented reality technology has made some exciting leaps over the past year as broader consumer adoption and new tools have inspired companies to invest time and resources in creating better virtual experiences.

This was the focus of the activity at this year’s OrlandoiX 4-day digital festival, held at Full Sail University in Winter Park. Conversations centered around the hardware, software, and business of VR, AR, and video games. The event featured presenters and panelists who are working with these rapidly-evolving technologies, highlighting evolving best practices and the most important of the many recent advancements in the field.

The festival took place in two parts, with a two-day conference on Thursday and Friday and a tech and gaming expo on Saturday and Sunday. Additional activities included a kick-off party hosted by the Orlando VR/AR Meetup and indie game association Indienomicon, as well as nightly tournaments in games like Super Smash Bros, Hearthstone, and FIFA.

This year the annual Thursday and Friday conference featured a lineup of heavyweights in the technology industry, including keynote speakers from Microsoft and HTC Vive. Topics included the development of social VR, live streaming of 360 video, business development and investment in VR, and the evolution of augmented reality.

I had the chance to moderate several panels this year, as I have in previous years, and over the course of the festival I found that, while what was happening on stage was fascinating, it was complemented by the conversations happening offstage, in the green room, or in the crowds. These conversations are representative of conversations happening nationally and internationally around the industry, and I wanted to highlight a few of the most interesting topics I heard.

Apple’s AR Kit

Before Apple’s developer conference (WWDC) in June, everyone was wondering whether or not Apple would have any big announcements relating to virtual and augmented reality. Well, they sure did, and it was a hot topic of conversation at iX. Apple’s ARKit, a development tool that supports AR for Apple’s devices, has advanced the capabilities of mobile AR with improvements in areas like depth sensing, 3D space analysis, lighting, and accuracy of movement. These are all areas that are important to the evolution of AR development.

It’s also interesting from a business perspective. Over the last few years, Google and Samsung have led the way in mobile VR. Toward the end of last year, Google launched the Daydream platform, expanding on the proven succcess of Google Cardboard. Samsung has recently been developing hardware and software for mobile VR with improvements to its Gear VR headset and VR platform.

Apple has been very secretive about its plans in the space, revealing little info until recently, when the company skipped right over VR and went straight to a technology that is even further out in terms of its full potential- augmented reality. Apple’s patent filings and job postings have revealed the company’s interest in VR/AR, but the recent move to provide developers with tools to create quality AR experiences has many people very excited.

The potential of AR is widely acknowledged, but only Pokemon Go has so far managed to reach a broad audience, and that was largely attributable to its proprietary Maps technology, the data set it collected during Niantic Labs’strial AR experience called Ingress, and the name recognition and player base of the Pokemon brand. There’s also the fact that much of the gameplay appeal was due to its incorporation of location-based technology and not necessarily its AR components. Many people were turning off the AR to save battery life once they had experienced the feature. So there is a big question hanging over the technology- what kind of experience will take off next, and what will it look like? The easier creating the software is, the sooner we’ll find out.

AR Kit is already being used by developers to create some next-level AR. For a peek into the future, check out the work being done by Peter Jackson’s AR company, Wingnut AR.

The evolution of design principles for VR and AR

As a researcher in user experience and storytelling for AR and VR experiences, this is a subject that’s close to my heart. My favorite part of this year’s conference was my fireside chat with AR/VR designer and evangelist/futurist Cathy Hackl on this topic.

With the release last year of a range of accessible, lower-cost consumer VR headsets, including the Oculus Rift and HTC Vive, we began to see an explosion of fun, interesting, scary, odd, and beautiful VR experiences. This first wave of experiences developed for a broad audience created a foundation for a new form of art, and as designers and developers become more comfortable with the technology and the initial challenges of VR software development, they’re able to focus more on the art of the medium, to start pushing boundaries and identifying new ways to apply evolving design principles.

Artists can now reference previous experiences in their designs, pull from the best of them, and add ideas they’ve had bouncing around in their heads. At the conference this came out in many conversations. One point made by a speaker from Microsoft related to the inherent trust between a designer and a user of the tech. The deeply immersive nature of the technology allows for a deeper level of engagement and emotional connection. VR has been proven to increase memory retention and empathy. But it can also enhance fear and emotional distress. Designers have a responsibility to use that connection with care.

Use it they should, though. Some of the best experiences I’ve had in VR have made use of this emotional connection to draw me out of my comfort zone. There have been plenty of haunted house experiences, which will only improve. There’s a lot of potential to extend this type of design to create virtual escape rooms. One of the best- and strangest- experiences I’ve had was VR Accounting, produced by the some of the creators of the popular show Rick and Morty. I did terrible, terrible things. And it was great. This emotional connection can also be helpful in the educational and therapy realms. Researchers are using VR to help people overcome their greatest fears, deal with trauma and PTSD, and treat chronic anxiety and depression.

Over the course of iX there were also many conversations about the evolution of design for gaming. One of the most interesting elements of Pokemon Go was its combination of gameplay elements- including the discovery/ gathering mechanic, location-based tracking of game objects, and battle mechanic (which left a lot of room for improvement). There are plenty of other game types left to be explored. With the increasing activity in mobile AR, we may soon see new game types being developed- augmented reality MMORPG, anyone?- and new creative experiences of all sorts.

The economic future of VR is mobile

High-quality console and computer games get a lot of attention, so not many people realize that the mobile game market generates the most revenue- by a significant margin. This year smartphone and tablet games represent 42% of the game market, and the market for mobile is growing 19% year over year.

High hopes for computer-based “high-end” VR did not translate to sales this past year. Recent numbers show Oculus Rift right around 383k units sold and the HTC Vive around 667k. These numbers are well below projections and the high hopes of the industry. However, console VR has shown greater promise. Playstation VR launched in October of 2016 and has already seen nearly 2 million units sold.

The challenge for these platforms is obvious- cost (originally around $1,000 with computer upgrades), isolation of the user, being tethered to one location- usually the home.

However, many people believe that VR will see greater commercial and consumer adoption as mobile technology improves. There have been some great mobile experiences to date, but they aren’t nearly as impressive as the high-end experiences and they don’t demonstrate the full potential of the medium. The next round of smartphones will have better depth sensing capabilities, will be less likely to overheat, and will have controllers that can be used for VR experiences.

Mixed reality headsets

Virtual and augmented reality technologies are similar in many ways, but they’re currently separate devices used for different types of experiences. However, this may be- at least in part- because augmented reality is still new and in development.

Some companies, including Microsoft, are exploring what a combination of the two technologies looks like from both a tech and a user experience standpoint. In its exploration of a single device for delivery of both AR and VR, the company’s latest experiments have reflected a creative approach to how users of both technologies might interact, not just in the two different types of environments, but also in a single joint environment.

The mixed reality hardware appears to vary little from current virtual reality technology, but this imagining of a user experience that accommodates both areas of the technology, so similar in design and production, is interesting. Microsoft’s approach is also interesting from a communications perspective. The terms we use to describe these technologies may change and evolve, and “mixed reality” is currently often used as an umbrella term.

Another component of the conversation is the development of mixed reality platforms for mobile. When mobile VR headsets were first released some experiences popped up that used the phone’s camera to allow the user to look “through” the headsets and view AR elements. This use of one headset for both types of experience will likely increase over time with improvements in the hardware, software, and developer tools. Hopefully we’ll also see a mobile headset that is more compact and easier to carry around, which may increase user adoption.

360 Video Streaming

There are those who question whether or not 360 degree video should be considered virtual reality, but regardless- 360 video has certainly been one of the most successful applications of VR technology- especially for mobile. It has been used for virtual tourism, marketing and advertising, journalism, and many other video-based purposes. It puts the user right in the center of the action, allowing them to view a concert as though they were in the audience, watch a sports game from the sidelines, or enter a scene from a popular movie. The New York Times has an app just for 360 VR journalism experiences.

The next step in the evolution of this form of media, already in progress, will be the ability for users to experience events as they happen through live streaming of 360 video. Companies like Visbit had a presence at OrlandoiX this year, and there was a lot of discussion about the continued application and expansion of 360 video for live experiences. Imagine being part of a developing news story as it happens, or watching a soccer player score the winning goal in real time, from the field.

There is an increased value for the user to observe these events as they happen, and the technology is improving that will allow anyone to have a deeper level of immersion in real-time experiences.

That’s a wrap

That’s it for my iX list, though there were many other interesting areas of conversation. On the more technical side, the discussion revolved around improving artificial intelligence (AI) and machine learning for VR and AR experiences, and also on the potential of “inside-out tracking” . There were some conversations about investment trends, which have been covered extensively in the news. There was also some buzz about Google’s progress in VR. The company’s research in wireless VR headsets is interesting, though development of a quality experience to rival that of tethered headsets is still in progress.

Many thanks to the OrlandoiX team for a great conference and expo this year, to Full Sail University for the beautiful venue and support of this important local event, and to all of the sponsors and participants.