#S02E05 - Augmenting Reality: How AR and VR Are Changing Entertainment

Pokemon Go brought Augmented Reality to the public on an unprecedented scale, becoming many people’s first experience using the technology. But while most people have moved on from playing Pokemon Go, there is still plenty of development happening in the AR space. So how is AR going to make the leap into everyday life, and what will that future look like?

Featured in this episode:

• Joseph Purdam - Phoria.
• Ted Schilowitz - Paramount Pictures
• Greg Sullivan - >Microsoft

CREDITS

This episode of Moonshot was hosted by Kristofor Lawson (@kristoforlawson) and and Andrew Moon (@moonytweets).

Research for this episode by our intern Caralene Ho.

Our theme music is by Breakmaster Cylinder.

And our cover artwork is by Andrew Millist.

TRANSCRIPT

John Hanke: “Imagine discovering a Squirtle hiding along the waterfront in San Francisco, or a Bulbasaur at Shinjuku Station, or even Pikachu, hiding by the Eiffel Tower.”

KRIS: Welcome to Moonshot - the show exploring the world’s biggest ideas and the people making them happen. I’m Kristofor Lawson.

ANDREW: And I’m Andrew Moon.

KRIS: And you’ve just listened to Niantic CEO John Hanke announcing their game Pokemon Go at a press conference in 2015. Pokemon Go took the world by storm, with 130 million players downloading it in the first month alone.

Reporter: “We’re back now with something that may explain any odd behaviour you’ve seen in the streets recently. The smash-hit mobile game app Pokemon Go has only been out for a few days but it’s already got millions following their smartphones to the most random places.”

KRIS: Millions of people walked around their neighbourhoods and cities to try and find digital creatures hidden in everyday places.

Reporter: “Pokemon Go uses the GPS data from your cellphone to let you track Pokemon in the real world. We’re in central park.”

KRIS: And in the process - it set 5 world records as it became the most downloaded app - topping charts around the world, and grossing more than $100 million in revenue.

Newsreader: “And according to Apple, the Nintendo nostalgia has already 30 million downloads, with more downloads in its first week than any other app, ever.”

ANDREW: Augmented Reality - or AR - takes elements of the virtual world and places them into our physical, real world.

ANDREW: And Pokemon Go brought Augmented Reality to the public on an unprecedented scale, becoming many people’s first experience using the technology.

KRIS: Most people have moved on from playing Pokemon Go, leaving a dedicated core group of players. So how is AR going to make the leap into everyday life, and what will that future look like? We’ll find out after this.

Ted Schilowitz: A lot of people thought, "Oh, that was just like amazing overnight success. It just came out of nowhere and suddenly everybody was playing this." But if you study this field, you know that Pokémon GO was not, by far, was not the first what we call ARG or alternative reality game that was created.

KRIS: This is Ted Schilowitz, the futurist in residence at Paramount Pictures. His job is to take emerging technology and experiment with ways it can shape the future of entertainment. We spoke to him at the Magnify World conference in Melbourne.

Ted Schilowitz: The same company that created Pokémon Go created another game that's very popular called Ingress. Ingress had millions of users and still does. It didn't get quite to the scale of Pokémon Go, but it was a much more sophisticated game than Pokémon GO that people had to really put a lot of energy and time into if you really wanted to play it. But it's the bones of what Pokémon GO was built on.

Ted Schilowitz: And before that, there was a phone based technology that we simply just called Geocaching, which was like the world scavenger hunt. And that's like the base core technology that Ingress and Pokémon GO were built on is the idea that you can map the world with your phone, and it would know where you are in the world.

ANDREW: Now Pokemon Go put AR on the map but the technology has been around since the 1990s, it was pioneered by Boeing’s Tom Caudell and David Mizell. They were asked to come up with a replacement for expensive schematics that factory workers used to build wiring systems for Boeing’s aircraft.

KRIS: Tom and David proposed a head-mounted system which would project the blueprints directly onto the workspace. Workers would receive customised instructions from a central computer, replacing the inefficient and costly analogue solution Boeing had used until then.

KRIS: Tom coined the term Augmented Reality when developing their solution, but it’s a concept that has only become reality in the past few years.

Greg Sullivan: In 2016 when we started shipping HoloLens, it really set the high watermark for what was possible in terms of an untethered, self contained, hands free, holographic computer.

ANDREW: This is Greg Sullivan, he’s a Director of Communications at Microsoft.

Greg Sullivan: And I get to work on our mixed reality stuff, which means HoloLens and the software that lights it up, so it's pretty amazing.

KRIS: Microsoft’s Hololens is a headset device that you wear to project images right in front of your eyes. It kind of looks like a space-age mask that you may have seen in Ready Player One - although it doesn’t actually take you out of the real world.

KRIS: I’ve actually tried a hololens and in many ways it’s like putting on a very futuristic ski mask. You put the headset on and then you’re presented with a menu that you can use to open different apps. And you select those apps by putting your hands out in front of you and making gestures. I was looking at a cross section of the human body that medical students could use to learn about anatomy. Overall the experience is really great and you can really feel the potential of this tech… but it’s very limited by its size and also the field of view which only covers around 35 degrees. Meaning most of your vision doesn’t actually include digital overlays, so I still felt like the experience was still somewhat disconnected from the real world.

ANDREW: You might have heard Greg refer to the HoloLens as mixed reality, instead of Augmented Reality. See AR is just one piece of the puzzle when it comes to alternate reality. Another one you might have heard of is Virtual Reality. VR builds a virtual world around you, replacing your entire field of vision with a computer generated one. And some of the more highly developed VR experiences create a 3D soundscape too - building a fully immersive digital world.

ANDREW: Mixed Reality is everything between the real world and building a virtual reality. Now this includes AR, which takes the real world and places virtual information over the top.

Greg Sullivan: To us, this continuum exists between the real-world and the digital or the virtual world. And the view is that any time you have some of the real-world and some of the digital world meaningfully blended together, you're mixing realities, and let me give you a couple examples of this continuum. At one end of it, you can have something like… Google Glass I would say is the very simplest idea of you have a real view of the world or the environment that you're in and you have a system that sprinkles in just a tiny bit of digital information that may or may not be world or directionally locked, so depending on where you look, it might change, but in any case, you're sprinkling in a little bit of the digital realm into your primarily real view of the world or you are augmenting reality.

Greg Sullivan: If you traverse that continuum all the way to the other end where you are almost at the 100% virtual or digital, these are the immersive systems where you can't hear the sounds of the real-world. You only hear the digital sounds. You have a headset on that blocks off your ... occludes your view, and so you're immersed in this digital realm, but our view is that even in those environments, the environmental awareness of the system, the headset that you're wearing knows where the floor is and incorporates that into your virtual experience. It knows where the walls are most likely and prevents you from walking into them by translating that somehow into the virtual experience.

Joseph Purdam: I guess we fall under the banner of a technology studio or an XR company. XR is a term that's recently been coined in the industry more around ... I guess it's an umbrella statement that encompasses a couple of different alternate realities being AR, VR, and MR.

ANDREW: That’s Joseph Purdam. He’s the Experiential Director at Phoria, a company that designs augmented reality experiences, but also works in other realities too.

Joseph Purdam: We have done some AR implementations where we've, I guess, translated across existing projects that have an entertainment value. We've just sort of done that in good faith or something, that we could contribute. But in the virtual reality space we're running several clinical research trials, initially around using VR as a pain and anxiety management tool in youth oncology with the Royal Children's Hospital and our research partners for Murdoch Children's Research Institute.

Joseph Purdam: We started that project over two years ago, and the appetite for virtual reality and, I guess, these kinds of immersive media solutions as healthcare interventions, it was ... They were pretty scared about implementing these kinds of things, because there weren't many examples of it out there at that point. It's really nice to look back now and see that there's five or six simultaneous VR-related projects running just within the children's hospital.

KRIS: Phoria has been developing augmented reality media since 2014. Even in such a short amount of time, Joseph has seen rapid growth in the technology and demand for AR.

Joseph Purdam: What has changed the most for us, I guess, is the toolset has evolved massively, and the marketplace has evolved as well. So the types of experiences that we're able to produce and facilitate through AR and VR are much more refined in-depth, interactive experiences.

ANDREW: When the Hololens was released to the public, there were a handful of AR games that came alongside the system. One of them stood out from the rest.

Joseph Purdam: There's an experience called Fragments in HoloLens, where it's basically like a murder mystery sort of thing, and characters come up and look you in the eye, or they sit on the couch next to you. And it's that level of interaction with the environment, and the blurred line between reality and the experience that you're absorbing through these headsets. I think that's the most interesting direction to take this

Ted Schilowitz: It's actually fairly functional, and dynamically relatively achievable to do that in a mixed reality environment where you could have a crime scene play out in your living room, or in your office, or in a train station, and you have to go and pursue that and figure out the clues and work through the story dynamics.

ANDREW: That’s Ted Schilowitz again. He sees the Hololens as a strong beginning to developing AR hardware, but insists there’s a long road ahead.

Ted Schilowitz: It's not a mainstream device yet, it’s relatively expensive for what it is, and it's really, it hasn't captured the imagination of everybody yet, but there are plenty of them out there in the world. If you can get one and play that, you'll be well on your way to learning about the possibilities of storytelling in a spatial universe, in a mixed reality spatial universe.

KRIS: Now HoloLens isn’t the only mixed reality device on the market. There’s also the much hyped - Magic Leap One which tried to package all the technology into a smaller headset that more closely resembles some oversize sunglasses. And Facebook’s Oculus have also been producing a number of lower price-point headsets for Virtual Reality applications which have the potential to open up a mass market. And Ted says with all of the technology that’s becoming available - the future of mixed reality devices looks a lot different than what it is today.

Ted Schilowitz: It's not wearing a big box on your face and playing games. That's one little, little fisher of what's really going to happen when we actually start wearing the right device in the right way that connects to our humanity correctly, and our iPhone or Android phone finds its way into a drawer and you're like, " Yeah, I used to use that thing but I don't use that anymore. I use something better and more interesting, more evolved."

KRIS: Many industry insiders say the technology is still in its early stages and will be radically different in the future. But for now, the most immersive experiences we have are confined to a wearable headset.

Joseph Purdam: Yeah, I guess like MR and HoloLens are heading in the right direction, I don't think that they've completely solved it, with things like field of view limitations. But I think that that's the best foot forward for immersive technology at this point. There's still ways to go, and I mean, HoloLens 3’s reportedly coming out end of this year, start of the next, so I think that'll be another really big leap forward.

ANDREW: Now Headset developers aren’t the only one on the AR train. The next step in augmented reality is already underway, and there’s no cumbersome headset involved.

ANDREW: Apple’s ARKit and Google’s ARCore are new platforms that seek to contain the whole experience within your phone or tablet.

Joseph Purdam: Obviously with ARKit and ARCore coming out, you can see really big tech companies like Apple and Google doubling down and investing in this space. So especially with our relationships with Google, we've had early access to some really interesting tech stacks that are allowing us to produce incredible experiences like shared persistent markerless AR experiences that just weren’t possible. The tech stack wasn't there four years ago.

KRIS: But with so many branches of augmented, virtual and mixed reality, it’s not only a little confusing to the outside observer – it’s become a real challenge to tie them all into a single unifying experience.

Joseph Purdam: So if you have a phone, that's, you know, an iPhone 6S, or say an iPhone 7 or newer, Android back to about year 7, you can use AR, markerless AR, to project into your environment. If you own an MR headset it's easy, it does all the room scale tracking for you, projecting into your environment. In VR you create, I guess, an environment that you could project that into.

Joseph Purdam: So building content that works completely ubiquitously across those platforms, I think, is also a really interesting prospect, rather than being limited to where the technology is at, where the ... There's always this conundrum of do you build technology that everyone can access, or do you build something that utilises the latest and greatest hardware. And I think if you can do both, that's ideal.

ANDREW: Here’s Greg Sullivan again, outlining Microsoft’s goals with Hololens:

Greg Sullivan: In general, the trends that we're trying to achieve in some ways are three almost mutually exclusive goals. We also need to do all of them in a way that doesn't sacrifice any one. Those goals are really to make the technology more immersive, more comfortable, and more affordable.

Greg Sullivan: It's easy to do one of those, perhaps even two; to do all three of them is the real challenge.

KRIS: And we’ll continue to look at what an augmented reality future will look like, right after this break.

Greg Sullivan: One way that people are familiar with is something as simple as I download the Pokemon Go app on my phone and I play it. We would argue that's a simple augmented reality experience.

KRIS: Welcome back to Moonshot - I’m Kristofor Lawson. And when it comes to augmenting reality, the technology is just one piece of the puzzle - the end goal is an immersive, interactive user experience. And as we heard earlier, the original concept of AR was designed around aircraft.

Greg Sullivan: If you are an aircraft mechanic at Japan Airlines, you would use a HoloLens to learn how to work on an aircraft engine. It would be a digital holographic aircraft engine, so that means you could do things like stand inside it when it was running and learn how the systems interoperate and evaluate it from kind of inside. There are a whole bunch of advantages to doing something like that, but most people aren't Japan Airlines aircraft mechanics.

KRIS: But while AR might be a great way of learning how to fix an engine.. the effectiveness of AR in the long term is certainly not limited to just the field of engineering. And Greg Sullivan sees the technology being applied in other more traditional industries… like retail.

Greg Sullivan: You're already seeing the digital and physical worlds blend in some really interesting ways, and so I think for most average people, whether ... Your first experience with this might be going to a VR arcade and playing a game where you're immersed in this alternate universe, and you can move around in it, and it offers you this fully immersive 360-degree experience. That's what most people think about when they think about VR or mixed reality today from a consumer standpoint, but the ways in which this technology trend will manifest are varied.

Greg Sullivan: It will be on the floor of a retail establishment where they don't want to have people slip on something wet that spilled in an aisle, and today, that is actually a considerable challenge for some large retailers, and so what they have to do is wait for a customer to slip on some ... the spilled item or wait for an employee to find it and mop it up, but imagine if you had sensors in your store that had the superpower of perception. They could see in three dimensions. They had a flight, time-of-flight depth sensor that could differentiate between a less than 1 millimetre thick puddle of water on the floor and the regular waxed floor, and based on that awareness, could send an alert to the custodian to quickly get there and mop up that spill before a customer slipped and fell, and you have all kinds of other problems.

Joseph Purdam: A project we did for the city of Melbourne last year, Santa's Little Helper, was definitely an entertainment focused experience for the user, but the motif behind it was more of a dispersed business strategy for the city of Melbourne.

ANDREW: This is Joseph Purdam again.

Joseph Purdam: The idea was by giving the user rich AR interactions in various spaces throughout the city, could we disperse them and actually get them to travel to places that they wouldn't go to otherwise? So while the user experience is an entertainment based experience, the driving factor for the client is actually something much more dynamic.

ANDREW: And Ted Schilowitz sees so much potential in the technology, that he thinks it could lead to people completely reimagining storytelling in entertainment.

Ted Schilowitz: It's essentially the beginnings of learning about simulation behaviour, about turning entertainment into simulations and not just traditional story-based cinema-based language narratives. It's a bit like learning how theme parks take you on journeys and tell you stories along the way with that journey with physicality and how to apply that into a home-based or commercial-based medium that can go to a much wider scale.

KRIS: Ted also says the future is sooner than people think. After all, we have been learning simulation behaviour for some time now.

Ted Schilowitz: I kind of make the point that we have now for at least 10 years, been firmly in the art and creation of simulation-based behaviour. Everything we do as humans that used to take certain degrees of physicality, writing a letter was a physical thing. You took out a piece of paper, you wrote it with a pen or a pencil, you put it in an envelope, you put in a stamp, took it to the mailbox, you mailed it. It’s a physical behaviour.

Ted Schilowitz: If you wanted to know what was coming up on your calendar, you went to the wall, and you flipped around on a calendar that was on the wall, you looked at this thing called a day planner, which was like a little book. That physicality, right? If you wanted to plan a trip, you opened up a map on a big piece of paper, and you folded it out on the table. That physicality. If you wanted to take a picture and then share that picture with people, you used Analogue Devices and you took a picture and then you would either mail that to somebody, or put it in a photo album and show them. It all had physicality to it, right?

Ted Schilowitz: Then we moved into an age where the digital transformation started to replace physicality with a simulation. Now when we actually look at our calendar, and we're not really looking at our calendar, we're going at this digital simulation of our calendar inside this little four inch shard of glass that we hold in front of our face all day long. We want to look at and share pictures, we do that in a simulation. We want to send an email, we want to communicate in almost any way, shape, or form, we do it in some sort of simulation behaviour. We don't really do it anymore. We do a digital simulation of it, right? And we buried all of that inside a tiny little device that we carry with us, that is effectively the biggest screen we can get and still put in our pocket. That's kind of what we've created. That's the art of where we've gotten so far.

Ted Schilowitz: And what I believe is we're getting ready to emerge out of that cocoon. That's the caterpillar stage, right? It's like it's all inside this little caterpillar-based device, and we're getting ready to become butterflies. We're getting ready to turn it into this amazing, remarkable world where when the device gets light enough and nimble enough, and smart enough, and effective enough, we're not just going to hold it, we're going to wear it. And when we can wear it, we can remove the small simulation behaviour, and then enter an age where we're now back to big behaviour, but it's simulated. So it's that digital transition is going from like the pinch point in the hourglass to the other side of the hourglass where things can be gigantic and all around you, just like the real world, only we're going to do a digital layer onto the real world. And that's kind-of what's so intriguing to me about the real opportunity around this

ANDREW: And for the millions of people around the world who downloaded Pokemon Go, walked outside, and set off in pursuit of their neighbourhood Charmander - they literally had that caterpillar to butterfly experience for themselves. And for developers and creators - it showed there was real, commercial potential in the AR space.

Joseph Purdam: I think it had an impact on anyone who works in this kind of immersive technology space, because it gave everyone a common ground and terminology for the validity of the technology. So it allowed us to have conversations with clients where before they'd never heard of AR. Pokemon Go was at least something that they could ground their presumed knowledge in. I guess ... I mean, I was probably one of the only people who didn't play it religiously. I downloaded it but didn't get to far. I was a fan of the original Pokemons, but I found it was, as an AR experience, pretty... It was a little bit gimmicky, it was really just looking through the lens of your camera. It didn't assume ground planes, it didn't actually do any environmental calculations like what a modern AR application would do.

Joseph Purdam: So I found that while it was amazing for actually bringing AR into the forefront of people's thoughts and giving AR a platform to stand on, in a way, I think what ... So Niantic are now working on the Harry Potter game, and I think now with the kind of persistence and shared AR capabilities that are out there, along with being able to build ARKit and ARCore, do multiple wall and ground plane detection ... So you can have a table detected and a ground plane detected, and a wall behind them detected. You can have really dynamic 3D environments come to life through AR now. I think that's gonna be really exciting. I think AR as it was depicted in Pokemon Go, is, I think, pretty limited, but obviously a phenomenal app for the whole industry as a discussion point, at the very least.

KRIS: Most augmented reality technologies are still in the prototype stage, or even earlier in their development. The few AR headsets currently on the market are highly expensive, with many of them designed for developer use only. I asked Greg what pricepoint Microsoft had in mind where they would see widespread adoption of their technology and he couldn’t give me a specific answer other than to say they had one in mind - but he also suggested that it’s a lot like every other emerging technology, starting out as an expensive rarity, then as time goes on it becomes an accessible and widely adopted part of everyone’s life.

Greg Sullivan: I'll never forget, actually my stepdad had one of the first microwaves I ever saw in anyone's home. This was back in the early 70s. In any case, the microwave was a very expensive commercial item that over time the economies of scale were achieved. The ROI was clear and obvious for restaurants or other commercial enterprises, but it took a little while before microwaves came into all of our homes… I think the same's true of cell phones, the same was true of the PC, the personal computer. It was probably comparable to HoloLens in terms of the initial price of the IBM PC back in 1981 when it launched. And the only people that had them were the folks at work, it was probably the accounting department.

Greg Sullivan: But this tool cost, you know, $3,000-$5,000 but man oh man it changed their lives. It gave them superpowers almost literally. It took a while, and then of course the economies of scale kicked in and the price of those things came down and they started showing up in everybody's homes... I think this story is going to be similar. I think we've maybe become over- fixated on a specific manifestation of the technology in the context of a head mounted immersive or see through head mounted display. That is but one of the ways that Mixed Reality technologies will start to come into our lives. And in some ways, it will just be easier to check out at the grocery, the lines at Disneyland will flow themselves more.

Greg Sullivan: When what we're doing here is giving computers the ability to understand their environments and interact with them in ways, and then put the human in charge to really benefit from that interaction of the digital world in 3D space. We're just really starting to figure out the ways it will benefit us.