What is the difference between Virtual, Mixed and Augmented Reality?

Paul Walton/ January 31, 2017/ Augmented Reality

A women using the hololens to interact with a 3D model on her desk

At the dawn of humanity there was only one reality. The physical world you could point at and interact with. That ended with the first abstract thought. Early men recorded information in petroglyphs on cave walls, communicating that information. The drawings were physical but the information they conveyed was intangible. This abstract world was pervasive. We have records of information from ancient Egyptian builders, the Domesday book which recorded the ‘great survey’ of Britain. The abstract world of data and information has been with us for a long time. The same is true of physical technologies which allow us to access abstract information. From cave walls through Victorian ledgers to the latest computers, they all achieve the same effect, allowing us to bridge between the physical and abstract realities. We recently explored ‘The Future of Augmented Reality app development‘, today we’re answering the burning question: What is the difference between Virtual, Mixed and Augmented Reality?

So, mixing up realities is nothing new. What is different now is the scale of these two worlds. In the past, the abstract world has been a series of localised bubbles, only relevant to the traders transporting goods, or the historian addressing a series of events. That has changed. Today information has equal importance with our physical possessions. Bank accounts, medical records, even our identity, exist primarily in abstract space. Losing the physical references to them can be catastrophic. The advent of, first, computers and then the internet, has allowed an explosive growth in both the amount of information stored, but also its importance to society and the individual.

Computers have also allowed us new ways of looking at that abstract world. Charts, graphs, visualisations. At first flat versions that allowed us an easy way to abstract over intractable sets of numbers, ways to see trends and patterns. Then more sophisticated ways to present the same information, 3D graphics, animations. Still, our computers were a window into another world, an abstract reality.

Now, the technology has evolved to the point where, after our entire history of the realities being separate, we can bring them together. Computer vision techniques allow the computer to analyse both the real, physical world, and the abstract world of data. Computer graphics techniques and technologies have advanced to the point where we can display both simultaneously.

Now this isn’t a simple thing. The physical world has rules. It has positions, it has adjacency, it has physics. The abstract world is one we’ve created ourselves. There are various measures we can use to add structure to it. Let’s just add one in for this discussion. Relevance.

This isn’t such a bad measure as we use it all the time in the real world. Think about your home or office, maybe the street you’re walking down or the coffee shop you’re sitting in. The things around you are relevant and you care about them. The people next to you, the coffee on the table. The other places around the town, the country, the world, you could not care less about. They’re not relevant to you.

Same with the abstract world. We’ve all experienced the flood of responses to a web search and the frustration we go through as we refine the search to eliminate as much of the irrelevant as possible. Smart searching, or finding the relevant, is becoming more and more a base need in the overwhelming world of stored data. This is even more important when we start to bring realities together.

We simply do not have the capacity to show ‘all’ the data for even simple requests. We have to acknowledge its presence and scale, but we can’t look at it all at once. George Miller, a psychologist at Princeton, back in 1956, published a famous paper, ‘the magical number seven, plus or minus two’, stating that this is the average number of objects a human can hold in working memory. So, putting ‘everything’ into a display is pointless. We must hone the information down to the relevant, and, beyond that, to the most relevant.

So, the physical reality has its spatial laws, which locally define relevance, and we’ve got an abstract reality, also defining relevance. This allows us to use software to build a bridge between the two realities. At its lowest level, this allows us to view abstract data using the same tools evolution has honed over millennia. Our sight, our depth perception, our colour sense, our ears. It raises the abstract to the same level as the ‘real’ objects around us and, done correctly, allows us to interact with them in a similar way.

We have terms for this. Virtual reality, augmented reality, mixed reality, more obscure terms. What we need to consider comes down to a simple list.

1.) What is the environment? It could be real, using camera(s). It could be a virtual space, created entirely within the computer.

2.) Objects. These can be virtual or real. Of course with real objects we need a way for the computer to gain information about them. This could be done through image recognition techniques or they can be part of the IoT (Internet of things) having their own internal electronics to pass data and commands over the internet.

3.) Where the objects appear in the environment. This can be done in a variety of ways, from simple placement in front of the camera, to placing on some tracked image, through to more complex techniques that use SLAM (Simultaneous Localisation and Mapping) technologies to map the real space into a version the computer can reason about.

4.) Interaction. The objects need to handle interactions with them as a physical object would. Perhaps not with the same range of possibilities, but a pretty picture that you can’t do anything with is still just a pretty picture.

Virtual reality is where we have a virtual environment and virtual objects. There is no element of the physical reality in there, although environment and objects may well be based on physical objects.

Augmented reality is where the environment is real and may include real objects, but virtual objects are overlaid on that display of the physical world, augmenting it.

Mixed reality is a more general term, where the environment can be virtual or real and have virtual or real objects imposed on it. Think of it like the holodeck on the starship Enterprise from Star Trek. You can impose whatever environment you want. A bar, the surface of mars, the bottom of the ocean, and bring real objects, possibly people, into that environment. Augmented reality is a specific case of mixed reality.

And there are problems. Primarily placement of the objects within the environment. This is difficult. First there is the problem of recognising the spaces and surfaces of the real world, determining depth. There are problems when multiple users are trying to exist within the same space and see the same objects. Their different positions can be tricky.

Then there is the problem of making that space useful. Setting up interactions that allow the users to complete their tasks. Pretty demos are fine, but at some point the new technologies have to earn their way. Do they make us better? The technologies for worn computers such as the Hololens or DAQRI helmet are in their infancy. They will improve dramatically, as we’ve seen with other, similar technologies, but the underlying issue that will determine success or failure is back to our earlier discussion point.
Relevance. Do they assist us in making sense of the enormous mass of information we’re flooded with? Of course, they don’t do this by themselves, there’s a while Iceberg structure of unseen processing going on with every data request. When we receive that filtered response from the search engine or the query engine, does that bridge between the real and abstract make it easier to understand?

Like any other bridge. It requires engineering. Reality engineering. We need solid techniques for displaying retrieved information in a consistent and seamless manner. We need to restrict the display to the relevant and highlight differences within the displayed items. We need to spot the tiger’s tail hanging out from the bush, using the tools evolution has given us.

And that is the advantage of the new technologies. They allow us to apply our best tools to our most pressing problems. Just showing pretty pictures isn’t good enough. We need solid tools bridging the worlds, handling interactions and allowing us to handle the information explosion in the same way we handle the real world. We need Virtual, Mixed and Augmented reality engineers and software development companies to bridge the gap.

If you’re a business or brand new to the world of Virtual, Mixed and Augmented Reality, the possibilities associated with the technology can be mind blowing, the potential scope for business application is enormous. If you’re looking to extend the capability of an existing mobile software offering, or rolling out a new Augmented Reality initiative we’d love to hear from you. Contact Augnite today to kick-start the conversation.