The future of Augmented Reality app development

Paul Walton/ January 24, 2017/ Augmented Reality

Image of a man using the hololens interface to browse his photos

Back in 1970 computing was a very different place. It was the realm of the engineer and mathematician. And this was a problem as they were rarely the people who needed to control the computer’s actual tasks. Alan Kay at Xerox PARC introduced the idea of a desktop metaphor, a level of indirection where blocks of computer storage could be represented by an icon that looked like a file, a familiar object in the offices of the day. They could be grouped using folders and a disk was regarded as a filing cabinet. It was a way of mapping familiar office objects, by function, onto the underlying computer technology. Today we’re exploring the future of Augmented Reality app development and how it will impact your business in the next decade.

Extensions were made beyond the direct metaphor to create the WIMP interface (windows, icons, menus, pointers) but the entire notion was firmly rooted in the use of a computer as a business tool.

The metaphor continued to be developed at Xerox Park, through the seventies and into the eighties while computers grew smaller and faster and became much more ubiquitous. As computers were mainly used for business at the time the desktop metaphor was excellent. It marshalled the restricted space of a two-dimensional screen, allowed stacking of objects in that visual space and gave users a familiar set of objects to work with. It also allowed event driven systems which reacted to the user’s choices, rather than presenting limited choices in a set pattern to the user. The Apple Macintosh brought the metaphor to a much wider audience in the early 1980’s and desktop based GUI’s became the norm over the next decade.

Command line interfaces with their arcane key-press combinations and abbreviated names still lurk underneath the GUI layer but very few people delve into them. The desktop metaphor has been the standard in computing for the last forty years.

But the world has changed.

Computing does not just serve office workers. It is pervasive in our society. From controlling nuclear power stations to supporting chatter between groups of teenagers on phones, the computer now supports every level of activity. Yet those kids on the phone and the engineers in the power station will still be dealing with familiar icons, files, folders, all drawn from that desktop metaphor. The value is in familiarity, a common agreed language we all subscribe to. People who don’t have a shred of each other’s spoken language could sit at a computer together and understand what each other were doing. It is important to understand that the desktop metaphor was and is enormously successful. Is it still relevant? That needs to be examined.

Data has grown in complexity and size by orders of magnitude over this period. The original metaphor has been stretched out of all recognition. Files, rather than holding text, now hold everything from images to sounds, entire films to spreadsheets of data. We discriminate between them by various patches such as file extensions. We have the idea of structured files, which aren’t files really, but complex objects crammed into a file-shaped box.

Data storage used to be simple. A file had an internal structure, usually indicated by a suffix on its name to say whether it was something you could run, contained text or a spreadsheet or had some more complex form. The data was contained wholly within the file and it was usually small relative to the size of the hard disk it was stored on and had to be tied to a specific application which could consume it by the operating system. For larger data a relational database would be used. These would usually operate on a separate machine often with large hard disks with levels of redundancy to protect the data. As they grew more sophisticated the storage would be spread over several machines with software added to fetch the data from the right location.

At the same time as this growth was occurring, the internet and world wide web were growing. A layer of connectivity with an address system that directed you to load at files in a particular format (HTML) allowed you to look at pages of enhanced text and images, which linked to other pages. At first it wasn’t too many of them, but it grew explosively. The number of webpages in the world currently varies between 45 and 50 billion (World Wide Web Size). Each page can be represented by multiple files (HTML, CSS, etc) so that number is much higher. Then consider the number of connections between pages and the complexity grows exponentially again.

And more data was being gathered every day. From bank transactions to clicks on web-sites. The data went beyond the capacity of any single machine, stopped being measured in gigabytes or terabytes, going into the realm of exabytes and zettabytes, each one a trillion gigabytes. Big data had arrived.

The world of Big data is concerned with data-spaces of a size and complexity inconceivable back in 1970. The internet was on a distant horizon. The internet of things, smart-phones, tablets, wrist devices still in the realm of science fiction. All unknown and with no obvious place in the metaphor. Over those forty years the metaphor has shown its power by stretching to encompass them. To a point. Databases have largely superseded file systems for large data. Again, databases were a mechanism for abstracting over storage on disk, a technology that is being phased out. Data persistence was always the goal and we have better ways of achieving that persistence now.

We can blithely store information ‘in the cloud’, access it from any of our devices without concerning ourselves with hard disk space or backup systems.

We are also much more aware of the distinction between information and data. And data is often intractable without tools to extract information from it. An entire science, data analytics, has grown out of the need to discover patterns in data and extract meaning from them. This information is used to detect fraudulent bank transactions and to target advertising at the individual among many other uses. Developing techniques to extract patterns from data and to detect variations from these patterns has been essential in controlling our online activities.

In addition, both available data sets and the content available on the internet have grown so large that new techniques have had to be developed to do effective searching. This growth has raised security issues that still challenge us and has given rise to an entire security industry to address it. The world is connected and has vast tracts of knowledge in the form of text, images, videos, sounds, available by simply selecting the right page. An overwhelming flood of information that has placed a premium on being able to filter out the small part that you want. Search engines became powerful for that one reason. 

Apart from massive information consumption, online payment has become commonplace with the associated security issues. Early systems such as PayPal paved the way, now Bitcoin and other blockchain enabled solutions are rising. Payments are made with computers, with phones, with watches. Millions upon millions of transactions, every day, all feeding into the connected web of data and devices as well as the existing infrastructure of electronic payments. The science of data analytics is used to try and detect fraudulent transactions. Security protocols are used to authenticate connections, structures upon structures building into a complete Virtual world.
It’s estimated 2015 there were 38.5 billion e-commerce transactions (Statista).

As the speed and spread of connectivity has grown people have used it to interact more freely, from emails, to instant messages to video calls. Shared projects supported by cloud computing have become common. And this data is valuable. When you look at an item on a shopping site, data is gathered. If you are dithering over the purchase you may find adverts being presented to you. It is very important to understand that data = money. Remember that as we’ll be coming back to it.

Just to add some perspective, it is estimated that by 2020 there will be around 44 zettabytes of data in the world. That is 44 trillion gigabytes. We produce 2.5 exabytes per day. To put this in perspective that is around 90 years of constant HD video (IT World Canada)

As if this new technical landscape wasn’t complex enough, It isn’t just humans and personal computing devices that are connected. The Internet of Things has arrived, common devices such as thermostats, fridges, lights connected to the internet, able to pass data about their operation or to accept data and commands remotely. Devices that, with the right programming, can talk to each other and co-ordinate their efforts. The IoT is in its infancy but the potential for growth is immense.

An existing use of the IoT is the idea of the smart home. When the garage door is opened by a member of the household, the key identifies them and communicates with the rest of the house ‘X is home’. The house can then adjust the heating, put the lights on, play music, all to X’s preference. That is a localised example, there is no reason why you couldn’t play a song in a player on your computer and have the sound come out from an IoT speaker system on the other side of the world. The applications are immense.

There are problems as well. Two things are needed to turn a device into an IoT device. It needs some form of transport layer such as wifi, Bluetooth, or other, more power-efficient methods to communicate with ‘something’. This isn’t as easy as it sounds as wifi, for instance, takes up a lot of space in a small device, say a thermostat, and also consumes a lot of power. Still, there are technologies to deal with this so it is a lesser problem. It is the ‘something’ to connect to where the issues lie. The software that sits on the device and the receiving software need to conform to some standard to communicate with other devices. This software will control the flow of data (and remember, data = money), so quite a few bodies are interested.

Some leading ones are AllJoyn to which Microsoft, Cisco systems, Panasonic and Sony subscribe. OIC (Open Interconnect Consortium) includes Intel, Dell, Samsung and, again, Cisco Systems. Apple have their own Homekit framework and Google have Brillo, a lightweight android like operating system that goes on the device and Weave, the middleware that lives on the hub end. One of the oldest is Zigbee and Z-Wave is also a market leader. There are a lot of players and lots of standards.

The problem was highlighted recently when Google’s home automation company, Nest acquired Revolvr, a hub that controlled a variety of household gadgets from a smartphone app. The software receiver was a cloud-based service. Nest shut down the cloud-service, after suitable notifications to clients, effectively leaving them with a $300 brick. Having a standard for communication and a reliable means of communication is going to be vital for IoT’s growth. Security is going to be another issue. While it may be irritating but not disastrous if a hacker can access your heating system, in industrial situations it could be disastrous. The Internet of things is coming, but there are likely to be some birthing pains. Fact break: It is estimated that there are 6.5 billion ‘things’ connected to the internet (Gartner). It is not unreasonable to expect to see the same exponential growth we have experienced in other areas.

As this Virtual world of computer data and connected devices has grown larger, the desktop and its ability to handle this world has shrunk. It has become a very small, restricted window on this world. Literally shrinking as we use tablets and smartphones that each contain computing power beyond anything dreamt of back when Alan Kay came up with the desktop metaphor back in 1970 but with screen sizes even smaller than PC’s.

Most discussions of Augmented Reality app development and Virtual Reality being with a discussion of hardware. Of wearable technology, such as the Oculus Rift or the Microsoft HoloLens. Let us start at a different place. We’ll push Virtual Reality aside for now. The most important technology here is Augmented Reality. Forget pretty imagery, of Pokémon infesting high-streets or whales leaping out of gym floors. Augmented Reality allows us to view both the real and virtual world in the same context. It makes a connection between these two vast worlds and allows us to interact with them both simultaneously.

There are three elements in augmented reality. The objects to be displayed over the environment, the means of placing them within that environment and a means of interaction with those objects.

The first is relatively simple, overlaying a graphic on a view from a camera. There are the complexities of producing the image, shaders, animations, but these are known techniques with software designed to support their creation. In mixed reality the environment can be virtual, but let’s stick with the simpler case of augmented reality for now.

The second element is far more complex. Placement of the virtual objects or artefacts, in the real world. The simplest is showing the artefact a set distance from the virtual camera, overlaying the real world but with no real link to it.

A higher level of interaction comes through tracker based positioning, where an image is encoded then used to position the image. This is currently the most common form of augmented reality. It is often used to attach 3D images to catalogues or labels.

Location based AR places the objects at fixed geographic locations, using the GPS functionality of the viewing device. This can be quite effective for tying the real to the virtual. Virtual tours can be supported in this way, or direction systems. Then there are the more complex techniques that attempt to understand the environment through complex algorithms and fit the object into the environment, these techniques cover object-recognition and SLAM systems (Simultaneous Location and Mapping) which attempt to construct maps of unknown environments in a tractable time and fit the objects into them. This allows objects to interact more closely with the environment, positioning the artefact in front of or behind real world objects.

The third element is regarded as the most important: the ability to interact with the artefacts. That is what allows us to truly bridge the reality/virtual gap. Oh, we can produce AR that doesn’t have this element. A model of the solar system floating in front of us in glorious animated 3D for example. How much more useful does it become though when we can interact with it, from touching a planet to get information on it from the web, or, better, changing the orbit of a world and seeing the effect on the solar system. Interaction changes it from a pretty display to a tool.

In augmented reality app development and mixed reality, we can make connections between the virtual and real in a far more direct way than we do through the medium of a two-dimensional depiction. This is a visible medium that lets us see what we are working with. It takes us away from the desktop and into the living, breathing world. Into two of them in fact, one virtual, one physical.

We don’t have to worry about the size of our monitor. We don’t have to stack windows on top of each other. We can use new metaphors. Fortunately, we have a wonderful model available to us. The world itself. Like the virtual world, it is vast, too vast to comprehend without deliberately blowing our minds by thinking on a galactic or universal scale. That’s OK, we’ve evolved with techniques to deal with it. We focus on what is around us, in our environment, what is relevant to us. We’ve evolved to reduce the background to a fixed pattern and to notice differences. The strange. The unexpected. The tiger’s tail sticking out from behind the bush.

This can be applied to a range of situations. Looking through spectroscopy information analysing the results of drugs tests as a real depiction of what is going on rather than a list of numbers, for example. Visualisation of the data where you can get in among a representation of the results as though it existed around you. A direct view of the physical state of a racing car as it is going around the track. A visualisation of a rover on a remote planet or of an inaccessible archaeological site. Students can be present at surgical operations taking place in distant locations, taught by the leading experts. Other students can learn geological techniques from live oil drilling operations seeing what is happening underground. Rather than learning from books they can learn from actual actions with all parts fully labelled and identified and linked to the web for deeper information. We can truly augment reality.

Finally, rather than operating in the framework of the technology, sitting in front of a rectangular display, stacking windows full of numbers or text, trying to draw information from them, we have the chance to work the way we’re designed to, interacting with the world using our senses. We are tool-makers. We’ve raised civilisations by using tools, applying them to resources to produce services and products. And destroyed quite a few using the same techniques. We can make tools now, out of the virtual and physcial combined. We can carry our toolkits with us through the real world, applying them as needed. It isn’t dependent on location or environment. The desktop metaphor simply isn’t relevant any more.

All this sounds wonderful yet there are some significant hurdles. One of the first is hardware. Currently tablets and smartphones run simple AR systems. This is a limited form of AR as it isn’t immersive. There are several wearable solutions emerging, the Microsoft HoloLens, the DAQRI helmet, a variety of others. Again, standards are needed. A morass of incompatible devices will not help public adoption of the technology. Still it is estimated that Augmented/Virtual Reality is set to be a $150 billion industry by 2020, disrupting the mobile industry (Digi-Capital).

The same problems that face IoT face AR. Standards are needed, consistent standards rather than a turf-war among the big players. There is hope that the process of settling on standards will be swift and allow developers to focus on creating tools that will allow us to interact with our Augmented world.

At the moment it may feel as though we’re sitting at the bottom of a valley with the avalanche that is big data crashing down from one side, an avalanche of IoT from another, and payments, security, all on an increasingly outdated hardware infrastructure, threatening to overwhelm us.

Fortunately, we do have some advantages. Years of experience in software development have honed development skills. We are very good at producing small, focused modules. Agile techniques such as test driven development, SOLID principles and SCRUM project management techniques have made us very good at delivering small focused units of software. Which is exactly what is needed. We need to form a new metaphor. Once concerned with tools, devices, connectors and owners.

We have developed strong techniques for defining what is relevant to us. Data mining and analytics, recognise patterns in data and extract information. Artificial intelligence holds a set of strong tools for determining relevance, such as decision trees and neural nets. In fact, artificial intelligence thrives on big data, the larger the training set, the better the performance. And of course, we have augmented reality, a new way to look at this world, to use our evolved faculties to extract relevance.

This is not the world of the desktop any more.

In order to engineer reality we have all the tools we need but need to shift our thought-patterns. We need to consider environments not windows. We need to consider artefacts, not files. We need to define reality bridges that allow real objects as well as virtual ones to be considered in equivalent terms and enable connections between them with security built in at the root level. We need data and, more importantly, information to be treated as first class objects, not just a line in a spreadsheet. We need to enforce relevance and significance in screening our view of the world so we aren’t overwhelmed.

It is time to move beyond the desktop and its limited user interfaces. At Augnite we are engineering solutions that harness technology to solve difficult problems in new and powerful ways, allowing us to encompass the new reality, making data and devices usable and visible, allowing us to apply our primary senses to navigating through the avalanches of new technology. We are forming a new metaphor based on how we function now , not how Victorian clerks worked. We are defining a new reality that will help us manage the information revolution.

If you’re a CTO, CEO or software development manager looking to kick-start a new Augmented Reality app development initiative, or enhance or optimise the performance of an existing AR app, there has never been a better time to get started. Here at Augnite, we believe in enabling large companies to innovate faster through the ideation and deployment of high performance Augmented Reality apps and optimisation services. Using a data-driven agile project management methodology, we enable our clients, throughout London, the rest of the UK and beyond, to get to market faster and succeed by focusing on the right KPI’s and measures of success. If you’re new to the world of Augmented Reality and seeking to discuss your project, contact Augnite today to get started.