It takes some sophisticated technology to create Augmented Reality (AR) experiences. How does Augmented Reality work? It all comes down to location, location, location (and identification, identification, identification).

The Equipment

AR images can be displayed on a multitude of devices including eyeglasses and goggles (remember Google Glass?), headsets, and heads-up displays like helmet visors, but the most common way to use AR these days is on a smartphone, in a variety of apps and games.

Types of AR

There are two broad types of augmented reality, these being marker-based and markerless.

Marker-based AR uses image recognition to identify objects that have been preprogrammed into your AR device or app. Fiducial markers (something placed in the field of view for use as a point of reference) help the AR device determine the position and orientation (called pose) of its camera.

Here’s how it works: First, the camera feed is switched to grayscale to speed up the processing time. When it detects a marker (often something simple but distinct, like a QR code), the device compares the information from the marker with all the markers in its brain. Once it finds a match, it uses the marker’s information to mathematically determine the pose and it then displays the AR image at the exact right place.

For example, let’s say you’re in a museum-like The Franklin Institute. By pointing your phone’s camera at a marker on an information plaque, you might be able to instantly see more about the artifact at which you’re looking.

Markerless AR is a bit trickier. Not having markers means that nothing has been preprogrammed into your device—it has to recognize items on the fly. The recognition algorithm in your device looks for patterns, colors, or other features that might tip it off.

Let’s take the popular mobile game Pokemon Go as an example. The incredibly popular game tracks your location via GPS, then uses geographic data and your phone’s clock to choose when and where to have Pokemon characters appear. For example, if you’re walking by a river in the daytime, you might see a water-loving creature. If you’re out at night, chances are you’ll spot a fairy or other nocturnal type. The game uses your phone’s GPS, accelerometer, and compass to orient itself—and then it uses the camera to overlay an image of a Pikachu or a Jigglypuff within your real-life surroundings.