"Why can't we do the stuff we see in movies, in the real world?"
According to Magic Leap founder Rony Abovitz, that's the question that helped kick off the company's nearly decade-long journey to build a better way of looking at the world.
The result is Magic Leap One, a new wearable computer (comprising a wireless remote and a headset connected via cable to a pocket-sized PC) designed to spatially map your environment and create the illusion that digital objects are sharing it with you.
Many devs have been following the project since Google invested half a billion dollars in 2014 and now, roughly four years and over $1 billion in additional investments later, the company has released the "Creator Edition" of Magic Leap One. It costs upwards of $2,300, and could potentially jumpstart a new wave of mixed-reality game design.
Last month Gamasutra visited Magic Leap's Florida headquarters to try out the Magic Leap One and chat with the folks who work with it every day, in the hopes of providing some useful insight into how the hardware works and what you need to know to get a head start on using it in your own projects.
Prior to that visit, the question of what Magic Leap is was still pretty hard to answer. The company had worked in comparative quiet for years, with only an occasional funding announcement or glowing hands-on article to remind people it existed.
But by the time I flew out last month, Magic Leap had begun broadcasting dev-focused livestreams in which staffers tried to showcase the One hardware alongside tech demos illustrating its Lumin OS and how a Magic Leap game might look. The demos were simplistic, a far cry from the promo videos the company put out years ago. After watching them on my way to the company's office, I was expecting a disappointing tech demo.
How it works
I was wrong. The Magic Leap One headset I messed around with for a few hours last month seems to match up pretty well with what the company was pitching years ago. Within the controlled conditions of the Magic Leap offices, moving no faster than a slow walk, the One performs remarkably well: it's comfortable to wear, intuitive to use, and most importantly, it didn't make me want to puke.
In fact, I felt no discomfort or dislocation when putting on or taking off the headset, which immediately sets it apart from just about every VR device I've ever tried. Much like Google's Glass headset, donning the One is like slipping a pair of translucent glasses (or in this case, goggles) over your eyes; unlike Glass, there's no sense that you're switching focus when your eye moves from looking at something in the real world to looking at something digital.
"I came out of biomedical engineering and the idea was, don't break the brain. Don't break the brain is the number one rule."
Instead, digital objects seem to coexist in the world with you. When I put on the One there was an initial scanning period whenever it detected a new space (if you change rooms, or just move things around a lot) during which it would direct me to look at different points around the environment for a few minutes to effectively map it (Magic Leap staffers call this "meshing"), and thereafter it could create the illusion that digital objects were coexisting with real ones remarkably well.
It seems to do this not by projecting anything into your eyes, or out into the world, but instead by projecting a signal onto the Magic Leap lenses (which are actually custom photonic wafers manufactured on-site) that then gets pulled into your eyes alongside the rest of the world's visual data.
What this looks like, in my experience, is a sort of translucent virtual world of audible digital objects that can interact with you in near-real time. However, much as with Microsoft's HoloLens, the objects are only visible within a virtual window in the center of your vision that's roughly the size of a Nintendo Switch held six inches in front of your nose.
"We know we can make it bigger," Abovitz said when I asked him about the limited field of view, which is noticeable but easily ignored. "We already have. In our early prototyping for Gen 2, we've already done some amazing things."
This sample indicator .gif (pulled from a Magic Leap dev doc) gives you an idea of the headset's digital FOV
The technical details of how Magic Leap works are still somewhat unclear to me, even after hours of using the headset (in controlled conditions) and talking to folks who work on it. It runs on its own operating system, the Lumin OS, designed to either multitask (via what Magic Leap calls "Landscape" apps: put a lo-fi hip-hop 24/7 YouTube video on the wall, put an email client on your desk, have the basketball game playing on your coffee table) or take over the user's entire environment for "Immersive" apps like games.
By now you can probably learn a lot about it by accessing the dev documentation and downloading the Lumin SDK via the company's Creator program. The company has also published a list of suggested dev machine specs, as well as a starter's guide to developing for Magic Leap.
But when I was there I hadn't looked at any of that, and the most common answer I received about how it works was actually an analogy: it's sort of like a cotton candy machine, if the cotton candy was your game.
"Obviously there's no actual cotton candy involved, but think about the sugar being the content you create," said Abovitz. "It gets processed here [pointing at a small black box on the headset called a spatial light modulator] so it's like this thin line, kind of like how a cotton candy machine sprays a little thin line of sugar. And then it gets pulled and stretched across this wafer [pointing at the lenses] and it comes out as this volumetric digital lightfield signal."
For your (human) eyes only
According to Abovitz, this means that what's displayed through the Magic Leap headset is in some sense only fully understandable by the human eye; if you position a camera to peer through the goggles, whatever you'll see won't be what's actually intended.
(Because of this, the various videos you see of Magic Leap One interfaces or experiences are produced by merging headset camera footage of the user's perspective with digital Magic Leap objects. A Magic Leap representative told Gamasutra that staff have tried to record videos with a camera pointing through the headset, but the result is a "very poor representation" of the real thing.)
"Our system is best understood by putting it on. We designed it so that the signal by itself makes no sense against any other sensor but your retina," said Abovitz. "We spent over a billion dollars optimizing this thing to hit the retina to induce all the normal flow. We didn't want to change biology, we wanted to actually use it as it is."
This is Abovitz's core argument for Magic Leap's value: according to him, it's a wearable device that shouldn't make people nauseous or uncomfortable because it becomes a part of the way you perceive the world, rather than overtaking it.
The goggles are branded Lightwear, the puck PC is a Lightpack, and there's a red version on display at Magic Leap
"Here's the part that's really hard to communicate: your brain is the coprocessor. We spent all of our money and eveything we're doing to send a signal for the human brain, not a camera CCD [charge-coupled device]. Not a monitor. Not anything else," said Abovitz. "The circuit only works when it's put on a person. You only know it works when it's on you. So it's not like I have a phone display that I can take a picture of to show people what it looks like. It's really the signal's interaction with the brain that allows all this to work."
It's a promising pitch for devs who have spent years fretting over the discomfort issues plaguing VR tech, and it was borne out by my hands-on experience. In the hour or two I spent trying game demos, creation tools, and utilities (think: web browsers and sports broadcasts) in the Magic Leap One, I never once felt dizzy or nauseous, even though I typically have a hard time in VR headsets.
According to Abovitz, this is the result of the company focusing from the jump on how to target the brain, rather than the eyes. He dismisses most modern 3D tech (from 3D movies to VR headsets) as being fundamentally flawed because it relies on stereoscopic trickery which fatigues the eyes and upsets the vestibular system, the spatial senses provided by, among other things, your inner ear.
"I came out of biomedical engineering and the idea was, don't break the brain. Don't break the brain is the number one rule. And going down that path, we took the longer and harder road," he said. "You could have built [stereoscopic 3D devices] in 1900, and people did. I call it a cockroach; it just keeps coming back."
Instead, Magic Leap is designed to change your brain's perception of the world by constantly monitoring your environment and injecting digital signals into it in such a way that your brain will interpret them as natural sights and sounds. The company likes to talk about it in terms of your brain visualizing a constant simulation of the world based on the lightfield, a term for the field of light (in both particle and wave form) you're immersed in at all times.
"Think of it like a GPU. It really is a GPU. So the world that you experience right now is built in your brain. The visual cortex is like a biologic VR. The world you experience, this is our thesis, is a biologic simulator that's fed by this tiny datastream that is a massive downsampling of this big analog stream [that is the real world]", Abovitz said. "And that's how all of us are. And part of the reason is that lightfield signal has been around for billions of years. The biology of humans goes back millions of years. So the brain has developed over millions of years of being exposed to this lightfield signal, and part of my thought is, don't break this. Do not screw this up."
Put more simply, the headset does three things in real time: it senses the world around it (including spatial environment data, gestures, ambient audio and voice commands), computes and displays digital illusions, and communicates wirelessly (via Bluetooth, Wi-Fi, and AT&T's network) with other devices.
No tags.
