Q&A: What devs should know about making games for Magic Leap

After years of speculation, Magic Leap's mysterious mixed-reality headset is now broadly available to developers.

Launched in six cities this week with an asking price of $2,295, the "Creator" edition of Magic Leap One may appeal to devs around the industry who are looking to blaze trails in mixed-reality game design.

It may also appear to share several weaknesses with the current crop of MR headsets: like Microsoft's HoloLens, for example, the One is expensive, still in active development, and only capable of displaying digital objects in a limited range within the wearer's field of view.

But unlike the HoloLens, or at least the HoloLens model I tried several years ago, the One's field of view seems large enough that clever devs will probably have no trouble making Magic Leap experiences feel immersive. When I visited Magic Leap HQ last month and demoed a One under controlled conditions, I was impressed by how well the headset meshes the real world with a virtual one, and how "real" its virtual objects can feel when devs take full advantage of the One's capabilities.

Of course, the demos I was put through are the result of years of work from folks who are intimately acquainted with Magic Leap's hardware. Anyone who buys a headset today and starts developing for Magic Leap next week has a long road ahead, one that's lacking in paths or guides. 

Context is key

With that in mind, I asked some of the leads at Magic Leap to share what they've learned about how to best design games for the One.

(By now you can also learn a lot more about developing for the One via Magic Leap's dev portal, which is filled with guides and code samples.)

As you might expect, they were all bullish about the headset's potential and eager to see more devs getting into Magic Leap. But I asked anyway because many of them are also veteran game devs, with a lot of experience working at triple-A companies like Ubisoft, Blizzard, and Electronic Arts. Their passion for the industry seems genuine, and on more than one occasion I was told that years of designing games for other paltforms gives you many skills you can apply towards developing for Magic Leap -- but you'll also need to start thinking within the limits of the hardware and the context of the real world.

"As a game developer working on this brand-new platform, it's sort of split in half,"  says Magic Leap Studios senior creative director Jeremy Vanhoozer, who previously worked at PopCap on Plants Vs. Zombies. "You have this half that's really recognizable, working in engines that you know, using a lot of the same language that you know, and figuring out how to create a lot of the same tools and experiences that you know.

"The other half is completely new. It's a brand-new platform, and something we've all had the chance to experience is working on software and hardware as they're being built, which I think is something developers are all too familiar with...but I also think there's opportunity," he continues. "So for me it's about taking a lot of the game development learnings that I had and then getting to use them in completely different ways...I feel comfortable in one sense, and completely exposed in another."

A promo shot of Magic Leap One, worn by the person in the sick denim jacket 

Multiple times during my visit I heard some version of this refrain: developing for Magic Leap means developing for the real world, giving up some control of a player's space in return for immersion in it. 

"As much if not more than the technical challenges of MR are the design shifts. Like, just everything is fairly different, from a design side," says Magic Leap Interaction Lab director Brian Schwab, who previously worked at Blizzard on games like Hearthstone. "A lot of things that make a hell of a lot of sense for a screen don't make much sense when you have all the room in the world and a very contextual environment and the ability to put pixels exactly where you need them, right when you need them, and then they'll go away. It makes a lot of rules about things we used to do on a screen not applicable.

"That being said, there's a lot of stuff that still does drag along," he adds. "You can do a lot of the things that you can do in a video game, but you can do them in a much more bio-mechanically sound way, because we have the ability to know where you're looking. We have the ability to have your hands in view if you're in the right place."

In my experience, the Magic Leap One is pretty good at tracking your hands and arms if you're waving them around in front of your eyes. This can give you the sensation of "touching" digital objects, even if there's nothing there to actually touch, because they can react to your movements fast enough to seem believable. 

This means devs can definitely do things like luminous, sci-fi menus which glow and hover in space, or affix themselves to nearby surfaces. However, Magic Leap staffers warn that flashy UIs can feel ephemeral and confusing if you don't render them in a contextual and believable way.

'Bias towards the body, and comfortable interactions'

"It might be helpful for developers to imagine this not as a 3D display or a wearable display but just as a wearable. To bias towards the body, and comfortable interactions," said Savannah Niles, a senior member of Magic Leap's UX team. "Another best practice would be to just keep [the player's view] clear. It's a see-through clear display for a reason; draw as little to the display as possible, allow for people to experience the real world, and leverage that. And then another thing we say in here is 'try to design experiences that take advantage of the best in bits and atoms.' So use physical real-world metaphors where it's appropriate, have digital things have object integrity where it's meaningful, and then use digital models and digital interactions where it's appropriate."

Multiple Magic Leap staffers cautioned against relying too much on purely digital objects when designing mixed-reality experiences. Since the ability to blend digital objects into the real world is the platform's selling point, they were quick to suggest devs include the physical world in their experiences whenever it makes sense to do so.

If you're designing a button someone has to press to launch a game mode, for example, you're encouraged to render that button on a countertop or other surface so that the player's hand pushes through it and hits something physical. It's not a real button, of course, but the folks at Magic Leap say even that little bit of physical resistance provided by a real surface is important to keeping players immersed in your game.

<iframe title="Embedded content" src="https://giphy.com/embed/8hZGoE5K5wvID5ZyUC" height="100%px" width="100%" data-testid="iframe" loading="lazy" scrolling="auto" data-gtm-yt-inspected-91172384_163="true" data-gtm-yt-inspected-91172384_165="true" data-gtm-yt-inspected-113="true"></iframe>

An example (pulled from a sample project built with Unity) of how Magic Leap renders virtual objects

"If you've ever tried to hit a button in like mid-air, with some exceptions, it's pretty unsatisfying," says Niles. "But the passive haptic feedback of the real world really brings that experience to life."

Magic Leap devs also have to get used to the headset's range of possible inputs, from gestures to location to facial expressions, none of which easily replicate the condensed, abstract input of a gamepad. The One does come with its own controller, a wireless remote (called the Control) with 6DoF tracking that sports a trigger, buttons, and a touchpad.  But it's hard to see how this might match the potential input complexity afforded to devs designing for gamepads and keyboards, and Magic Leap staffers are quick to suggest devs focus on breadth, not complexity, when designing mixed-reality experience.

"The controller's a good one, because I think for anyone who comes from game development, especially on the gameplay side, I remember spending so much time mastering the pad. Because developers who are used to traditional video games have this one input," said Aleissia Laidacker, interaction director at Magic Leap and a former Ubisoft vet. "So much of it is focused on, you know the user is sitting and looking at a screen, and the only bit of interaction is how can I make this game controller become super challenging. How can I make it like 'oh my gosh I beat this level with the skill in my hands.' 

"But now, we have so many different inputs. We can use our eyes, our hands, our voice. I love that because it's much more natural input, and I think one of the shifts for game development is, it's going to move away from the mystery of inputs and how we interact with things, and much more towards the accessible side," she continued. 

"One of the things that we've been seeing a lot, especially with our early access partners, is that now instead of just focusing on challenging inputs, they're really more focused on the experience happening around the user. One of the things we're trying to push for a lot is trying to think about how you create experiences that get someone to move around the space around them, to get them to maybe reconnect with their living room or the house around them, and let it become their playground. We hear in video games that open-world games are the playgrounds, and now your playground can also be your living room or the house around you. So how can developers on this platform start utilizing the space around them, and thinking about how to play in it?"

Designing games around the bodies that occupy that space is also a key concern for MR devs. In my experience the Magic Leap One isn't great at dynamically responding to other people in the space with you, but it seems to do a solid job of tracking your own body and responding to it. Some of the folks at Magic Leap think this gives devs&nb