This is a repost of the article from http://www.kertgartner.com/virtual-cinematography-for-vr-trailers
Questions or Comments? Please send me a tweet or shoot me an email
After creating the Mixed Reality trailers for Fantastic Contraption and Job Simulator I wanted to dive a little bit deeper into virtual cinematography by filming an entire trailer inside of VR rather than mixing live action and virtual reality.
The idea of filming in-game avatars rather than the actual physical player on a green screen is something we did in a smaller scope back on the Mixed Reality Fantastic Contraption trailer. I wanted to expand on that idea and see what we could do filming a whole trailer this way with different focal lengths and camera moves like you would in a traditional live action shoot.
Mixed Reality trailers are awesome, but creating high quality mixed reality videos are incredibly time consuming, expensive, and logistically complex from both a technical and creative standpoint. Filming an avatar in third person like you would in a normal live action shoot has many advantages and in many cases may be a better overall solution for showcasing your VR game/project.
<iframe title="Space Pirate Trainer - Official Trailer" src="//www.youtube.com/embed/xfa9Jqm34b0?enablejsapi=1&origin=https%3A%2F%2Fwww.gamedeveloper.com" height="100%" width="100%" data-testid="iframe" loading="lazy" scrolling="auto" class="optanon-category-C0004 ot-vscat-C0004 " data-gtm-yt-inspected-91172384_163="true" id="396828713" data-gtm-yt-inspected-91172384_165="true" data-gtm-yt-inspected-113="true"></iframe><iframe title="Fantastic Contraption Oculus Trailer" src="//www.youtube.com/embed/N3Ok7GYb4js?enablejsapi=1&origin=https%3A%2F%2Fwww.gamedeveloper.com" height="100%" width="100%" data-testid="iframe" loading="lazy" scrolling="auto" data-gtm-yt-inspected-91172384_163="true" id="777267737" class="optanon-category-C0004 ot-vscat-C0004 " data-gtm-yt-inspected-91172384_165="true" data-gtm-yt-inspected-113="true"></iframe>
WHY FIRST PERSON VR FOOTAGE (GENERALLY) SUCKS
There's a reason movies aren't shot in first person. Our brains react emotionally to seeing footage of actors performing in front of us in a way that's presented cinematically. We don't react emotionally on nearly the same level when we're literally seeing through another person's eyes. There's so much nuance that's lost about the performer when we can't see their body or they fit into their world. Virtual Reality is no different, but unfortunately first person footage is generally what is used to explain and promote VR games. This footage falls flat emotionally for many technical and creative reasons.
By default, first person footage is what is output from the current gen of VR hardware/software so most people just stop there and use that footage since it's easy to generate. Additionally, most raw video output for first person VR footage is un-smoothed which makes it hard to watch. Your brain and eyes naturally cancel out the hundreds of little micro adjustments and movements your head is making so your head appears to be moving smoothly to you. But take a look at most first person VR footage on Youtube, and it's a shaky mess. Your head moves a lot more than you realize, and the 2D footage feels unfocused and hard to watch.
If you absolutely need to use first person footage, make sure you create a custom HMD camera that takes the output of the HMD and smooths it so that it's easier to watch. We did this for the Job Simulator trailer it it was very successful.
Ideally, you need to see the player inside of their virtual environment to make the biggest visual impact and connect emotionally to the viewer. Whether that's through creating a mixed reality trailer or filming an in-game avatar depends on the needs and budget of the project.
Filming VR from a third person perspective allows you to create dynamic and interesting looking shots that capture the essence of what it feels like to be in the game. Space Pirate Trainer is a perfect example. What the game looks like from a first person perspective is not what it feelslike to actually play the game. Take a look at the different examples below. The third person footage feels kinetic, cinematic, dynamic and you emotionally connect to it. The first person footage feels cluttered, confusing, and visually uninteresting. Filming a player in third person through mixed reality, or filming an in-game avatar solves many of these communication problems.
THIRD PERSON CAMERA
This footage is more kinetic, cinematic and dynamic. The action is framed in a cinematic way, the camera follows the player's reaction to the incoming droid and the camera can anticipate the movement over to the large droid creating an interesting and compelling shot.
FIRST PERSON CAMERA
It's hard to tell what's going on in this shot. The gun and shield obscure many of the main aspects of it. It's not cinematic, it's not kinetic, and doesn't feel interesting or exciting to watch.
Here's an example from the Fantastic Contraption Oculus Touch trailer. Which shot feels more compelling and dynamic? Which shot gives you a better sense of what is happening in the game?
THIRD PERSON CAMERA
This shot feels dynamic and interesting to watch. The camera starts out wide, then pans in with precise timing on the player grabbing a wheel from Neko to illustrate how she acts as the toolbox in game. We then pan past her and focus in on the contraption the player is building following the motion of his hand.
FIRST PERSON CAMERA
This shot feels flat, boring and above all kind of vomit inducing because of the angle of the players head. Head tilt inside of the HMD feels completely natural to the player, but the resulting 2D footage feels wrong. Neko also gets lost in the green grass, and it's not clear what the player is grabbing from.
Here's another example from Space Pirate Trainer. This game is all about moving around in the environment to shoot/duck/dodge incomming bullets being fired at you from all angles. It's basically impossible to convey how that works through first person footage since you have no context for how the player is moving within their environment.
THIRD PERSON CAMERA
This footage clearly conveys what's happening in-game. The player is being shot at by a swarm of droids, and he's dodging out of the way in slow motion.
FIRST PERSON CAMERA
It's very difficult to tell what the player is physically doing in this shot. He's dodging the bullets by walking to the right in his VR play space, but that's not being communicated visually very well. All it looks like is the droids are shooting to the left and missing the player.
WHY FILM AN IN-GAME AVATAR INSTEAD OF FILMING MIXED REALITY
Mixed reality trailers are awesome and communicate what it's like to be in VR. But trying to create a professional looking mixed reality trailer requires a lot of money, time and resources. Though Owlchemy Labs Mixed reality tech should bring the complexity down once their stuff makes into more people's hands.
Filming an in-game avatar has many advantages:
It's more cost effective: All you really need is a wired/wireless third vive controller, a gimbal stabilizer (or even a cheap steady cam like the one I used pictured here), and some developer time to create an appropriately styled in-game avatar for the game on top of the extra time you need to film the trailer in VR. You eliminate all the overhead of having to buy/rent/create/light a green-screen setup and the crew necessary to operate this scale of operation and deal with the post production headaches to put them together.
You have more time to experiment: When you're on set with 5-10 people waiting for decisions - time is money. When you can shoot a trailer with less people over more days, you have more time to experiment and try different things. We shot for 3 days each for the Fantastic Contraption and Space Pirate Trainer Trailer to get all the shots we needed.
Change direction on the fly
When you're shooting on set there's very little room for experimentation since you're under the gun in terms of time and what needs to get accomplished during the day. Since we didn't have the time pressure associated with that, we were able to take breaks to check our footage and see if what we were shooting was working - and if not - adapt on the fly and change course.
It's amazing that what feels correct while in the HMD playing the game looks wrong when you're filming from a third person perspective. Blocking is important and where the player is positioned in the world can completely change the look and readability of the shot. Small details like the angle of your wrist can have a large influence on how natural the Avatar looks. Some movements might jank up the IK so knowing how and where to place your body is important - so checking footage so both the person playing and person filming are on the same page really helps with this constant iteration.
Actor performance/fatigue: This is something that most people don't think about, but it's tiring to play VR. Especially a game like Space Pirate Trainer. Getting consistent/exciting performances from your player is very difficult when you're dealing with a game with so many random elements and it's exhausting after about 30-45 minutes of play. Not being under the time constraints and pressure of being on-set allows more time to breathe, take breaks and feel more relaxed about the whole process.
It's incredibly surprising how much emotion comes through the avatar from just those three data points (the head and two hands). If your player is feeling tired or just not emotionally into what they're doing, it's amazing how obvious that is when you're filming them. When you're filming a live action person on set for a mixed reality trailer, that exhaustion is compounded about 10 fold and requires professional actors to keep up a level of performance for hours at a time.
Your character fits perfectly in the in-game world: It would have been extremely time consuming and expensive to create a costume similar to the in-game avatar in Space Pirate Trainer to make a player feel like they're part of the game's world. If we were to have shot this trailer with mixed reality, It would have felt out of place to have a player in their jeans/t-shirt facing off against all of these futuristic robots. We would have wanted to dress them up in costume, but even then, we wouldn't have been able to get close to what they were more easily able to achieve in-game since compositing a live player into a game's environment will never look as good as if they're part of the game world to begin with.
Variety is key: Seeing your game from one perspective (the player's) gets boring fast. To create a trailer or video that's compelling to watch you need to have multiple shots from different angles to give the viewer a sense of what they're looking at and how what they're seeing fits into the overall world of the game. That's basically impossible if you're limited to first person footage from the player's perspective. For a game like Space Pirate Trainer, the action is so fast and intense, that seeing it from different perspectives is the only way to properly visually communicate what is going on.
SENSE OF SCALE IN FANTASTIC CONTRAPTION
One of the things we wanted to get across was the new "seated scale" that's part of the Oculus touch version of Fantastic Contraption. Seeing the game though first person can be a bit deceiving and getting a real sense for how large or small the contraptions are isn't always obvious. Filming avatars in third person was a simple way to give the viewer a real sense of how large they were in the context of the rest of the world.
THIRD PERSON CAMERA
This shot clearly puts the size of the contraption in context. You can see how small it is compared to the player and the camera angle focuses your attention at the top of the shot where the contraption is travelling. The emotion of the player is also very clearly conveyed when the contraption hits the goal.
FIRST PERSON CAMERA
Until the controller comes into view at the end, it's almost impossible to know how large that contraption truly is in this shot. It could be very large and far away for all we know. The head tilt and motion also makes this shot hard to watch and it's impossible to read the emotion of the player based even though it's the exact same shot.
SHOOTING THE PSVR AND OCULUS VERSIONS AT THE SAME TIME
One small wrinkle that we had to deal with for Fantastic Contraption specifically is we wanted to use this trailer for both the upcoming PSVR version and the Oculus. So, how do you shoot two trailers at once when the in-game controls are specific to each platform? Lindsay Jorgensen came up with an incredible solution: We would split the screen into quads and place render only Oculus controllers in first person and 3rd person quads and PSVR controllers in the other two quads.
This worked amazingly well, and about 80% of the shots in the trailer just required a simple footage swap to change over to the PSVR version. We had to remove any references to room scale shots in the PSVR version of the trailer so the only major section that needed to be completely
