[In this reprinted #altdevblogaday in-depth piece, BitSquid co-founder Niklas Frykholm looks at the different options developers have for playing video in their games, and their pros and cons.] So you want to play some video? Shouldn't be too hard, right? Just download some video playing library and call the play_video() function. Easy-peasy-lemon-squeezy. Well, you have to make sure that the video is encoded correctly, that the library works on all platforms and plays nice with your memory, file, sound and streaming abstractions, and that the audio and video doesn't desynchronize, which for some inexplicable reason seems to be a huge problem. But this is just technical stuff. We can deal with that. What is worse is that video playback is also a legal morass. There are literally thousands of broad patents covering different aspects of video decompression. If you want to do some video coding experiments of your own you will have to read, understand and memorize all these patents so that you can carefully tip-toe your code and algorithms around them. Of course, if you had a big enough pool of patents of your own you might not have to care as much, since if someone sued you, you could sue them right back with something from your own stockpile. Mutually assured destruction through lawyers. Ah, the wonderful world of software patents. So, creating your own solution is pretty much out of the question. You have to pick one of the existing alternatives and do the best you can with it. In this article I'm going to look at some different options and discuss the advantages and drawbacks of each one:
Just say no
Bink
Platform specific
H.264
WebM
There are other alternatives that didn't make it to this list, such as Dirac, Theora, and DivX. I've decided to focus on these five, since in my view H.264 is the best of the commercial formats and WebM the most promising of the "free" ones. An initial idea might be: Why not just do whatever it is VLC does? Everybody's favorite video player plays pretty much whatever you throw at it and is open source software. Unfortunately that doesn't work, for two reasons. First, VLC's code is a mix of GPL and LGPL stuff. Even if you just use the LGPL parts, you will run into trouble on platforms that don't support dynamic linking. Second, the VLC team doesn't really care about patents and just infringe away. You can probably not afford to do the same. (As a result, there is a very real threat that VLC might be sued out of existence.) A quick introduction Before we start looking at the alternatives I want to say something short about what a video file is, since there is some confusion in the matter, even among educated people. A video file has three main parts:
Video data (H.264, DivX, Theora, VP8, …)
Audio data (MP3, AAC, Vorbis, …)
A container format (Avi, Mkv, MP4, Ogg, …)
The container format is just a way of packing together the audio and video data in a single file, together with some additional information. The simplest possible container format would be to just concatenate the audio data to the video data and be done with it. But typically we want more functionality. We want to be able to stream the content, i. e. start playing it before we have downloaded the whole file, which means that audio and video data must be multiplexed. We also want to be able to quickly seek to specific time codes, so we may need an index for that. We might also want things like audio tracks in different languages, subtitling, commentary, DVD menus, etc. Container formats can become quite intricate once you start to add all this stuff. A common source of confusion is that the extension of a video file (.avi, .mkv, .mp4, .ogg) only tells you the container format, not the codecs used for the audio and video data in the container. So a video player may fail to play a file even though it understands the container format (because it doesn't understand what's inside it). Option 1: Just say no Who says there has to be video in a game? The alternative is to do all cut scenes, splash screens, logos, etc in-game and use the regular renderer for everything. As technology advances and real-time visuals come closer and closer in quality to offline renders, this becomes an increasingly attractive option. It also has a number of advantages:
You can re-use the in-game content.
Production is simpler. If you change something you don't have to re-render the entire movie.
You don't have to decide on resolution and framerate, everything is rendered at the user's settings.
You can dynamically adapt the content, for example dress the players in their customized gear.
Having everything be "in-game visuals" is good marketing.
If I was making a game I would do everything in-game. But I'm not, I'm making an engine. And I can't really tell my customers what they can and cannot do. The fact is that there are a number of legitimate reasons for using video:
Some scenes are too complex to be rendered in-game.
Producing videos can be simpler than making in-game content, since it is easier to outsource. Anybody can make a video, but only the core team can make in-game content and they may not have much time left on their hands.
Playing a video while streaming in content can be used to hide loading times. An in-game scene could be used in the same way, but a high-fidelity in-game scene might require too much memory, not leaving enough for the content that is streaming in.
As engine developers, it seems we should at least provide some way of playing video, even if we recommend
No tags.
