'AI and Games' is a crowdfunded YouTube series that explores research and applications of artificial intelligence in video games. You can support this work by visiting my Patreon page.
The Forza series is home to one of the most impressive AI implementations in modern gaming. Drivatars are AI-controlled racers designed to drive like real people, learning from their behaviour and reproducing it in other peoples games. Allowing players all over the world to face off against their friends and other players across the Xbox Network, without ever actually playing the game together. It's one of the longest-running applications of machine learning in the videogames industry. So let's crack open the bonnet and find out what makes this engine tick.
Whether you prefer the simulation of Turn 10's Forza Motorsport or the open-world anarchy of Forza Horizon from Playground Games, you'll come into contact with Drivatars: a portmanteau of the words Driver and Avatar. Drivatars are the product of a highly complicated system, inspired by an AI technique we call Player Modelling. The idea of player modelling is that by gathering enough data of how players play a given game, an AI system can begin to make intelligent decisions that reflect that knowledge. In the case of Drivatar: by understanding how you drive cars in Forza, it can then drive an AI car in the game as if it was you in the driving seat. If you play any entry of the Forza series offline, all of the other racers you come across are Drivatars. They're all AI drivers that attempt to clone the behaviour of real people.
But what makes Drivatar so special is that it has been in the Forza series since the very beginning, meaning it's been in ongoing development since the first entry of the Forza Motorsport series, on the original Xbox in 2005. Making it one of the longest-running machine learning applications in the video games industry.
What Does Drivatar Do?
What makes Drivatar so impressive, is that it does a whole lot more than reproduce actions it has seen the player make before. It's making intelligent assumptions about how you would race in situations you might never have tried before. Your friend might jump into Forza Horizon and race against your Drivatar on a track you've never experienced and driving a car you've never driven. Hence the Drivatar isn't copying exactly what you would do, but rather figuring out 'how would you drive this car in this situation?'. This is where Machine Learning, rather than traditional symbolic AI, really begins to shine.
If you consider a lot of the AI applications we see in this series, techniques such as Half-Life's Finite State Machines or the Behaviour Tree of Alien: Isolation's Xenomorph, they are chock full of rules and logic that tell the AI what to do in a given situation. If the player gets too close, have a Combine soldier melee attack or fall back into cover. If the alien saw you go into a locker, chase the player down and rip the door off. In either case, it requires programmers and designers to figure out what these rules are going to be, and then put them into the system. If there isn't a rule for a given situation, then the AI might look stupid or broken, given you expect it to do something in that situation, but the character doesn't have a response.
Now consider this in the context of a racing game. You want to make a turn into the bend by a certain degree, as well as apply a certain amount of gas. Okay, fine. But that's going to be based on our current direction and velocity, so you need to factor that in there, oh we also need to think about the weather, given that will influence the traction of the tyres, oh we also need to factor the weight of the vehicle, the types of tyres we're using, the horsepower of the car and - you know what I think you get the idea. If any one rule is missing, then the AI is not going to drive that car properly and you then need to factor this for every corner, of every track and every vehicle.
It is possible to achieve this with an approach that models the physics of the car and handles how it would follow the race ribbon. But this complexity increases the number of tracks, race conditions and vehicles are added. If we consider the most recent release, 2018's Forza Horizon 4: that game has close to 100 racing events and around 700 vehicles. Simply put, it's increasingly demanding to do this effectively before we even get to modelling the player's behaviour.
But, with machine learning, this is much more plausible. Why is this? Well, one of the things machine learning is very good at is identifying patterns in large complex datasets. And it can then generalise them, so it knows how to respond when it sees a particular pattern in the data. What does that mean for a racing AI? Well, instead of having to write thousands of different logic rules for different conditions, a machine learning system will recognise that a lot of these situations are *very* similar to each other, and as a result, it can say on a per-frame basis, what steering and gas should be applied across the myriad of similar but unique cases.
So let's get into it: we're going to explore how Drivatar works, but to do that, we need to talk about the one gameplay system both players and AI interact with: the physics simulation system.
Forza's Physics Simulation
When the Forza series was devised, it was focused on delivering an authentic and realistic simulation comparable to that of rival series Gran Turismo on the Playstation. Hence it has a highly complicated physics layer built into the game engine. This physics simulation was built by pouring over the likes of Bill and Doug Milliken's 'Race Car Vehicle Dynamics', the textbook for anyone seeking to understand the physics of driving. This allowed the development team to capture the physics of race car driving in their own rigid body simulation engine. In much the same way simulators used by the likes of MacLaren or Ferrari do when training F1 drivers.
But they don't stop there. The physics system is continually updated with each release of the series, and often the team at Turn 10 studios work with car manufacturers to get more accurate reproductions of how their cars handle in specific conditions. Ignoring the marketing schpiel and relying on the expertise of motorsport engineers, sometimes even contacting the manufacturers of car components themselves. In essence, the Forza physics simulation is one of the most complete and accurate race car simulators that exists in the world.
The physics system is hyper performant, to a point that while the games renderers have fluctuated between 60 and 30 frames per second depending on the target hardware, or the demands of Horizon's open-world environments, the physics simulator runs at 360 frames per second and has done since the very beginning. This system is so complex and reacts to the smallest of changes, that it actually has several layers built into it that act as buffers between a human or an AI as they attempt to drive a vehicle. This is why Forza Motorsport and Forza Horizon feel completely different from one another. There are layers of adaptation between the player and the physics system designed to separate the simulation-heavy approach of Motorsport from Horizon's more arcade-y vibe.
Plus there are layers on the player controller level, such the how triggers and sticks of an Xbox controller would map to the realities of driving a car. Plus there are many assists that real drivers would use: from the likes of automatic gear changes to using the ABS for braking, or the traction control to ensure the car doesn't skid out. It helps even the most novice players get a feel for how to race in a variety of conditions and in each game the player and customise these in the Assists menu.
Now before the Drivatar gets its hands on it, we need to talk about the AI controller layer. The AI controller is designed to know exactly how to drive a car in a given situation based on the current physics simulation. Whenever you race a track in Forza, you are presented with the racing line on-screen, this is the most efficient path you take for this immediate section of the track based on the physics simulation. The AI controller layer adopts this information to achieve the maximum utility of the vehicle on that stretch of road.
So for example, if you're driving a car around a particular bend, there is an AI layer that already knows what speed and steering is required to drive that car so it follows the race line as effectively as possible. This isn't a secret, and if you need evidence of this, you can turn on the Super Easy steering assist introduced in Forza Motorsport 7, that when combined with the ABS and Traction Control, pretty much drives the car for you. All you have to do is keep your foot on the gas.
So with all these layers in place, the last step is Drivatar. Drivatar will then tweak that AI controller layer to drive like a specific humans behaviour. So let's start, by winding the clock back to 2005 and the original Forza Motorsport.
The Original Drivatar
Given the novelty of what Drivatar represents, it required a unique insight into making it happen. Hence, while Forza Motorsport was developed at Turn 10 Studios, in Redmond, Washington in the United States, the original Drivatar was conceived and developed at Microsoft Research Cambridge, in the United Kingdom. This isn't the only time that Microsoft Research has had a significant impact on Xbox game development, with the likes of the Kinect peripheral and the TruSkill matchmaking systems built on the AI research conducted by their divisions.
The original Drivatar is quite interesting, given that it exploits a feature unique to the Xbox at that time: a hard drive. Unlike Nintendo's GameCube and Sony's Playstation 2, the original Xbox came with an 8GB hard disk installed and ready to go. An idea that became more commonplace with the next generation of consoles. So in the early Forza Motorsport titles 1 through 4, your Drivatar was trained on the hard drive of your console, meaning it only existed on your Xbox. Meanwhile, other drivers are based on existing pre-trained data from the developers at Turn 10 that was shipped on the disc. But as we'll see in a moment, each Drivatar is fundamentally a really small amount of data. And while they wouldn't be shared via Xbox Live, you could grab an Xbox Memory Unit and copy the Drivatar for use on other consoles.
The reason that Drivatars are so easy to transport is that they are what are known as artificial neural networks. Neural networks are a popular form of data inferencing inspired by the human brain, where you have sets of neurons - which are each simple individual processing units - that are connected to other neurons on subsequent layers. The idea being, that when data is entered at the input layer on the left-hand side, each neuron processes the information and passes it along to the next layer. This process continues through the layers very quickly and the output comes out the other side. The trick with a neural network is that - outside of the basic parameters of the neurons that are consistent across all versions of that network - the thing that separates one Drivatar from another is the weights on the connections.
Training a neural network to solve a given problem involves a process of modifying the weights of the connections. This is a long-form process, but when it is done, the resulting Drivatar is essentially a list of numbers that correspond to the weights on the connections of the network. So transferring those files is relatively easy, given the data being transmitted is more or less just the numbers for the connection weights.
Now the one problem is that neural networks typically only have one answer for one general scenario. You feed the input, - in this case a particular part of the race line - it will have one output: what the car should do in that scenario. But we want some variety, and that's where bayesian neural networks come in.
Bayesian Neural Networks apply the idea of Bayes theorem to a neural network. What this means is that any decision it can make is also based on the probability of its outcome. And that probability is going to be influenced by the frequency with which specific behaviour appears in the recorded training data. Bayesian neural networks are designed such that you don't just have one set of weights, you have a collection of different sets of weights attached to a probability distribution for that input. When you want an answer, you run it several times, using the different weight sets. In each case, you have an answer, but also a confidence value as to whether the network thinks that should be the answer to that problem.
In the context of Forza: if we've seen the car take this particular turn on the track across several laps. At each step along that race line, the Drivatar can suggest one or more options for the angle the car should turn and the gas it should apply. It then selects the option with the highest confidence. That's what the car should be doing at that point because it's the most similar to what we've seen the player do before.
But this was just the beginning of the Drivatar system, given it was heavily reliant on whatever data it could gleam from that training set. Meaning it had difficulty adapting to new tracks and generalising for scenarios it had not seen before. Plus it was limited by only being able to base its knowledge on the cars you had driven. The next step was to make it a continuous and ongoing form of learning: adapting and evolving after every race you enter in every game of Forza.
Drivatar v2.0
Starting with Forza Motorsport 5, a launch title for the Xbox One back in 2013, the Drivatar system received a significant upgrade. The most notable change is that instead of being trained and iterated upon locally on your hard drive, Drivatars are now hosted on the cloud as part of the Xbox Network. This means two big changes: first, in order for your Drivatar to learn, you need to maintain an online connection as data is uploaded to th
