At the beginning of July 2021 I released You Will Be Found, a short, fast-paced and semi-online game. The game places you in an environment where you are not welcome and need to evade detection for as long as possible. Inevitably though you will be detected and then you will be actively hunted. The game has two modes: online and offline.

In the online mode the state of your game world is influenced by other recent players. The paths they have taken will make those sections of your world more hazardous just as you in turn make your path more hazardous for others. The game involved a number of technical challenges ranging from both the online side (with realms supporting hundreds of players); the visual feedback required; and, the need to balance for wildly different scenarios. In the sections that follow I’ll break down many of these areas and provide code snippets where possible.

<iframe title="You Will Be Found [Gameplay Trailer]" src="//www.youtube.com/embed/Pf_EdnT54G8?enablejsapi=1&amp;origin=https%3A%2F%2Fwww.gamedeveloper.com" height="360px" width="100%" data-testid="iframe" loading="lazy" scrolling="auto" class="optanon-category-C0004 " data-gtm-yt-inspected-91172384_163="true" id="891136033" data-gtm-yt-inspected-91172384_165="true" data-gtm-yt-inspected-113="true"></iframe>

Building a semi-online game

Before getting into the details the first question is likely “what do I mean by a ‘semi-online’ game?” You Will Be Found (YWBF) has both an ‘online’ and an ‘offline’ mode. In this case we’re talking exclusively about the online mode. In that mode a number of key things happen:

The end result is that in online mode you never directly interact with another player but their presence will make your game more challenging just as you will make their game more challenging.

Tracking a player's movement (Unity client stage)

The client side for sending updates to the server was straightforward. Most of the heavy lifting is done by the server. As the player is moving through the level the game client is continually keeping track of the player’s location. That location information is quantised into a grid (each grid cell is 3x3 m and corresponds to an individual block in the game). Alongside storing where the player has been the game also tracks how much the player has corrupted that location.

This set of ‘player tracks’ is represented as a map with the location (a Vector2Int) and corruption added (a float ranging from 0 to 1). This made it easy to update and to query. The map is reset every time data is sent to the server and then starts to accumulate again until the next upload to the server. The upload interval was an important decision here. Longer intervals would result in larger uploads which would also increase the peak workload on the server due to having more data to process. Smaller intervals would mean smaller upload sizes but also potentially create a high ambient/baseline load on the server. With some testing I settled on 30s as the synchronisation rate (for the client and the server). 

The amount of data needing to be sent would also vary depending on the length of the player path. The more a player has moved then the larger the amount of data which will need to be sent. As the game itself pushes the player to continually move it was likely the paths would tend to be fairly long. In the worst case scenario of a player continually sprinting they could cover roughly 450 metres (150 blocks). That would result in 150 entries that would need to be uploaded. 

While not a large amount of data it is enough to warrant consideration of how the information is being uploaded. The server side is built on similar approaches to ones I’ve used before where there is a layer of PHP between the client and a MySQL database. For previous games the size of data being transferred was quite small so I relied upon encoding the data within the URL (ie. using GET). That could have been made to work in this case but it would have been quite messy so I decided to use POST (ie. encode the data in the body of the request).

The process of sending POST data through Unity is fairly straightforward and the code below shows how this can be approached

IEnumerator ServerRequest_SendData(Dictionary<Vector2Int, float> playerTracks, List<Vector2Int> playerWilhelms)
{
    using(UnityWebRequest sendDataRequest = new UnityWebRequest(SendDataURL, "POST"))
    {
        ServerMessage_PlayerData playerData = new ServerMessage_PlayerData(playerTracks, playerWilhelms);

        byte[] encodedPlayerData = System.Text.Encoding.UTF8.GetBytes(JsonUtility.ToJson(playerData));

        sendDataRequest.useHttpContinue = false;
        sendDataRequest.uploadHandler = (UploadHandler)new UploadHandlerRaw(encodedPlayerData);
        sendDataRequest.downloadHandler = (DownloadHandler)new DownloadHandlerBuffer();
        sendDataRequest.SetRequestHeader("Content-Type", "application/json");
        sendDataRequest.timeout = 5;
        yield return sendDataRequest.SendWebRequest();

        // go offline if there was a failure
        if (sendDataRequest.result == UnityWebRequest.Result.ConnectionError || sendDataRequest.result == UnityWebRequest.Result.ProtocolError)
        {
            State = ELinkState.FailedToRegister;
        }
        else
        {
        }
    }

    yield return null;
}

Some key things with this:

An important thing to be mindful of here, and that can cause hard to diagnose issues, is to ensure that the URL you are using does not require any redirects. Information passed via GET (ie. in the URL) is preserved with a redirect. Information passed via POST is not preserved with a redirect. I mistakenly used http (rather than https) at first for the URL and this resulted in some of the data coming through (the GET passed data) but not the POST data.

Tracking a player's movement (PHP-based server stage)

On the PHP side reading the POST data (in particular JSON-based POST data) is handled differently to form data. As seen in the code below you need to read from a specific filestream and then you can convert that received data into objects which can then be processed.

// connect to the database
$databaseConnection = YWBFServer_Connect();
if ($databaseConnection == null)
    return;

// extract the player data
$jsonPlayerData = json_decode(file_get_contents('php://input'), true);

// check the structure of the data
if (!array_key_exists("Track_Locations", $jsonPlayerData) || 
    !array_key_exists("Track_Values", $jsonPlayerData)    || 
    !array_key_exists("PlayerWilhelms", $jsonPlayerData))
{
    echo json_encode(["Result" => "failure"]);
    return;
}

// extract the data
$trackLocations = $jsonPlayerData["Track_Locations"];
$trackValues    = $jsonPlayerData["Track_Values"];
$playerWilhelms = $jsonPlayerData["PlayerWilhelms"];

// check that the track data matches
if (count($trackValues) != count($trackLocations))
{
    echo json_encode(["Result" => "failure"]);
    return;
}

Once the data was received and extracted it’s necessary to then pass it on to the server. It wasn’t possible to directly pass an array or dictionary to the database so a different approach was needed. There were options such as constructing a comma separated string in PHP and then deconstructing that on the database side to then process the data. That would have involved a lot of messy, and error prone, data manipulation to transform the data into the (and out of) the comma separated format.

Instead, I opted for issuing multiple commands (one per entry in the received data). On the server side it maintains a table of queued updates and has a stored procedure to add entries to that table. The PHP code reads every entry in the uploaded data and executes the stored procedure to enqueue each update. That resulted in significantly simpler code and that simplicity also resulted in code with solid performance.

Tracking a player's movement (MySQL database stage)

At this point the server now has a table of queued updates and also has access to a table per realm that maintains the state of that realm. The server needed to periodically process (and then clean out) all of the queued updates. That actual logic for the update was quite straightforward, in pseudocode it would (for each location) look like this:

The Cell Delta is the change to apply to a location (ie. comes from the queued updates). The Min/Max Cell Value constrained the cell values to be within a set range.

While the maths is straightforward there may be multiple entries for the same cell (due to multiple players traversing the same area). Initially, I went for a very simple approach where it would process each queued update individually and apply that to the table. Initially, I also entirely forgot about setting up indices for the table. With small sets of data it wasn’t noticeable but then when I tested with larger chunks of data the update time was very poor. A few hundred updates (equivalent to 5-10 players) would take several seconds. Which was somewhat less than a viable level of performance.

I wanted to comfortably support hundreds of players which meant I needed it to easily handle very large (100,000+) updates in a couple of seconds at most. With the current setup that would take far too long. I used two approaches to improve the performance. The first part of the solution was to collapse multiple updates for the same location into a single entry. The solution for that was to create a temporary table in memory. MySQL allows for a temporary table to be created based on a query of another table. In this case the query extracted queued updates for the realm being processed and collapsed them so there was one entry per included location. It also, for security, excluded any data that was not from a known active player. That temporary table had fewer entries which in turn reduced the amount of work needing to be done to update the realm itself which sped up that process.

The second improvement was, unsurprisingly, to do what I should have done from the start and setup indices. I added an index to both the realm and the queued updates tables. In both cases the index was the cell location and it was configured to be generated automatically. Indices make a huge difference in how queries are processed, in particular ones where you are retrieving/updating specific rows. Without indices the database needs to search for each row. With indices it is able to look up the row (typically via a tree structure) rather than search. In this case the cell location was used to control which rows were updating so using that index provided a substantial improvement. Combined with the temporary table it was possible to process over 150,000 updates in under a second.

Now that the update process could run rapidly the final area to support was the client retrieving the latest copy of the realm. Similar to the update handling I did not want to have this place a high load on the server by generating it every time it is requested. Instead, what happens is that as part of the maintenance process which runs regularly for every realm it also generates and stores a realm snapshot. When a client requests the current realm map it is sent that snapshot which requires very little processing on the server side to pass on. 

With those pieces in place the client-server communications were working reliably and robustly. However, the server side also needed to repair/heal the cell values and to do so proportional to the number of players active (to try and keep the realm playable). Which leads on to the next key area: dynamic balancing.

Dynamic Balancing

During the development of the game a major concern and consideration was balancing the experience for the player. In particular with having both an online and an offline mode the potential for very divergent player experiences was high. The game needed to balance dynamically depending on the circumstances and that balancing needed to happen both on the client side and the server side to achieve the desired player experience.

Client Side Balancing

Regardless of whether the player is playing the offline or the online mode there are two key mechanics that they will run into: corruption and detection.

Th