Localization in The Machinery’s UI

JAN 14, 2019

One thing we never really addressed back when we were making the Bitsquid engine was localization of the tools. We supported localization of the games made in the engine, of course, but the engine tools themselves were only available in English. As both the number of tools and the number of frameworks used for the tools (Winforms, WPF, HTML, …) grew, the localization task became more daunting. When you’re writing an engine, you’re never short of stuff to do, and there always seemed to be something more pressing than addressing localization.

With The Machinery, we didn’t want to make the same sharp distinction between “tools” and “games” as we envision lots of use cases where the creative experience is taken all the way to the end user. So it was important to get this right from the start.

The scope of the problem

“Localization” in the context of game development can mean many different things. Localization of strings in the UI is probably what first comes to mind, but language can also be found in dialogue audio files and text baked into textures. Finally, there can be legal and cultural issues. For example, localization to Germany requires removing blood and gore. Plot or animation changes might also be necessary.

In the Bitsquid engine, all these things were all handled in the same generalized localization framework. I.e., everything was a “resource” (string resource, audio resource, texture resource, script resource) and we could localize these resources for different targets.

Treating everything the same has the advantage of being conceptually simple, but it also has disadvantages. Data-oriented design tells us to “look at the data” and the data for these different cases looks completely different. The UI text typically only amounts to a couple of KB for an entire game, whereas the audio dialogue can easily end up in the GB range. Solutions that work well in one case might not work well for the other.

For example, we can easily keep the UI text for all the supported languages permanently in memory and provide instant switching between different languages. For dialogue sounds, on the other hand, switching language might require unloading and loading megabytes of data — a significantly slower and more complicated procedure. If we force everything in under the same “localization abstraction”, we might force the UI text localization to become as slow and complicated as the other kinds of localization, even though that is completely unnecessary.

So in The Machinery, we treat these as separate problems and do not try to solve them all using the same mechanism. In this article, I will only look at the specific problem of localizing UI text.

Basic approach

The Machinery uses an immediate mode GUI approach (IMGUI), where the UI is built in code rather than in a UI editor. The advantages and disadvantages of this could be the topic of a whole another post. What it means, though, is that since all the UI text is in the C code, what we need for localization is a way of localizing strings in C code.

I.e. we need a function that given some key key that identifies a string in the UI, returns a translation of that string in the user’s currently selected interface language:

localize(key) → text

In The Machinery, this function is actually implemented as a macro TM_LOCALIZE(key), so I will use capital letters to refer to it from now on.

I’ll go into details on how TM_LOCALIZE is implemented later, but for now, just think of it as having some kind of table that provides the interface text for each key in all the supported languages and using the key to do a lookup in that table:

So far, I’ve been deliberately vague about what the key that we use to index into this table actually is. One common option is to use an integer identifier for the key and have a big header file that enumerates all the possible key values. Something like this:

#define TM_STRING_ID__FILE_MENU 0x0001
#define TM_STRING_ID__SAVE 0x0002
...

However, I find that this approach has a number of problems:

What can we do instead? Whenever we need unique identifiers but sequential enumeration is problematic, hashing is a good approach. So instead of:

TM_LOCALIZE(TM_STRING_ID__FILE_MENU)

We could do:

TM_LOCALIZE(hash("TM_STRING_ID__FILE_MENU"))

This gets rid of several of the pain points. We don’t need the big header file enumerating all the IDs. There is no risk of collision (if we hash into a big enough key space). And we don’t have any ghosts or holes any more (since we don’t have the big list anymore).

Isn’t hashing expensive compared to using a sequential ID? True, hashing has an extra cost, but in this case, it doesn’t really matter. In an IMGUI you should only be drawing the things that end up on the screen (you can always do a quick rect intersection test to determine if your control is visible.) If we assume that the hash is only called for things that are actually drawn, the cost doesn’t really matter, because it will be dwarfed by the cost of drawing the glyphs on the screen.

We can make the macro look a bit nicer too. We don’t really need the TM_STRING_ID__prefix anymore since these are no longer global defines that need to be unique, but just ordinary strings, and we can bake in the call to hash() into the TM_LOCALIZE macro itself, leaving us with:

TM_LOCALIZE("FILE_MENU")

There is still something I don’t like about this though: every time we want to add a new string to the UI we have to come up with a new identifier name for it, and naming is one of the two hard things in Computer Science. It is easy enough for things like the Filemenu, but what about a string like "Number of rendered triangles: %d". You end up with something ridiculous like:

TM_LOCALIZE("NUMBER_OF_RENDERED_TRIANGLES_WITH_INTEGER_FORMATTER")

There is also a bad disconnect here. To know what the