One-button source code builds

Niklas Gray (aka Niklas Frykholm) is the co-creator of the Bitsquid game engine, acquired by Autodesk in 2014 and rebranded as Stingray. He is currently blogging and working new technology to help developers at Our Machinery.

Something I’ve been wanting to do for a long time is to create a one-button setup and build system. I.e., on a fresh machine, on any supported platform, it should be possible to type a single command that will first fetch, install and setup everything that is needed to build, and then perform the build and print the result.

Why? Mainly to make the source code more friendly to new users. Complicated, or even worse, incomplete build instructions are not a nice way to get started with a project. A user faced with something like:

may just give up, unless they are already deeply invested in the project.

Building something, running it and playing with it is the first step to understanding how it works. And we want to encourage curious people to experiment with our stuff.

Short and simple build instructions are good, but having a single button that you can just “press to build” is even better.

Programs that do things are always better than instructions that tell you how to do things. Having clang-format format you code is better than having a document with source formatting guidelines. Similarly, a tool that builds your code is better than a document with instructions on how to build your code. Documents can be vague and ambiguous whereas executable code is always precise. Documents easily get out-of-date, whereas if the executable gets out of date and no longer does the right thing, it is easily discovered. Executables can be put into pipelines, enabling further automatization and they free the people from rote memorization of rules and guidelines, so that their minds can be used solely for programming, the way they were intended to.

In the past, there was always something that prevented me from achieving full automatization — some legacy code that was just too complicated to rewrite. But that’s the nice thing about starting from scratch with Our Machinery. We get a chance to revisit old hang-ups and do them right from the beginning.

Picking a language

The first question is what language to use for our one-button build script.

In the past I’ve used Ruby for similar scripting tasks, but Ruby doesn’t come pre-installed on all platforms that we support; If you have to install Ruby before you can run the build script then it isn’t really a one-button solution. In addition, Ruby easily runs into version problems and gem troubles and I’m just not as fond of the language as I used to be.

We could use the built-in scripting environments (bash on OS X, .bat files on Windows), but that would mean having to maintain multiple scripts and also, those languages are kind of horrible to program in.

Something I’ve discovered when working on large projects before is that you really need to watch out for language proliferation, or before you know it your code will be a mix of C++, Lua, C#, JavaScript, CMAKE, Ruby, Python, Go and Rust. This creates all kinds of inefficiencies. It raises the barrier of entry for anyone wanting to understand the source code. The same routines will often have to be implemented multiple times in different languages and data has to be marshalled back and forth between them. So even though some time might be saved initially by using a scripting language instead of C for some administrative task, in the long run we believe it will be a net loss.

At Our Machinery we have taken the drastic decision to use C and (simple) C++ for everything. (As you know from my last blog post, we like drastic decisions.) So tools, helpers, tests and experiments will all be written in C, using our foundation APIs which provide platform independent utilities for memory allocation, networking, file system access, etc.

The advantage of this is that to understand the code, you only need to know one language and everything can be debugged together in the same debugger. In addition, it is a good litmus test for our foundation libraries. If we can’t easily write our various little tools and snippets on top of these libraries, then something is clearly wrong. Eating your own dog food is the best way of making sure that it tastes good.

Following this rule, our build script is just a simple one-file C program built on top of the foundation libraries. It compiles to a platform specific executable which we call tmbuild. We distribute these executables together with the source code and to build the source you just need to run tmbuild in the source code directory. Currently, it has the following options for customizing the build:

Usage: tmbuild [OPTION]...

    -h
    --help
        Display this help and exit.

    -t <tool>
    --build-tool <tool>
        Specifies the tool to use for building [gmake, xcode4, vs2017].
        If not specified, the platform default is used.

    -c <config>
    --config <config>
        Specifies the configuration to build [Debug, Release]. Default is
        Debug.

We expect to add more options in the future as we support more kinds of builds.

Operation of tmbuild

To build the project, tmbuild goes through these steps:

The dependencies are all the third-party libraries and binaries that we need in order to build the project in the current configuration. For example premake5 is a dependency, because we use it to generate solution files for the build tools.

The dependencies are specified in a JSON file called packages.json , which looks like this:

{
    "premake-osx": {
        "build-platforms": ["osx"],
        "lib": "premake-5.0.0-alpha11-macosx",
        "role": "premake5"
    },
    ...
}

The build tool has the concept of a build-platform (the platform we use when we run the build tool) and a target-platform (the platform that we are targeting the build for). Only the dependencies that are needed for the current build and target platforms are installed.

Each version of each dependency is identified by a unique string, such as premake-5.0.0-alpha11-macosx . To install the dependency we simply download premake-5.0.0-alpha11-macosx.zip from the web server where we host the libraries and unzip it into the library directory. The library directory is ./lib by default, but can be configured with a TM_LIB_DIR environment variable.

We use our own HTTP API from the foundation for downloading the files. Since we don’t have zip file support in our