Scaling Dedicated Game Servers with Kubernetes: Part 3 – Scaling Up Nodes

Originally posted on compoundtheory.com.

This is part three of a fivefour-part series on scaling game servers with Kubernetes.

In the previous two posts we looked at hosting dedicated game servers on Kubernetes and measuring and limiting their memory and CPU resources. In this instalment we look at how we can use the CPU information from the previous post to determine when we need to scale up our Kubernetes cluster because we’ve run out of room for more game servers as our player base increases.

Separating Apps and Game Servers

The first step we should make before starting to write code to increase the size of the Kubernetes cluster, is to separate our applications -- such as match makers, the game server controllers, and the soon-to-be-written node scaler -- onto different nodes in the cluster than where the game servers would be running. This has several benefits:

Kubernetes makes setting up a heterogenous cluster relatively straightforward and gives us the tools to specify where Pods are scheduled within the cluster - via the power of Node Selectors on our Pods. It’s worth noting that that there is also a more sophisticated Node Affinity feature in beta, but we don’t need it for this example, so we’ll ignore its extra complexity for now. To get started, we need to assign labels (a set of key-value pairs) to the nodes in our cluster. This is exactly the same as you would have seen if you’ve ever created Pods with Deployments and exposed them with Services, but applied to nodes instead. I’m using Google Cloud Platform’s Container Engine, and it uses Node Pools to apply labels to nodes in the cluster as they are created and set up heterogenous clusters - but you can also do similar things on other cloud providers, as well as directly through the Kubernetes API or the command line client. In this example, I added the labels role:apps and role:game-server to the appropriate nodes in my cluster. We can then add a nodeSelector option to our Kubernetes configurations to control which nodes in the cluster Pods are scheduled onto.

For example, here is the configuration for the matchmaker application, where you can see the nodeSelector set to role:apps to ensure it has container instances created only on the application nodes (those tagged with the “apps” role).

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: matchmaker
spec:
  replicas: 5
  template:
    metadata:
      labels:
        role: matchmaker-server
    spec:
      nodeSelector:
        role: apps # here is the node selector
      containers:
      - name: matchmaker
        image: gcr.io/soccer/matchmaker
        ports:
        - containerPort: 8080

By the same token, we can adjust the configuration from the previous article to make all the dedicated game server Pods schedule just on the machines we specifically designated for them, i.e. those tagged with role: game-server:

apiVersion: v1
kind: Pod
metadata:
  generateName: "game-"
spec:
  hostNetwork: true
  restartPolicy: Never
  nodeSelector:
    role: game-server # here is the node selector
  containers:
    - name: soccer-server
      image: gcr.io/soccer/soccer-server:0.1
      env:
        - name: SESSION_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
          limits:
            cpu: "0.1"

Note that in my sample code, I use the Kubernetes API to provide a configuration identical to the one above, but the yaml version is easier to understand, and it is the format we’ve been using throughout this series.

A Strategy for Scaling Up

Kubernetes on cloud providers tends to come with automated scaling capabilities, such as the Google Cloud Platform Cluster Autoscaler, but since they are generally built for stateless applications, and our dedicated game servers store the game simulation in memory, they won’t work in this case. However, with the tools that Kubernetes gives us, it’s not particularly difficult to build our own custom Kubernetes cluster autoscaler! Scaling up and down the nodes in a Kubernetes cluster probably makes more sense for a cloud environment, since we only want to pay for the resources that we need/use. If we were running in our own premises, it may make less sense to change the size of our Kubernetes cluster, and we could just run a large cluster(s) across all the machines we own and leave them at a static size, since adding and removing physical machines is far more onerous than on the Cloud and wouldn’t necessarily save us money since we own/lease the machines for much longer periods. There are multiple potential strategies for determining when you want to scale up the number of nodes in your cluster, but for this example we’ll keep things relatively simple:

Creating a Node Scaler

The node scaler essentially runs an event loop to carry out the strategy outlined above. Using Go in combination with the native Kubernetes Go client library makes this relatively straightforward to implement, as you can see below in the Start() function of my node scaler. Note that I’ve removed most of the error handling and other boilerplate to make the event loop clearer, but the original code is here if you are interested.

// Start the HTTP server on the given port
func (s *Server) Start() error {
        
        // Access Kubernetes and return a client
        s.cs, _ = kube.ClientSet()

        // ... there be more code here ... 
        
        // Use the K8s client's watcher channels to see game server events
        gw, _ := s.newGameWatcher()
        gw.start()

        // async loop around either the tick, or the event stream
        // and then scaleNodes() if either occur.
        go func() {
                log.Print("[Info][Start] Starting node scaling...")
                tick := time.Tick(s.tick)

                // ^^^ MAIN EVENT LOOP HERE ^^^
                for {
                        select {
                        case <-gw.events:
                                log.Print("[Info][Scaling] Received Event, Scaling...")
                                s.scaleNodes()                          
                        case <-tick:
                                log.Printf("[Info][Scaling] Tick of %#v, Scaling...", tick)
                                s.scaleNodes()
                        }
                }
        }()
      
        // Start the HTTP server
        return errors.Wrap(s.srv.ListenAndServe(), "Error starting server")
}

For those of you who aren’t as familiar with Go, let’s break this down a little bit: