Introduction

So, you want to make a rhythm game or you tried to make one, but the game elemtns and the music quickly became out of sync, and now you're not sure what to do. You've come to the right place. I've been playing rhythm games since high school, when I frequented the DDR machine at my local arcade. Today I'm always on the lookout for new takes on the genre, and with entries like Crypt of the Necrodancer or Bit.Trip.Runner, there's still a lot that can be done in rhythm gaming. I put some work into a few rhythm-based prototypes in Unity, ultimately devoting a month to creating Atomic Beats, a short rhythm/puzzle game. In this article I'll cover a few of the most useful coding techniques I learned in creating these games that were either not covered somewhere else, or I thought could be covered in more detail.

Firstly, I owe a huge debt of gratitude to Yu Chao's blog post 'Music Syncing in Rhythm Games'. Yu covered the core of syncing audio timing to the game engine in Unity, and made the source code for Boots-Cut available, which helped immensely in getting this project off the ground. You can take a look at his post for a quick guide to music syncing in Unity, but I'll be going over the core of what he outlines and more. My code derives heavily from both his article and Boots-Cut.

At the core of any rhythm game is timing. People are extremely sensitive to any deviations in rhythmic timing, so it's extremely important to make sure that any actions, movement, or input in a rhythm game is directly synced to the music. Unfortunately the traditional methods for tracking time in Unity, like Time.timeSinceLevelLoad and Time.time will quickly lose sync with any audio playing. Instead we'll access time according to the audio system using AudioSettings.dspTime, which relies on the actual number of samples the audio system has processed, and therefore will always remain in sync with the audio as it plays (This may not be the case with extremely long audio files as some sampling effects may come into play, but for normal length applications, it should work perfectly). This function will be the core of how we track song time, and we'll build our main class around it.

The Conductor class

The Conductor class is the main song managing class that the rest of our rhythm game will be built on. With it, we'll track the song position, and control any other synced actions. To track the song, we'll need a few variables:

//Song beats per minute
//This is determined by the song you're trying to sync up to
public float songBpm;

//The number of seconds for each song beat
public float secPerBeat;

//Current song position, in seconds
public float songPosition;

//Current song position, in beats
public float songPositionInBeats;

//How many seconds have passed since the song started
public float dspSongTime;

//an AudioSource attached to this GameObject that will play the music.
public AudioSource musicSource;

When the scene starts, we need to do a few calculations to determine some of the variables, and also record, for reference, the time when the audio began.

void Start()
{
    //Load the AudioSource attached to the Conductor GameObject
    musicSource = GetComponent<AudioSource>();

    //Calculate the number of seconds in each beat
    secPerBeat = 60f / songBpm;

    //Record the time when the music starts
    dspSongTime = (float)AudioSettings.dspTime;

    //Start the music
    musicSource.Play();
}

If you create an empty GameObject with this script attached, and add an Audio Source with a song and start the program, you can see the script will update with the time when the song started, but not much else will happen. You'll also need to manually enter the BPM of the music you're adding to the Audio Source.

With these values, we can now track the location of the song in real time as the game updates. We'll determine the song timing, first in seconds, then in beats. Beats is a significantly easier way to track the song as it let's us add actions and timing in time with the song, say on beats 1, 3, and 5.5, without having to calculate the seconds between beats each time. Add the following calculations to the Update() function of the Conductor:

void Update()
{
    //determine how many seconds since the song started
    songPosition = (float)(AudioSettings.dspTime - dspSongTime);

    //determine how many beats since the song started
    songPositionInBeats = songPosition / secPerBeat;
}

this gives us the difference between the current time, according to the audio system, and the time when the song started, which amounts to the total number of seconds the song has been playing, which we store in the variable songPosition.

Note that while music counting typically starts at 1, with a beat of 1-2-3-4-etc., songPositionInBeats begins at 0 and increases from there, so the third beat of a song will occur when songPositionInBeats is equal to 2.0, not 3.0.

At this point, if you wanted to make a traditional Dance Dance Revolution style game, you could spawn notes according to the beat you wanted them to be pressed, interpolating  their position towards a trigger line, then record the songPositionInBeats when a key is pressed and compare it to the intended beat of that note.Yu Chao goes through an example of to set that up in his blog here. Instead of repeating that though, I'll cover a few other potentially useful techniques that can be built on top of the conductor class that I used in building Atomic Beats.

Adjusting for Starting Beat

If you're creating your own music for your rhythm game, it's easy enough to make sure that the first beat starts exactly when the music begins, which will make the Conductor's songPositionInBeats align correctly to the song, as long as the BPM is entered correctly.

However, if you're using pre-existing music, there's a good chance that there's a small period of silence before the song starts. Without accounting for this, Conductor's songPositionInBeats will think the first beat occurred when the clip began playing rather when the first beat occurs. Anything you have aligned to subsequent beat numbers will no longer be synced to the music as the game plays.

To fix this, we can add in a variable to account for the offset. In the Conductor class, add the following:

//The offset to the first beat of the song in seconds
public float firstBeatOffset;

In Update(), songPosition changes from:

songPosition = (float)(AudioSettings.dspTime - dspSongTime);

to
songPosition = (float)(AudioSettings.dspTime - dspSongTime - firstBeatOffset);

Now songPosition will calculate the song position correctly relative to when the first beat occurs. You will have to manually enter the offset to the first beat however, as this will be unique to each song file. There is also a short window during that offset where songPosition will be negative. This may not affect your game, but some code that relies on songPosition or songPositionInBeatss may not be able to process negative numbers during this time.

Looping

If you have a song that runs from start to finish, just using the above Conductor class is sufficient for tracking how far through a song you are, but if you have a short track that loops around and you want to work within that loop, it's necessary to build loops into your conductor.

With a perfectly looping clip (for instance if the beat is 120bpm, and the clip you want to loop is 4 beats long, it will have to be exactly 8.0 seconds long at 2.0 seconds per beat) loaded into the Conductor Audio Source, check the loop box. Conductor will work the same way as before, providing the total time since the looping clip was first started as the songPosition. To determine the position in the loop, we'll have to provide a way for Conductor to know how many beats are in one loop, and how many loops have been completed. Add the following variables to the Conductor class:

//the number of beats in each loop
public float beatsPerLoop;

//the total number of loops completed since the looping clip first started
public int completedLoops = 0;

//The current position of the song within the loop in beats.
public float loopPositionInBeats;

Now every time the SongPositionInBeats is updated, we can also update the loop position in Update()

//calculate the loop position
    if (songPositionInBeats >= (completedLoops + 1) * beatsPerLoop)
        completedLoops++;
    loopPositionInBeats = songPositionInBeats - completedLoops * beatsPerLoop;

This will give you a marker for how many beats through the loop you are with loopPositionInBeats, which will be useful for a lot of other synced items. Don't forget to enter the number of beats per loop on the Conductor GameObject.

Something else to be careful with here, again, is the counting of beats. Music always begins at 1, so a 4 beat measure goes 1-2-3-4-, while in this class, loopPositionInBeats startss at 0.0 and loops at 4.0. As a result of this, the exact middle of a loop, which would be 3 when counting by music beats, would occur at a loopPositionInBeats value of 2.0. It's possible to modify loopPositionInBeats to account for this, but will carry through to all other calculations, so be careful how you insert notes.

It will also be useful to add two more things to the Conductor class for the remaining tools. The first is an analog version of LoopPositionInBeats, called LoopPositionInAnalog, which measures the location in the loop between 0 and 1.0. The second is an instance of the Conductor class, so it can easily be called from other classes. Add the following variables to the Conductor class:

//The current relative position of the song within the loop measured between 0 and 1.
public float loopPositionInAnalog;

//Conductor instance
public static Conductor instance;

In the Awake() function add:

void Awake()
{
    instance = this;
}

and in the Update() function add:

loopPositionInAnalog = loopPositionInBeats / beatsPerLoop;

Synced rotation

It can be really useful to have movement or rotation that is synced to the beat in order to ensure elements are in the correct location. In Atomic beats I used this to dynamically rotate the notes around a central axis. They were initially placed around a circle according to their beat within the loop, and then the entire playing area is rotated so that notes would align with the trigger line at their intended beat.

To achieve this, create a new script called SyncedRotation and attach it to a GameObject you want to rotate. In the Update() function of SyncedRotation add: