By video game composer Winifred Phillips | Contact | Follow
Welcome back to this three article series that's bringing together the ideas that were discussed in five different GDC 2017 audio talks about interactive music! These five speakers explored discoveries they'd made while creating interactivity in the music of their own game projects. We're looking at these ideas side-by-side to broaden our viewpoint and gain a sense of the "bigger picture" when it comes to the leading-edge thinking for music interactivity in games. We've been looking at five interactive music systems discussed in these five GDC 2017 presentations:
Always Be Composing: The Flexible Music System of 'Plants vs. Zombies Heroes' – Speaker: Becky Allen, Audio Director
Different Approaches to Game Music (Audio Bootcamp XVI) – Speaker: Leonard J. Paul, Educator
Epic AND Interactive Music in 'Final Fantasy XV' – Speaker: Sho Iwamoto, Audio Programmer
Interactive Music Approaches (Audio Bootcamp XVI) – Speaker: Steve Green, Sound Designer
The Sound of 'No Man's Sky' – Speaker: Paul Weir, Audio Director
In the first article, we examined the basic nature of these interactive systems. In the second article, we contemplated why those systems were used, with some of the inherent pros and cons of each system discussed in turn. So now, let's get into the nitty gritty of tools and tips for working with such interactive music systems. If you haven't read parts one and two of this series, please go do so now and then come back:
Ready? Great! Here we go!
Tools and tips
To implement his interactive music system for the ocean exploration game ABZU, sound designer Steve Green relied on the popular Wwise middleware application in order to best facilitate transitions from one piece of music to another. "This idea of changing music on the fly rather than bringing instruments in and out was a good example of how to do ABZU," Green says. In order to implement these transitions, Green relied primarily on simple horizontal resequencing, with some limited vertical layering to ease transitions. 
At times, synchronization points were placed (usually at the beginning of each measure) in order to enable the music system to switch from one track to another. Sometimes, this approach was too abrupt. "Transitionary pieces are basically to help two tracks that are just not going to flow well together," Green explains.
For instance, when the player is traversing a particular portion of a level (i.e. stage) the audio engine is able to keep track of the player's progress."We used a parameter that would gauge how far along you are in this stage of the level," Green says. "Once you reach the 90% range, (the audio engine) would call a transition." This musical transition would play simultaneously with the previous track, allowing the previous track to fade out and facilitating either a more gentle culmination into silence, or a more graceful segue to another piece of music.
During educator Leonard J. Paul's GDC presentation, he discussed his work on the audio team of the Vessel platforming game, but he also shared insights into several other projects from his repertoire, along with lots of general advice that's worth considering. "When working in games, of course, you've got to think about code," Paul says. "You've got to think about RAM – how much space this is all gonna to fit in – streams, if you’re streaming. You’ve got to think about what your RAM window size is and how fast you can get information in and out – how many streams you can do at once, what kind of compression you’re going to use," 
Paul continues, listing areas of technical concern for the game audio expert to think about when working on a project like Vessel. "Because if you don't," he warns, "then you’re going to run into some issues."
Paul also mentions some of the more famous software tools available to audio pros, including Wwise, FMOD Studio, Fabric, and Pure Data, along with a number of general words of advice. "Technical tips – use spreadsheets, be organized, use tools to refine your process, embrace prototyping," Paul urges. "Do audio sketches and get this stuff done early."
In his presentation, Leonard J. Paul brought up a good point about the importance of sketches and prototypes in the workflow of a video game composer. Most game music composers have gone through this type of iterative process. For instance, for the SimAnimals game from Electronic Arts, I composed lots of sketches and prototypes before settling on the ultimate style of the game. The audio team had many varying ideas about what the music should sound like, so the initial brainstorming process involved quite a bit of trial and error. It was a game about heartwarming animal relationships, so should the music feel old-fashioned and lyrical? On the other hand, the game focused on strategy mechanics, so should the music feel more puzzle-like and intellectual? Also, the game included darker underpinnings associated with suffering and dissolution, so should the music stick with a light and airy feel, or should it weave dark and sinister textures into the mix? I tried tons of permutations on all these ideas until I hit upon the right balance of elements, and in the end my music for SimAnimals took me in unexpectedly challenging directions. The style couldn't have been achieved without the process of trial and error that characterized my early efforts at the beginning of music composition. As an example, I'm including here two videos showing the difference between the brightly cheerful main theme music of the game, and one of several grim and dissonant variations on the same melodic theme: |
<iframe title="Embedded content" src="//www.youtube.com/embed/D2e43w4XWUo?rel=0&enablejsapi=1&origin=https%3A%2F%2Fwww.gamedeveloper.com" height="100%" width="100%" data-testid="iframe" loading="lazy" scrolling="auto" data-gtm-yt-inspected-91172384_163="true" id="616736106" class="optanon-category-C0004 ot-vscat-C0004 " data-gtm-yt-inspected-91172384_165="true" data-gtm-yt-inspected-113="true"></iframe>
For Final Fantasy XV, audio programmer Sho Iwamoto created an audio engine he dubbed MAGI (Music API for Gaming Interaction). While the audio engine accomplished tasks similar to the functionality of other applications such as Wwise and FMOD, the MAGI system also included some creative and original specializations. "There are a lot of time signature variations or changes in JRPG music, so I designed MAGI to support them," Iwamoto says. Among its other capabilities, MAGI has the ability to adjust to changing tempo and time signature values across the length of a composition, allowing flexibility in the placing of synchronization points within the track. This capability was born out of both a spirit of innovation and pure necessity. Because Iwamoto was hired by Square Enix just two years earlier, much of the music of Final Fantasy XV had already been composed and recorded before he joined the team. "Many of these (compositions) were not planned to be interactive," Iwamoto admits. Because of this, the musical compositions included variable tempos, dramatic pauses, and frequent shifts in time signature.
In order to make these tracks interactive, Iwamoto used the MAGI system to insert custom synchronization points into each composition so that the transitions would be pleasing in a musical way. "All you have to do is just move or delete (sync points) when you think the transitions are not musical, or add the sync points when you think the transition is musical," Iwamoto comments. "Sometimes these sync points can be hard to find, and you may have to wait sixteen bars or more to (find a workable) transition." This problem became a frequent issue during combat music, so to solve this problem, Iwamoto devised a strategy that he called the pre-end section. This was so named because it is always used as a preparation for the ending of the combat track. "The pre-end section is designed to have more constant and dense sync points," Iwamoto explains. "That makes for really quick transitions." Existing as a separate music file, the pre-end section would always be in the same style as the main body of the composition, so the music could switch from the main body to the pre-end section smoothly. Designed to be more consistent in terms of tempo, key and time signature, the pre-end section could accommodate lots of sync points, allowing the music to transition to a finishing flourish with almost instantaneous precision in accordance with the end of combat. Despite its seeming complexity, Iwamoto meant for this system within the MAGI engine to be intuitive by design. "We used very simple and basic approaches," Iwamoto says. "I wanted to make (MAGI) very easy and versatile."
For audio director Becky Allen, simple and basic approaches are important for very different reasons. "We are a mobile studio and we have to be very mindful of our size because of the devices we’re played on," Allen says, describing her strategy for the MIDI music system of Plants vs. Zombies: Heroes. "We’re always thinking about size, and ways to be smart." Part of that strategy involved a plan for limiting the amount of memory required for the MIDI files and their associated library of instrument sounds. At first, those limitations were meant to be sensible but not extreme. "80 megs," Allen says. "40 for music, 40 for sound effects and VO." But then, the audio development process hit a snag when another mobile game came along and made a big splash in the marketplace. "Along came Clash Royale, and that was a quick game, it loaded quickly," Allen says. "We had some pressure to come down to 30 megabytes."
