This article was originally posted on the Joy Machine blog; maybe check it out too! <3
When I was working on Starhawk, we had an critical feature in the game that allowed for “hotfixes” to be deployed for data updates (game/balancing adjustments, generally) without requiring a major patch that would have to go through Sony’s week-long (at a minimum) patch and QA process. This was my first experience with game updates that didn’t require a full-on binary update.
Shortly thereafter I worked on mobile games for a couple years and, as I’m sure many people know, any opportunity to bypass the iOS App Store’s reviewing process is a wonderful — well, necessary — thing to do. So, for our team, we handled all game data through an extensive amount of JSON data for level definitions, player profiles, settings, game and balancing data, triggering special events and deals, so on and so forth. I believe we also stored player data on a secure server/database as JSON data, but that wasn’t and still isn’t my area. Thankfully.
Goals for Game Data and Live-Ops
I like storing as much game data as I can in text-based data files (specifically JSON files). And when I refer to game data, I mean things like mech data (which is, largely, just a list of references to other data files and game asset paths), mech part templates that are used to procedurally generate in-game items, game world parameters, player profiles and settings, and so on. These same formats are also used for storing data that is created in-game; while an item may be generated from one or more data templates, it eventually has to get saved and stored somewhere. Those files aren’t really ever touched beyond that outside of saving/loading.
That said, those files also retain references to their original templates, meaning that if some template was found to be asininely overpowered, the original template can be tuned, and the effects of doing that will ripple out to whatever was created in-game.
Since these are all basic text files I can access, I can easily tweak them and just refresh them in-game. The ability to easily modify game data in-editor and out with a simple rile refresh is just lovely, but it’s also been the core of game development and design for ages (called data-driven gameplay/content). Unreal Engine 4 allows for this entire workflow, but at some point game content has to be packed and, to my knowledge, beyond that they are contained within binary packages. And one thing I like about my game data is that it’s always easy to get at, tweak, load, etc. during the course of development regardless of whether it’s a loose build or a packaged build. Something I’ve also gotten accustomed to after a few years in mobile games is not having to rely on a full binary update/patch to a game in order to make changes, especially if it all amounts to little more than a tuning pass or a non-binary-patch hotfix. I’d much rather have players just load up the game, have a ten-twenty second update period, and just have all the updated tweaks. This whole process is also the core of any post-launch “live ops” setup that I’ve worked with: update some files on an authoritative CDN, players download them when going online, and now you have a 24 hour special event with unique items.
And as I’m starting this article, I really hope that the first comment somewhere isn’t “wait, you didn’t just use <x>?”, revealing a critical and completely relevant feature set that would have been incredibly useful. But, I don’t think that’s the case in what is to come.
A Surprising Realization
When I first started working with Unreal, I was doing all my prototyping in blueprints. As I’ve noted before: that didn’t work out well at all. I was under the assumption that blueprints were intended to be a nice frontend over, essentially, scripts. And that these blueprints could easily be tweaked/updated dynamically after building a project.
After writing that, I realized I should note: this was very early on in my time with UE4 (and I had never used a prior version of Unreal Engine).
Anyway, yeah, so: no. That’s not the way blueprints work at all; in fact, it’s quite the opposite: upon cooking a project’s content, blueprints actually get built into C++ code — potentially the lowest rung on the ladder of easily-updated game data. So, it really went in a different direction than I was expecting.
There was a fairly lengthy duration of time after Blueprintpocalypse 2017 that I had to just spend establishing the very basic codebase necessary to begin work in earnest on the actual game. And then establishing the basics of the game’s critical actors/components/etc. And it was after that initial pass of development that I started searching for a raw (uncooked. get it. because… well… content gets cooked? hush.) way to maintain game data, configs, local profiles, and other such things that didn’t require cooking to be used by the game runtime.
Despite my desire to just jump to JSON for game data, I wanted to see what the conventional/recommended practices were for UE4 for data-driven design and development. My inclination was to use Data Tables given that they’re covered in documentation that is literally titled “Data Driven Gameplay Elements”. And it sounded, well, it involved Excel, so it didn’t sound great, but it was something. And then I looked into it and discovered that you could manipulate the data externally, but it would eventually have to be imported into UE4 to be used. You can export it later, if need be, but eventually it has to be reimported. And eventually cooked.
So, yeah. Wheeeeee.
<iframe title="Heavy JASON!" src="//www.youtube.com/embed/0cgOti7gLus?rel=0&enablejsapi=1&origin=https%3A%2F%2Fwww.gamedeveloper.com" height="100%" width="100%" data-testid="iframe" loading="lazy" scrolling="auto" class="optanon-category-C0004 ot-vscat-C0004 " data-gtm-yt-inspected-91172384_163="true" id="935819364" data-gtm-yt-inspected-91172384_165="true" data-gtm-yt-inspected-113="true"></iframe>Yes. At least once a day, when I internally think about JSON, I instantly hear JASON! JAAAASON! JASON! in my head. Anyway, yeah, I liked working with JSON files either due to their simplicity or just habit, so I then went about figuring out how to make that work. Luckily, I found UE4 had Json and JsonUtilities modules!
So, I checked out the documentation after a cursory glance at the source files. And the documentation for Json and JsonUtilities was... Well. It was what you see. And in case anyone reads this in a year or something and it changes, for posterity let me inform you: there isn't much to look at.
That said, the source wasn’t complicated to wrap my head around. Putting it to use in practice, however, was a different story. My first test of the modules’ functionality was to serialize out an instance of my mech actor. And I was able to get it working pretty quickly and easily! And if you’re expecting a “but…” to follow that statement… … … Yes there is a but:
The serialized output of an actor is a horrifying disaster of data that you would never want nor even need to see in a maintainable data file. It was the entire actor. Including data you didn’t even know actors had. And it did an admirable job of recursively serializing everything in said actor, but it still was a lot of data that was clearly not meant for the purpose of, say, a data config for a mech part.
After a bit of digging — and a fairly extensive amount of trial-and-error — I eventually got to the point where, so long as I stuck to structures consisting solely of fairly simple data types, I could use the UE4 serialization utilities to easily serialize and deserialize the structure’s data.
Using that data structure in the nitty-gritty of the rest of the game code, however, isn’t quite as simple. C++ with UE4 is not entirely standard C++. A structure, to be recognized by the Unreal Header Tool (and Unreal Build Tool), must be signified by the USTRUCT macro. And, unlike standard C++, a USTRUCT struct is not "basically" the same as a class. It's intended (and strictly enforced) to be a structure of basic data that doesn't need to be carefully managed by the engine's garbage collector. I get a bit fuzzy on the specifics since I haven't worked on this stuff in a while and also working on all areas of a game tends to obscure the specifics of work done a week or two ago; a month ago is just way too far back to even consider.
Long story short: to really be able to just use the data that gets serialized/deserialized via a USTRUCT, I decided to, essentially, create a near-carbon copy UCLASS (with some transformations/setup occurring during the transfer in some cases). And once that UCLASS is all filled with delightful data, it can be used all over the place as intended.
I tossed up a gist (seen below as well) of an older version of my highest-level mech part data structure that exists in my codebase (as you go down the tree, it gets rapidly more dense). That gist also uses my absolute favorite, if totally ugly, macro sets I’ve ever created: the accessor macro generators (these do not result in the methods defined being UFUNCTIONs, however -- which basically means they can't be exposed to blueprints/external modules; in this case, that's exactly what I want). This is nothing more than a shortcut for generating basic get/set accessors on the UCLASS version of a data structure (so, not entirely unlike C# properties), but it was a handy way to trim down the definition and remove as much room for error in implementation as possible. That high-level part data structure serves, essentially, as a great-great grandparent to more specific part logic, but each child is only responsible for managing its own data; any parent to that child will take care of its data and that parent's parent will take care of its data and so on.
// FMechPartDataBase Data Structure. // This structure is solely for serialization/deserialization purposes (it gets transferred to the UObject instance after that process is done). USTRUCT( ) struct FMechPartDataBase_SerializationStructure { GENERATED_BODY( ) public: FMechPartDataBase_SerializationStructure( ) : ConfigName( NAME_None ) , PartType( 0 ) , Mass( 1.0f ) , HealthMax( 1.0f ) , AssetPath_MaterialPrimary( NAME_None ) , AssetPath_MaterialSecondary( NAME_None ) , AssetPath_MaterialAccent( NAME_None ) , AssetPath_MaterialChrome( NAME_None ) , AssetPath_MaterialMisc( NAME_None ) , AssetPath_SoundCue_Impact( NAME_None ) , AssetPath_SoundCue_Ricochet( NAME_None ) { } public: /* * Config Name. */ UPROPERTY( ) FName ConfigName; /* * Part Details. */ UPROPERTY( ) uint8 PartType; UPROPERTY( ) float Mass; UPROPERTY( ) float HealthMax; /* * Asset Reference Paths. */ UPROPERTY( ) FName AssetPath_MaterialPrimary; UPROPERTY( ) FName AssetPath_MaterialSecondary; UPROPERTY( ) FName AssetPath_MaterialAccent; UPROPERTY( ) FName AssetPath_MaterialChrome; UPROPERTY( ) FName AssetPath_MaterialMisc; UPROPERTY( ) FName AssetPath_SoundCue_Impact; UPROPERTY( ) FName AssetPath_SoundCue_Ricochet; }; // UMechPartDataBase Class Definition. // This is the class used for actual game code (as opposed to the data structure below, which is strictly for serialization. UCLASS( BlueprintType ) class UMechPartDataBase : public UObject { GENERATED_BODY( ) public: UMechPartDataBase( const class FObjectInitializer& ObjectInitializer ); private: UPROPERTY( ) FMechPartDataBase_SerializationStructure BaseDataStructure; protected: // TSharedPtr< FMechPartDataBase_SerializationStructure > DataStructure; FMechPartDataBase_SerializationStructure* DataStructure; UPROPERTY( ) TEnumAsByte< EMechPartType::Type > PartType; public: virtual void SetSerializationData( const FMechPartDataBase_SerializationStructure* SerializationData ); // Read-only access to the data structure (weapon data structure, specifically, unlike the prior two). const FMechPartDataBase_SerializationStructure& GetBaseSerializationDataRead( ) const; // Writable access to the data structure (weapon data structure). void GetBaseSerializationDataWrite( FMechPartDataBase_SerializationStructure& SerializationDataOut ); FMechPartDataBase_SerializationStructure& GetBaseSerializationDataWrite( ); public: // NOTE (trent, 1/25/18): These accessor-generation macros do not result in the methods they define being treated as UFUNCTIONs (blueprint-exposable). #define DEFINE_METHOD_SET_ACCESSOR( MemberType, Member ) \ FORCEINLINE_DEBUGGABLE void Set##Member( MemberType Member ) \ { \ DataStructure->##Member = Member; \ } #define DEFINE_METHOD_GET_ACCESSOR( MemberType, Member ) \ FORCEINLINE_DEBUGGABLE MemberType Get##Member( ) const \ { \ return DataStructure->##Member; \ } #define DEFINE_METHOD_ACCESSORS( MemberType, Member ) \ DEFINE_METHOD_SET_ACCESSOR( MemberType, Member ) \ DEFINE_METHOD_GET_ACCESSOR( MemberType, Member ) // Mech part type. inline void SetPartType( TEnumAsByte< EMechPartType::Type > PartTypeIn ) { PartType = PartTypeIn; DataStructure->PartType = PartTypeIn; } inline TEnumAsByte< EMechPartType::Type > GetPartType( ) const { return PartType; } // Part JSON config name. DEFINE_METHOD_ACCESSORS( const FName&, ConfigName ) DEFINE_METHOD_ACCESSORS( float, Mass ) DEFINE_METHOD_ACCESSORS( float, HealthMax ) DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialPrimary ) DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialSecondary ) DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialAccent ) DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialChrome ) DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_MaterialMisc ) DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_SoundCue_Impact ) DEFINE_METHOD_ACCESSORS( const FName&, AssetPath_SoundCue_Ricochet ) #undef DEFINE_METHOD_ACCESSORS #undef DEFINE_METHOD_GET_ACCESSOR #undef DEFINE_METHOD_SET_ACCESSOR };
Anyway, yeah, so that’s the gist — at a very basic level — of how I ended up handling text-based data-driven design/development. It may not be (and I actually hope it’s not) the best or most elegant solution, but it accomplishes what I want: external data files that do not require cooking to be used by the game; though, you do have to ensure that any data files that are necessary for a cooked game project are included into the resulting project builds. Unless you just want to keep everything on a server and download it at startup, which I can do as well, but I don’t want to force first-time players to have to get online to even get into the game, so a “starting set” of data is included with builds.
Also, I can’t toss up the entirety of my JSON serialization module, but I did include a very, very basic (but super helpful) wrapper for basic operations: JoySerializationManager.h and JoySerializationManager.cpp.
NOTE: I’m glossing over a whole lot of complexity that exists as part of this whole system and setup; partially because this isn’t the sexiest of topics to write about and I don’t know if anyone will ever even see this note. But also because it was really annoying and somewhat time-consuming to sort out, so I’m not 100% confident that the solution I have is the right one. I don’t like peddling lies. Well. At this moment in this context.
Additional Features and Future Work
Ease-of-Use (“Quality of Life”) Features
I generally keep my entire JSON file library open in VS Code at all times as it is, by far, my favorite editor for work that isn’t done in C/C++/C#. If I need to change game settings or tweak the balancing of a mech weapon or its legs (I tweak legs sometimes), I just alt+tab into VS code, make the change, and go back to work.
