Using static databases and simple code generation to describe almost-dynamic game content.
I’m a solo indie dev working on my magnum opus of undisputed genius, _space_train. The daily grind of handling all aspects of game development is my burden, except when I’m not doing that because I have a regular job as a software developer instead. The major flaw with _space_train is that there’s no compelling reason to play it, and that that will continue to be the case for the foreseeable future. Nonetheless, if you want to follow development and watch as I slowly descend into madness and ignominy, it has an itch.io page here with a downloadable build and a Twitter here.
I had an issue building _space_train, my magnum opus of undisputed genius. The path to complex user interactions lay in the direction of adding ‘content’ for prospective users to engage with. In material terms, this meant adding more of the things that people do in the game: items to pick up and use. The problem was, adding new items was proving a surprisingly laborious task for me, the developer. Items in _space_train are represented by unique id numbers associated with an entry into a centrally managed map which has an id, a type, and a ‘display modifier’, a bitfield which specifies the appearance. The list of possible item types is represented with an enum, and a separate mapping exists to convert between the enum value for each type name and the string name for each type — the string value being necessary for accessing external resources for each item type, specifically Lua scripts which say what they do in various scenarios.
Adding a new item then, means adding both to the enum, and then to the name map, the name of the new item. In addition, over on the client side, the item type then needs to be associated with coordinates into the item texture, represented in another enum. All this before any actual material properties of the new item can be expressed!
The elephant in the room is the possibility of loading and representing item types at runtime. This would necessitate separating items into some extra-compilatary zone where they can be manipulated distinct from the game code, all together all in one place. This was appealing in some ways — it would remove business data from the scrappy C++ engine I’ve insisted on building and relocate it, in a potentially more manageable format. I didn’t want to give up, however, on the ease of referring to item types directly in code, or take on the overheads of runtime look-up.
Around a year ago, I became aware of a software development tool called CastleDB (www.castledb.org), which appeared to do something to do with static data. The website described it thus: “CastleDB is a structured static database easing collaboration.”
Structure sounds good. Static sounds good. Unfortunately there’s only me working on _space_train so the collaboration might go a little under-utilised, but I was still excited.
Unfortunately, at the bottom of the page, castledb.org also says this: “CastleDB was created using the Haxe technology, it also allows some Haxe-specific integration.” I’ve heard of Haxe, and the website for it only really confused me more as to what it does, I’m aware it’s some sort of web-development-y thing. And I myself, a terrible C++ programmer, don’t understand the first thing about it — or any other webdev library. Thrown into confusion like this, unclear over whether or not CastleDB was something only web developers got to use, I did what any good programmer ought to do: I left the tab open for a full year and skim-read the page again, once every other month.
Eventually, I decided to go one step further: I downloaded the damn thing. Inside the zip there was an index.html, which loaded into a blank spreadsheet with buttons that didn’t work and an error in the console saying “CONSOLE ERROR HERE”. It turns out CastleDB is bundled as some kind of self-hosting web executable, and I was supposed to go against years of instinct and open cdb.exe rather than the enticingly-named index.html. Never grow complacent. With CastleDB opened properly, I found that it was actually exactly what I wanted: for want of a simpler phase, it’s Excel but not dynamic. It’s phpmyadmin but not php or mysql. It’s hand-editing JSON but without any hand-editing JSON. You enter static data in a familiar-ish interface (minor gripe — copy and paste doesn’t work on MacOS, but I’m narcissistic enough to enjoy retyping my own work) and it comes out in JSON. This was the start of something good.
What I wanted, was to enter data using this convenient interface, and have it represented in code for me to use when designing game elements (in theory the bulk of game development; in practise I merely add more engine features and release nothing). This necessitated some kind of transformation of the data into code, either at compile time or at runtime. As discussed earlier, I’m loath to defer this process to runtime for several reasons, most of which boil down to wanting to pretend my program is less complicated than it is. If elements are loaded at runtime, then all references discovered while debugging are contingent on that load, and it’s harder to have good instincts about what is out of place or unusual. So I wanted this to happen at compile time.
The missing link here for C++ development specifically, is some kind of all-constexpr JSON processing library that can automatically infer data structures from some input. I am not the person to write that library though, and decided to make do with some homegrown codegen.
Codegen is the art of code which produces code, and often leads to some of the worst abominations of programming imaginable. All the personal terribleness of the programmer who wrote it, combined with a computer’s indefinite ability to spew vast reams of procedural content. Nonetheless, codegen has a proud history in games: in the Wolfenstein 3D Black Book, Fabien Sanglard describes how iD used codegen to produce (and patch!) code for transformation tables for sprites at distance at runtime, in order to minimise the size of the executable shipped. If it’s good enough for shooting Nazis, it’s good enough for me.
JSON was originally invented to serve as a serialisation format for Javascript, so I immediately decided the best tool for this job would be Python. Python has a first-class JSON loading library, and very simple string manipulation functions (compared to working directly in C++ for this). Using a Python script to write some simple C++ types and functions into a header and cpp couldn’t have been simpler, and I very quickly had a working enum class generated, included and compiled.
From this quickly thrown-together table of existing content, a handful of Python lines in the fashion of the ones below:
output_h.write("""enum class list {
""")
for item in item_db_items:
output_h.write(" "+item['type']+",\n")
output_h.write("""
};
""")
Were able to throw out a quick enum:
enum class list {
none,
tile,
ticket,
food,
screwdriver,
shovel,
terminal,
pickaxe,
access_card,
hat,
corpse,
fuel_brick,
mug,
oxygen_mask,
urine,
};My enthusiasm for this approach quickly snowballed. Not only was I now generating the enum and the static constexpr map from the JSON file, I added a column for the item descriptions, making them easy to read altogether and also amend, and also for the short descriptions used when the items are hovered over.
Then I went one step further and moved a function from the engine client that selected an image to pair with each item, and had that generated from the JSON too. CastleDB has built in support for (varyingly-sized) tiles and will handily display them inline for easy glancing. I didn’t need the relative path to the image, both because resources in the _space_train engine come from specific named textures, and because the directory structure from compile-time has changed by runtime, but were that not the case it would have been easy to pull it out of the provided ‘tile’ data type in CastleDB.
With this under my belt, I drove further forward and converted the floor tile types and image mappings too, as well as room bounds (which are done under the hood in an extra layer of tiles. Three whole compile units are now being generated! I made myself feel extra professional by abstracting common functionality, mainly related to the opening of files, into a python module called codegen.py. I refrained from getting too smart and making a special python class with scoped constructors and destructors which would function to emulate C++ namespaces and other cleverness like that, because down that road lies madness and the aforementioned template-based constexpr JSON library.
Since adding the codegen functionality to _space_train I’ve identified several other candidates for static codegen, including a table of sound effects and a table of background music entries. If you made it this far, thank you for reading and I hope you enjoyed my terrible programmer jokes.
If you want to look at the codegen Python in more detail and send me a message about how bad it is, it can be viewed in a pair of gists here (codegen.py) and here (item_codegen.py).
