Way back in June, I was working on getting the game code running on Mac and Linux. And while the game runs on those platforms, there are still a bunch of details to implement for a final product, plus testing. But I've been itching to work on new games, so I've been working on new things more than working on Mac and Linux versions of Banished.
Here's a rundown of the past few months.
A lot of the new games I have in mind require a larger terrain than is in Banished, and unlike Banished, they would require a free camera that can look up to the horizon. So I went ahead and coded a new terrain system.
I started with a paper called CDLOD (Continuous Distance-Dependant Level of Detail Rendering) for height maps. My implementation started out true to the paper, but now I use a slightly different tessellation of the terrain with alternating diagonals, rather than all diagonals in the same direction. So my LOD is slightly different. And at the moment I still wanted to support hardware that doesn't support vertex texture fetch (or does but it's slow), so I cache chunks of the terrain on the CPU instead of reading the height data with the GPU. Having the vertex data available on the CPU is required anyway for collision detection.
This gives me a terrain that can render all the way to the horizon with no real size limits and nice LODs that fade gently with distance. A 128km x 128km terrain is easily rendered on low end hardware, and the level of detail can be tweaked to reduce triangle count, or increase detail depending on hardware. Of course a terrain that big has lots of procedural detail, since storage would be limiting.
Here's a view of the underlying triangles that would render when flying far above the terrain. I'm not trying to win any art awards here, this terrain and texture is purely programmer art for testing.
Beyond just getting the triangles to render, I also need to add visual detail that fills in the gaps between triangles with something that looks like much higher detail. So I added a system that can handle unique texturing across the entire terrain and use higher frequency data than the triangles used to render.
With some quickly generated normal maps, the view above turns into this.
The terrain also does simple occlusion culling so that large mountains block whats behind them and it won't render things far away.
I still have a bunch of work to do to turn this into something usable in a game, such as good color textures and terrain generation that supports gameplay instead of simple fractal noise.
Cross platform toolset
Since I had terrain working, I wanted to make sure it worked on all Linux and Mac as well as Windows, since I'd prefer to be able to launch my next game cross platform all at once instead of doing it after the fact.
But I had a bad workflow. On Windows, I can edit resource files and the game reloads them on the fly, or the resources compile when needed as they get loaded from disc.
On Mac and Linux, I only ported the game code, and none of the toolset. So the workflow goes:
- - Make sure PC build works.
- - Compile all resource and generate a pack file.
- - Put the pack file somewhere accessible by the Mac and Linux machines.
- - Make sure Mac and Linux compile.
- - Run Mac or Linux build.
This is fine until I need to change data to debug a problem. If I'm working under Linux and want to do something simple like view the terrain in wireframe mode, this requires a data change (yes, all my render state is data), which requires a pack file recompile on Windows, etc, etc. Additionally when I travel, I prefer a light and small laptop, such as my MacBook, but if I can't edit data on it, then there's a limit to the kind of things I can work on.
And so I was motivated to make the entire toolset work cross platform. Yikes. All that Windows specific code had to go!! So I ended up having to rework the engine by using cross platform code for fonts, image loading, dxt block compression, audio decoding, and a few other things. Thankfully the FBX format works on OSX and Linux, so I didn't have to change model formats.
A bunch of changes later and the OSX/Linux version can now build the project from raw resources and no longer requires a Windows machine to build the data. Woot. I wish it had actually been as simple as writing that sentence...
The only downside to this is that Linux/OSX can't compile DirectX shaders into binary form. It generates the text though. So I can't use data created on Linux directly on a Windows machine at the moment, but that's only a problem building releasable products. For day to day work I can now work on any platform. However 3DSMax forces me to use Windows, but I don't do artwork every day, especially during early development.
Next I decided that if I was going to start a new game, I was going to clean things up properly. Just implementing the new terrain made me wish parts of my game engine were different.
Back when I made console games, as soon as a game shipped, the code would be branched and the next project was lined up and work started immediately. Sometimes this happened even before the previous product shipped. This required just adding new code on top of the existing game engine to support the features of the new project. After several games, this starts to add up, and makes making changes fairly difficult. Occasionally entire systems would be rewritten, but it was rare.
So while I knew there wasn't a whole lot of change to what my game engine could do after being refactored, I decided to clean up the things that were implemented quickly or not as well as they could have been so that future projects aren't impeded by code pileup.
I felt two ways about this. On one side, I was longing to be prototyping a new game, testing new ideas. I've been thinking about new game ideas since early on in Banished development. On the other side I knew that my changes would make creating prototypes easier. If I happened to keep the prototype (which might happen), I wouldn't have to refactor that code or the data that went with it, if and when I decide to change the way things work.
The first refactor I did was to merge the Windows OpenGL code with the OSX/Linux OpenGL code. They were almost identical, but the Windows version was different because it's in a dynamically loadable library so that multiple rendering back ends can be supported. I may end up with dynamic libraries on OSX and Linux as well, as Metal and Vulcan may be implemented one day in addition to OpenGL. Either way, having just one OpenGL path to maintain is much better.
I then removed support for Shader Model 2.0 under DirectX. I can't do instancing in a memory efficient way with SM2, and I can't do efficient deferred rendering effects with it either, so I'm not going to support it. This removed a lot of code that was used to make instancing work and really cleaned up the rendering path across all back ends. So the next game will at minimum require Shader Model 3.0, but I may even drop this. Time and GPU surveys will decide.
Often times I want to test some new code outside of the current game project, which is quite hard once the game is in place. Where do you test? I would end up commenting out the startup code of the game and calling some other temporary code. So instead I made the 'game' code a dynamically loadable library. With this setup I can maintain multiple test projects and the test code can stay around and doesn't interfere with the game. A command line switch specifies which game library to load and that's it. This change also allowed me to have multiple data directories so I can keep test data separate from the main game.
I also removed all serialization versioning code. Between the beta release of Banished and now, a bunch of data formats have changed, but the game still has to support the old versions. This is really just code bloat, and a new game doesn't have to load Banished resources, so it's pretty nice to clean them all up and have very readable serialization functions.
I changed the object model that the game uses as well. Previously I had resources fully constant - the game wouldn't let you change them once loaded from disc. So if you have an object, say a character, it has one C++ class for the runtime data (where it is, what animation is playing), and a second class for the data that comes from disc (what model to use, what textures, etc). Now I can have the two classes be one in the same (or separate if the shared data is large). This cuts down on code classes and maintenance and it also allows flexibility. So now the constant data object is cloned when I need a runtime version, and can be changed as needed. If I need to change models or textures at runtime, it's easy, instead of adding more overhead to both the data class and runtime class.
Next I changed the way I store meshes and load them from FBX files. In Banished, every model I exported from 3DSMax had the same format - position, normal, and 3 texture coordinates. (with skinning data if it was animated). This was a little wasteful, as only trees used all 3 texture coordinates. Some models only used 1. To fix this, I changed mesh formats to pair models with the vertex format that the shader programs use. This allows a custom format per model if desired, saves memory since only what's used is stored, and allows internal verification that materials applied to a model will actually be able to render properly.
And then a ton of other small changes. Moving files around, renaming classes, removing hard text strings and making them configurable, making the engine more generic, moving game level code that can be reused down into the engine, the list of small things goes on and on.
All this should make the development of the next game smoother. Coding these changes feels a little bit like spinning my wheels, or going down a rabbit hole that doesn't have a bottom. But when the change is made, it's nice to see the code get smaller and have better organization.
Ok, it was time to start prototyping!
Steam Dev Days
But then I went to Seattle for Valve's second developer conference. I met a lot of other awesome indie developers, talked to old friends, made new friends, and saw the new Valve VR controllers. I had a great time. The conference was fairly VR centric, but was also about where Steam is heading in the future.
Talking to other programmers
Back from the conference, there I was, done with refactoring, done with getting the engine ready for a new game! I was writing gameplay code! New camera controls and behavior! A new method for placing objects and paths on terrain without a grid! Hooray!
Then one day I had lunch with a programmer that I used to work with. Talking with other programmers, especially when working solo, is really good to go over ideas and design. And sometimes it spawns ideas that you wouldn't have thought of yourself.
Somewhere in the conversation, my friend said "In my game engine, I've been thinking about having the hierarchy of objects be its own system." That thought stuck with me, and my friend and I talked about it over the next few days.
My code didn't do this, and against my better judgement, I decided to take a stab at implementing it and see if I liked the change.
And it really cleaned up my game engine. Positional and rotation control, animation, and attachment is its own system, and the graphics, collision, and audio systems simply reference transformations from the hierarchy system. It also really cleaned up the other systems that used to store the scene hierarchy in duplicate. It made rendering just about rendering, animation about animation, and collision just about collision. It also removed a lot of high level management that was used to keep things in sync that probably shouldn't have been at game level anyway.
It's a good change. However making this change was pretty large. Several hundred source files were affected, and I don't think I could have done it with a game in place due to the number of things that would have broken. Making the change felt a bit like being in a pit of despair until it was done. I just kept getting deeper and deeper into changes to the lowest levels of the rendering, collision, and audio code. Several times I considered just reverting all my changes because it was such a large modification.
So while it's a good change, it made me decide I'm done with major tech changes and code refactors until I really have a need for them. I don't want to just make my code better and better forever. I need to be adding features I need or working on game code.
Things not done
One interesting thing I found while doing all this refactoring are the things I didn't do. I had maintained a list of things I wanted to change about the code from before Banished shipped, but were too massive to do without majorly breaking the game.
Once I got to working on them, I realized about half of them didn't really need to be done, or that they were implemented a certain way to handle edge cases I had forgotten about. Or the code wasn't actually bad enough to warrant modification.
Not that cutting half my planned refactors saved any time - as refactoring progressed, I ran into things that needed to be changed or were implemented poorly that I hadn't thought of ahead of time.
Back to Banished!
Most recently I've been back to working on a new prototype, but I took some time out to work on an update for Banished that will only effects mods. Aside from a few bug fixes, the base game won't change but the game will allow modders to use configurable resource flags and limits, and hopefully remove the limit on memory allocations. More on this to come, but for now, here's a screenshot.