Academia: School Simulator has just graduated from Early Access. It stayed in EA for 4 years and we developed it for almost a year before EA. The whole development took almost 5 years. While the game is not an critical hit, it’s kind of a sleeper hit that’s good enough for us. We won’t be buying yachts soon but the game’s profits allow us to make another one.
I looked at the past blog posts since around 2017 when development kicked off. Reading old posts brings me back to those posts’ time. They’re like time capsules. I have like two other topics that I’m supposed to write about this January 2021. However, I thought that I should post something more relevant and nostalgic now that the game is officially released. So here’s an enumeration of lessons learned during development in the context of programming a builder/tycoon game that can have hundreds of agents. The list is not organized or ordered in a certain way. I just write what comes to my mind. So here we go!
Choose the game engine you’re most capable with
This is a no brainer. Our previous game was made in Unity. The devs have years of experience using Unity. We have existing libraries of C# code for Unity. Our development momentum would be at the highest if we use Unity. Unity it is.
Make custom tools for your game
It’s amazing to realize that the tools we’ve made since the start of the project is still being used until now and for the foreseeable future. One of the tools that I’m proud of is the GOAP AI authoring tool. This is the first and the most complex tool that I’ve made for the game. It has helped us tremendously.
Custom tools are especially useful if the artist and/or designer are going to be the ones who will use them. Most of our tools are data entry. When the tool is done, we pass the ownership and responsibility of the master file associated with the tool to the designer or artist. This frees us to work on more programming tasks. Making tools is like distributing work to non devs.
Choose the right AI for the job
Right at the beginning, we picked GOAP as the main AI system of the game’s agents. Back then, I thought that we needed an AI system with the least maintenance as I foresaw that the game will get bigger and more complicated. GOAP is perfect as it doesn’t have wires/connections that you have to manage like how FSM and Behavior Tree does. All you have to do is add more actions and the agent will automatically pick them up for execution when the situation arises.
The GOAP system that we had from the beginning is more or less the same until now. There were some fixes and updates but nothing too drastic like replacing the whole thing.
Stress test as soon as you can
Academia is my first attempt to stab at a complex simulation game with lots of agents. We’re competing with the likes of Prison Architect with no prior experience. So we’re coming in blind. Most systems are done as a first time. One of the unknown that I had to answer every time was “Are our systems performant enough for both processing and memory usage?”
I did stress testing the moment that we made the core of our game of admitting students and running the school day by day. At the time, we learned some depressing things. We’ve hit our performance budget with lesser entities as we had hoped. We realized early that we really don’t really have much wiggle room in terms of performance. Every code should be performant every time. The stress test kind of shaped the coding standards from then on.
However, do not stress test when you don’t have the core systems of your game yet. What I mean by this is don’t do stress tests that are outside the confines of your game code. For example, let’s say stress testing how many agents you can render. While this is useful, it’s less useful if not run with your game’s complex systems. It does not inform you of the actual limits that your game code can take. This is what I mean by “as soon as you can”.
Review code always
I can proudly say that our code is far from a spaghetti monster. I know this is true because we can still add or update features rather easily. The game doesn’t break too much for small changes. This is probably the result of mandatory code reviews. How is it mandatory? We set up our repository that way.
As the tech lead, I’m the only one who has access to our main branch. Other devs can’t push code here. The result is that all new and updated code must come from a branch. Each branch is associated with a task. Whenever a task is finished, code review is just natural. I can easily look at the branch, see the changes and review code as I go. I can choose to not merge a branch yet if I see that the code is not good enough.
Apply the boy scout rule
This was my biggest takeaway while reading the book Clean Code by Robert C. Martin. In Boy Scouts, there’s a rule in camping that says leave the place cleaner than you have found it, or something like that. Apply the same rule to your code. Every dev must adhere to this rule.
Most source code tend to decay because of quick fixes, no time to refactor, and technical debt. But imagine if every dev applies the boy scout rule. Your code has a chance to come out cleaner every time someone edits. The code gets better instead of decaying.
Find time to refactor
Not every project can afford the time to refactor. But if you can, take it. We’re privileged enough to work independently. We don’t have clients hounding on us. Our deadlines can be flexible. We refactor our code as much as we can. We refactor especially when a system or feature breaks often. The result is code that is maintainable even until now.
Integrate localization at the beginning
Do you remember when you started making your game? I bet that every UI label is a hardcoded English word. Later, you realize that you need localization. That means that you have to change every hardcoded label in your game so far. If you do this later, there will be more hardcoded labels to change.
This was us in a nutshell. Don’t repeat that mistake. Integrate your localization system the earliest you can. The sooner you do this, the sooner you can make a coding standard that all labels should be localized. Every hardcoded label from then on can be flagged during code review. There’s no excuse since the localization system is already in place.
Write dev blogs that give value
My posts are framed such that the reader can learn something as much as possible. The “what I added” part is just tangential to that. By doing it this way, the posts become timeless. It becomes some sort of documentation on how to do something. I can share old related post if someone asks a particular topic. I do this a lot. I hope that they learn something and discover the game at the same time.
Learn and integrate bleeding edge tech that might help your game
I’ve said during stress testing that we don’t have much wiggle room in terms of performance. We needed some kind of miracle algorithms or stuff that will make our game run faster. When Unity announced DOTS, I knew that it will help us and I couldn’t wait to play with it.
I studied it. Made some side projects. I was blown away by the speed and ease of writing multithreaded code. When I was confident enough, I made systems that will help speed up our game. I made a custom 2D renderer and ported our A* code to be Burst compiled. I integrated it to the game and it played better ever since. Suddenly, we have wiggle room in our performance budget. I would say that this was a huge turning point. We were getting mixed reviews because the simulation was not running well. After this upgrade, we’re getting more positive reviews than negative ones. Eventually, we turned the Mixed review into Very Positive.
Don’t try to learn every bleeding edge tech, either. You have limited time. Focus only those that might help your game. Unity released a lot of new tech during the development of Academia from 2017 to 2020. I think I only ever gave time to DOTS. I ignored the others. Shader Graph? Visual Effect Graph? HDRP? We don’t need those shit.
Invest on a better IDE
We started investing in Jetbrains Rider around 2018 when I was learning Unity’s DOTS. At the time, Visual Studio takes an awfully long time to load if you use the DOTS packages even on empty projects. I don’t know why. It’s just unacceptable. Then I tried a demo version of Rider and it loaded in a reasonable time. The coding experience was better than VS. From then on, I asked our CEO Ryan Sumo if we can spend some money for this IDE. I’ve never looked back. I learned to use DOTS and the rest is history.
Later on, I found a plugin that let’s you know where in your code have memory allocations. It’s just lovely! Very useful for code reviews.
Have a healthy balance of speed and maintainability
In the game genre like ours, speed is a top concern. Like I always say, there’s not enough wiggle room and we hit our performance budget a lot of times. This doesn’t mean that you should sacrifice maintainability. Maintainability is of equal concern as the game is big and complicated. Maintainability is the trait that will get you to the finish line. There is a sweet spot that both speed and maintainability are just good enough. Let me explain.
DOTS is a verbose beast. There’s no way around it. It’s verbosity makes it not that maintainable. However, it’s crazy fast. I can give up some maintainability to gain such speed. Moreover, it’s verbosity is not totally unreadable either. It’s still more readable as compared to a complex C++ code. I could say that I didn’t give up that much.
Another example is LINQ. Yes, it makes your code more readable. Produces garbage, though, so not advisable especially in our genre. But how unreadable is the non-LINQ version really? Honestly, it’s still fairly readable compared to C++ code. So yeah, LINQ usage is flagged in our code review. We avoid that shit.
Always strive to avoid memory allocations
The primary rationale is to avoid memory (fragmentation) related crashes. Game slowdown due to the garbage collection is secondary to that. We did have these kind of crashes during the early parts. I can’t fully explain the details why it happens but we fixed it by avoiding allocations, especially those that happens per frame (implemented in Update()).
When I make coding standards, I don’t think it’s wise to allow something in some cases and disallow in another. I make a choice to allow or disallow in all cases. It makes the coding standard simpler to remember and follow. So we avoid memory allocations as much as we can. Although, we can’t fully have zero allocations, it’s an asymptote to reach.
Use StreamingAssets if you want to support modding the easy way
Using StreamingAssets is like the old school way of maintaining assets. You put the asset in some folder with the game then you load the asset using a relative path. Modding is then easy because you can just load a user specified path instead of the original path.
If your game references assets inside the Unity editor, there’s no way to load a user specified path. There’s another new way by using Addressables but this will require your modders to download and use Unity. Addressables wasn’t available when we started project, anyway, so it wasn’t even considered.
Use Assembly Definition Files early
If you know that your project is going to be big (say more than 100k lines of code), plan the scripts folders carefully and use asmdef files as early as you can. If you do this later, it’s going to be harder to introduce as the class references would be all over the place.
Why use them? They reduce the compile time which reduces your iteration time. Compile time will only get longer as the project grows. It’s better to have asmdef files early so that new code files from then on are better organized if new asmdef files are needed.
Make your own A* framework
If you want to make simulation games with some kind of path finding, understanding A* is a must. There’s no better way in understanding A* than making your own. Believe me, it’s not hard if you’re an intermediate programmer.
The advantage of having your own A* code is you can bend it to your custom use cases. You will also know how to optimize it for your specific game modeling.
In our case, I was able to use DOTS to optimize our pathfinding. I wouldn’t probably be able to do it if I used a third party solution.
Learn simple multithreading
There are times when you don’t want the main thread to be bogged down by a long complicated task. When this happens, it looks like your game is not responding and some players would report it as a crash. You know it’s not a crash per se. You can simply fix this by firing a thread and run the complicated task there.
Strive for zero NullPointerExceptions
Oh boy, I hate NullPointerExceptions. When it happens, it messes up your game state that could propagate to other parts of the game that then could result to more serious bugs. Unfortunately, C# is one of those languages that has the million dollar mistake: the usage of null. With a big, complicated game, it’s bound to happen. There’s a cheeky programming law for this. I just forgot the name. It states that “any variable that can be null will be null at some point”. From experience, I agree.
I wrote about this here and here on how to mitigate. In Unity 2020.2, C# is upgraded to 8.0 which supports Nullable Reference Types. This might help but I don’t have experience with it yet as we are still stuck with 2020.1. For now, we are using Option<T> as much as we can.
Tackle the most complicated tasks first
When you have a list of tasks, prioritize the most complicated one first. The rationale is if you work on the most complicated ones first, you will make the big systems that is essential for that task. Later on, if you work on the less complicated ones, the systems are already in place and you can possibly use them thereby saving you some work.
If you did it the other way, there’s a chance that the more complicated task will need to override some code that was done for the simpler task or you need to refactor the simpler task to support the complicated one. You will add work instead.
Accept that there are some things that you can no longer change
It’s ironic that I say this while also shout about always refactoring. AI planning is the biggest thing that slows down our game. It’s heavily OOP, uses class references from everywhere, therefore it’s very hard to turn into ECS and DOTS the shit out of.
It needs an overall rewrite which I would say is impossible. If the project was just getting started, I would have done it. However, the project is already huge when DOTS was released. It would take an unknowable amount of rewrite and regression testing to get it to the same state. Needless to say that DOTS is very new at the time and definitely not production ready.
So I decided against it and accepted that our AI is slow. Do better on the next game instead.
That’s it! That’s 20 lessons from 5 years of working on Academia. There’s probably more but this is at the top of my head.
Academia 1.0 is currently on sale with 30% discount. It’s a good game because it has positive reviews. 🙂 Buy the game now!
Like my posts? Follow me on twitter.