I overengineered a leaderboard to handle a million users for $5 a month

I make a game called Blocknado and last week I launched a global leaderboard. Leaderboards can be deceptively complicated things to set up, but I didn't want to spend a bunch of money on a fancy service geared to much bigger games. I also wanted to see how far you could get in terms of functionality, and which tradeoffs would be necessary, by capping my budget at the lowest possible service tier on Cloudflare. Well, second lowest, the free tier didn't meet my needs (more on that later).

The board

It's a pretty straightforward leaderboard. It only shows the top 10 global scores, but it also lets you watch a replay of the game that got them the title. The replay wasn't part of the initial plan but as I built this thing I realized it ended up being a low hanging fruit and a really cool feature.

The pieces to this puzzle

The leaderboard is set up with the following components: A Worker, which is a little program that wakes when a request arrives, does its work and goes back to sleep. A database (D1), though there's not a lot stored in there. And a file store (R2): three buckets, one for the game replay data, which is basically a list of pieces and moves for the server to replay in the game engine, another bucket for the board data itself, and a third one for pages generated when sharing games outside the app.

Starting with the Worker, it handles validating the game score being submitted and putting all the right data where it should go. Finally it tells the user what their ranking is globally. The code that validates the game in the server is the game itself. The Worker and the game both run TypeScript and the actual game class is copied into the Worker on deploy. When someone finishes a game all the moves and pieces are sent and scored on the server. If it fails to restage the game, well I get a flood of automated github issues and have to go bug hunting.

The tables

There's are four boring tables making this all work. One holds each player's best game. It's just one row per player so a million players would only fill up a million rows. When you beat your previous best score the board remembers only the best of you and nothing else, which is a kindness the rest of the world does not extend.

Another board holds user names and another holds shared game replay data indexed by a random ten-character code. This one is cool because it lets me share and relive my moments of glory playing with a bunch of colorful blocks. Can you tell I miss childhood? And the last table holds flags, a few values the server reads fresh on every request so I can change them without shipping anything.

The game replay is quite small, just single digit kilobytes, but with a million players that can start to fill up a database, so I don't keep it there. Instead that data is kept in a file store which could easily handle all the games needed on the included base storage of ten gigabytes. And it only costs a cent and a half per gigabyte if I need more. The game file is written first and the row second, so a failure between the two leaves an orphaned file rather than a db record with nothing to say for itself.

Don't trust the user

Moderating user generated usernames is a recipe for disaster if you want to do a leaderboard on a budget. You can either integrate into some other game system, which didn't really fit my use case, or you can simply generate safe names and give users some choices to pick from. So that's what I did. A username in Blocknado is an adjective and a noun like "northern bonsai" and "copper hedgehog". This is a pretty common approach, and worked well for my purposes here.

Usernames aren't identifiers though, if I'm going to impose usernames on people the least I can do is let them change them and pick from a list. So instead the game generates a random identifier the first time a user opens it, and that remains their actual ID. If the user wants to remove their data from the system it just cleans up everything associated with that ID. There is no sign-in and nothing to reset. Wipe the app and you are a new person, I wish it were that easy in real life. Nothing public ever includes this identifier. The leaderboard itself, the game replays, and anything else only shows the player's username. It's not the most secure setup in the world, but I doubt a falling blocks game is a particularly lucrative target for would be hackers.

Working offline, the simple way

The game itself keeps a local top scores list for the user. It's basically the same data that gets sent to the server but stored on your device. There's a piece of metadata in that list saying whether it's been submitted to the global leaderboard. At certain points like opening the game, game over, or detecting a network state change, the game checks if any of your local games are marked as unsubmitted, and if they are they are sent to the server. So if you play games offline, your games will still make it out to the global leaderboard once you reconnect.

Actually showing the top 10 list

The board is read every time the app opens and I really don't want it to go stale. But if I am running db queries and invoking Workers on every request there's no way this is staying in budget. Instead I flipped the problem around, nothing original here, it's basically a Materialized View. So whenever someone scores a score high enough to make the top 10 it generates a static file that gets served as the global leaderboard. Then I put Cloudflare's cache in front of it. It has a TTL of about a minute but the user's app has an optimistic update, so everyone else sees your record within a minute, and you see it at once.

One thing to know is Cloudflare's cache is applied selectively depending file extensions, and .json files are not part of the list. So I ended up making a custom subdomain for the leaderboard and applying a blanket cache rule against it instead.

Sharing is caring

One happy sideeffect of this approach was I had full game replay data on the server. And it's easily serialized and put into static files. This is what led to the sharing feature in the app. A user can just share a link that includes a short random code, that link pulls up a static replay file from the R2 bucket. The shared URL is just a static asset as well, so it all fits quite nicely in the threshold provided.

Gotchas

The most important thing to know about the database's pricing is that you pay for every row it has to look at to answer a query, not just the rows it finds, and indexed entries count. So here are some gotchas that almost got me.

• When I first wrote the top 10 query it was sorting the whole board. When two players tie on a score it goes to whoever got that score first. This meant my query was sorting by score, and then by time. So although I thought I was in the clear because my table was indexed by score, potentially just reading the first 10 rows, it would then need to sort by submission time and sorting would mean reading all the rows in the database. The solution was simple though, you just need to also index on time, making the query actually only go through 10 rows.

• Log lines are billed, and each request in the server was being logged. This is an easy one to miss but can become very costly if you start getting floods of requests. You do get twenty million log lines per month though, so it's definitely manageable. The solution here, for now, is just sampling. I log about 10% of the Worker jobs. If there's a systemic issue it'll definitely show up, and since the only reason is debugging nothing is really lost.

Doing the math for a million players

Let's say a million players each finish two games a month and open the app once per game. Against the plan that comes to:

• Requests: two million app opens, plus two million games at two requests each, is six million of the ten million included.
• Program time: two million replays at about ten milliseconds each, plus four million cheap requests, is around twenty-five million of the thirty million milliseconds.
• Rows read: with the rank capped, about six billion of the twenty-five billion.
• Board reads: two million, served from the cache when it's warm and from the bucket when it isn't, and ten million bucket reads are free anyway.
• Rows written: a million names, plus say four in ten games being a new personal best, is under five million of the fifty million.
• Files: a million replays at six kilobytes each is six gigabytes of the ten, and a million file writes is exactly the free line.

So at a million players the bill is still $5, and every extra million games past the included requests is about a dollar fifty.

Why not free?

My initial crazy idea was trying to figure out how I could pull this off on some free tier, but the math just wasn't mathing. The free plan gives a Worker ten milliseconds of CPU per request, and replaying a long game takes longer than that on Cloudflare's machines. Long games are the high scoring ones, which are critical, and those games would get cut off halfway through. The $5 plan gives you thirty seconds per request. This was the only actual blocker to going entirely free.

If you're willing to just trust the score the game sends you, or have some more authentication approach that can still execute within the ten milliseconds, the free plan works fine. A hundred thousand requests a day is about twenty thousand sessions, or six hundred thousand games a month, or roughly three hundred thousand players, as long as you only work out an exact rank for the top ten. That's totally doable for a lot of games that can ensure people aren't just cURLing imaginary top scores into the leaderboard.

Ok so it's not free, but we've all read the horror stories of runaway bills on cloud providers. And Cloudflare has no spending cap, and neither does any other provider I looked at. It kind of feels like the early days of phone data plans and I was starting to have some serious data range anxiety. So I very intentionally architected this whole thing to grow with users, not requests, as much as I possibly could. If I strike gold and end up with over a million users I won't really care about going over my $5 cloud bill.

Here's the general guidelines that let me achieve this:

• Don't use Workers on reads, and if you must, rate-limit them heavily. The top 10 is the cached file generated on write, and a shared game is a file for the same reason: when you share a game the server writes one page for it with the replay data baked right in. This is the most important guideline.
• For writes, the Worker sits behind an address on my own domain which includes a WAF and cache. So if no actual work needs to be done or if there's some abnormal increase in usage the Worker doesn't have to deal with it.
• Every route is rate limited per minute, per player and per address, so the number of requests is a multiple of the number of players and nothing else. So even if everyone having a fantastic time and playing this in their every waking moment, don't, it's not healthy, but even if so, there's no runaway cost. If submitting a leaderboard gets limited, it'll just try again soon and nothing is lost.

Constructive paranoia

With all this in place, I also know myself, and as well thought out as this might be now I could end up opening a hole in some future version as a result of some dumb update I was too careless to properly review. And that could be a costly mistake. So I set up thorough smoke tests that run every time I deploy. Those tests send real games to the real APIs, they ensure caching is working properly, that db queries remain limited, so that I don't end up with a surprise bill. Since these tests run against the unpromoted workers in the CI I don't even need to roll back if it fails.

The database is the exception. Cloudflare's D1 does not have reverse migrations built-in, so I made a migrations manager that calls the migration, and if the smoke tests fail it runs a reverse migration. I'm writing both myself, it's not automated. But I also set up automated unit tests on the migrations files that assert the db pre-migration is identical to the db post-reverse migration so I don't accidentally make things worse in a panic.

Worth it?

I don't know if all this work and complexity are really worth it in most cases. There are absolutely many easier ways to get leaderboards into your game, especially if you don't care about it being cross-platform, and don't have an unhealthy fixation on trying to see if something is possible for no other reason than wanting to know. I've certainly gone down this path many times before and gave up before reaching the point of writing a post about it, so at least there's that. I am really happy with how it came out though and getting the replays and sharing was a really cool side effect.