Skip to content

//Engineering

Scale is a series of trade-offs, not a technology

7 min read

Every few months a team asks us which stack scales best. It is the wrong question, and answering it honestly usually takes the conversation somewhere more useful.

The bottleneck is almost never the language

By the time an application is struggling, the slow part is rarely the code anyone was arguing about. It is a database doing a full scan because an index was never added. It is a synchronous call to a third-party API sitting inside a request that a user is waiting on. It is one service that everything else depends on, quietly holding the whole system to the pace of its slowest query.

None of those get fixed by a rewrite. They get fixed by looking. We have lost count of the times a team was certain they knew where the time was going, profiled it properly, and found it somewhere else entirely. Measure first. Intuition about performance is unreliable in a way that intuition about correctness usually is not.

Decide where state lives, early

The single decision that shapes how far an application can grow is where you keep state and who is allowed to change it.

On a telephony platform we built, session data, presence and idempotency keys went into Redis on day one. Not as a hardening pass later, because the constraint arrived immediately: the provider retries webhooks, and a handler that is not idempotent will charge someone twice or connect a call twice. The moment you accept retries as normal, every write has to be safe to repeat. That is a design decision, not an optimisation.

Call detail records went somewhere else entirely, into a relational store with a read replica so that reporting queries never touched the primary. Two kinds of state, two homes, chosen because they fail differently and grow differently. Putting them in the same place would have worked fine right up until it did not.

Concurrency is where check-then-write goes to die

Here is the bug that teaches this lesson better than any diagram. On a venue booking app, two people tap the same 6pm slot four milliseconds apart. The code checks whether the slot is free, sees that it is, and writes the booking. Both requests check before either one writes. Both are told yes. Two teams turn up at the same turf.

You cannot fix that in application code, because the app runs on several servers and none of them can see the others’ decisions. The fix is to stop asking permission and let the database refuse: a unique constraint on venue and slot, so the second insert simply cannot succeed. A conditional write — update where the status is still free — tells you that you lost by changing zero rows.

We added a five-minute hold in Redis on top, released if checkout is abandoned, so nobody pays for something that vanished mid-transaction. None of it is clever. It only shows up when two people want the same thing at the same moment, which is to say, exactly when the product is finally working.

Make the seams explicit

Most of the reliability work on that telephony platform did not live inside either system. It lived in the seam between them. The provider handles its own retries and fallbacks. Our side replaces unhealthy instances and fails over the database. Neither knows what the other is doing, so the contract between them had to be written down: handlers must be safe to call twice, and must respond fast enough that the provider does not give up and retry anyway.

Clear boundaries between interface, business logic and data are worth having for the same reason. Not because layering is tidy, but because a boundary is the only place you can change one side without holding your breath about the other.

What we would actually tell you

Make boundaries explicit. Decide deliberately where state lives. Assume every request can arrive twice. Let the database enforce what the application cannot. And profile the real workload before you change anything, because the thing you are about to optimise is probably fine.

Scaling well is mostly a sequence of unglamorous decisions made early, when they are cheap.