The leak moves up

In November 2002 Joel Spolsky gave a name to something every engineer already knew.

All non-trivial abstractions, to some degree, are leaky. Joel Spolsky, The Law of Leaky Abstractions, 2002

TCP promises a reliable stream over an unreliable network, until someone pulls the cable. SQL lets you say what you want rather than how, until a query plan goes quadratic and you are reading about indexes at midnight. The abstraction saves you work on the good days and hands you the layer below on the bad ones.

The part people forget is his corollary: abstractions save time working, but not time learning. You still need to understand what is underneath, because that is where you will be standing when it leaks.

For twenty years we accepted that deal because the alternative was unaffordable. Writing the thing bespoke, every time, for every case, cost more than any leak. Reuse was the only economics that worked. A generic layer running at 80% efficiency beat a perfect one nobody could pay for. Whole ecosystems were built on that arithmetic: frameworks, plugin systems, extension points for cases that might one day exist.

The price changed

A modern coding agent can hold an entire application in one coherent context and write the monolith: code specialised to one job, one data shape, one machine, with no registry, no adapters and no extension points for cases that will never arrive. The one-time application used to be a luxury. Now it is a week's work.

When specific code is cheap, generality stops paying for itself and what remains visible is its cost. Every generic layer encodes assumptions about what will vary and what won't. Those assumptions are exactly where it leaks, and the cost lands hardest on whoever gets their performance by violating them.

And Spolsky's corollary flips. If an agent can learn the layer below in minutes, the rational response to a leaky abstraction is no longer to study it and work around it.

Often, the rational response is to delete the layer.

An illustration, still in progress

I have been trying to bring ds41rt into vLLM. ds41rt, by @tpurtell, is a Rust inference engine written for exactly one model, DeepSeek-V4.1-Flash, on one specific mix of hardware, and it is unapologetically a monolith: its kernels, its cache, its scheduling and its network transport were all designed together, for that job and no other. It performs remarkably well for a freshly coded inference engine, and I think there are still mountains of untapped optimisation left in it, because in a monolith every one of them is within reach: nothing has to be negotiated across a seam first. I wanted that speed behind vLLM's serving surface, and my first instinct was to merge the two: teach the framework's own execution path to do what the monolith does.

vLLM has to serve a great many models, so it has to assume things. It assumes the cache is uniform pages that the scheduler owns, that a step is a synchronous call, that every device does the same work, and that batching is an optimisation invisible to the result. For the engine I was merging, each of those was false. One divergence traced down to a single four-bit value in a compressed cache row that differed depending on whether a request ran alone or in a group. In a quantised, persistent cache a rounding difference doesn't wash out. It becomes a fact.

None of that is a criticism. Those assumptions are why the framework can support everything. But I have become sceptical that vLLM's own execution path can match ds41rt without considerable surgery, of the kind that would leave it less generic for everyone else. The engine's speed does not come from one clever kernel that could be lifted out and registered as a plugin. It comes from every part knowing about every other part, which is precisely what a framework's seams exist to prevent. Merging meant paying for generality the monolith had deliberately refused. So I stopped merging and moved the boundary instead: the framework keeps what is genuinely model-independent, the engine keeps everything it is opinionated about, and they meet at a narrow interface where one side hands the other a tensor of logits. It is not finished. But the lesson arrived before the result did: the abstraction didn't disappear, it moved up, to a place where it has far less to leak.

This view did not arrive as a theory. It firmed up through this vLLM work, and through conversations with TJ (@tpurtell) about how he goes about building these engines. His approach is the demonstration: a working engine that shows what the argument only claims.

Where the contract belongs

"Token ids in, logits out" is true of every autoregressive model, so a contract there costs almost nothing. "Your cache is uniform blocks" is true of most models, so a contract there costs you every model where it isn't. That is the general rule: put the contract where the statement is true of everything, not of most things.

Follow that upward and you reach the only contract that was ever non-negotiable: the one your users hold. The interface they call, the behaviour they depend on, the promises their own work is built on. Everything beneath it is implementation, and implementation is now cheap enough to be disposable. Reuse still earns its keep at that top layer, because that is where other people's work depends on yours.

What this asks of frameworks

Agents made writing the monolith cheap. They did not make trusting it cheap. A bespoke implementation has no community finding its bugs; the only thing standing between it and silent wrongness is the test that says what right looks like.

So if I maintained a framework today, I would draw two conclusions. First: concentrate on the natural points of extensibility, the few seams where a statement really is true of everything, and make them stable, documented and narrow, rather than adding more generic machinery in the middle, where specialised code will increasingly route around it. Second: regression and conformance suites may now be worth more than the framework code itself. Code is what an agent can regenerate on demand. An authoritative definition of correct behaviour, one that lets you throw an implementation away and know the next one is right, is what it cannot.

The law of leaky abstractions still holds. What changed is what we do about a leak, and where we choose to let one exist.