Part 1 argued that coding agents have changed the economics of abstraction. When specialised code is cheap, a generic layer's assumptions stop being a bargain and start being a bill, so the contract should sit where its statement is true of everything, and the implementation beneath it should be treated as disposable. That was about the inside of an inference engine. The same law is now at work one layer up, at the boundary between the agent and the model, and it is leaking in a far more expensive place.
The cache is the asset
The interface almost every agent uses says: send a list of messages, receive text. It presents the model as a pure function of a string. Underneath, the engine holds state: the key-value cache, the computed form of everything the model has already read. For an agent, that cache is most of the work. The team behind Manus reports around a hundred input tokens for every output token, calls cache hit rate the single most important metric for a production agent, and notes a tenfold price difference between cached and uncached input.
On your own hardware the same fact shows up as capacity rather than price. In measurements I published earlier, a one-million-token context took four and a half minutes to compute cold and about one second to restore from a cache, and at thirty-two concurrent agents the same machines completed three times the work with the cache than without. How you configure caching decides how much of your hardware you actually get.
Two workloads the interface never imagined
The agentic loop: dozens or hundreds of turns, each adding a few hundred tokens to a context already a hundred thousand long. And the event bot: one session that never ends, woken by the world and expected to remember. Both re-read nearly everything on every turn. Both live or die by reuse. And both, sooner or later, run out of room and must shed context.
Where it leaks
The cache is ordered. Each token's state depends on every token before it, so it can be reused only as an unbroken prefix. Append to the end and everything stays warm. Change one token near the front and everything after it must be computed again.
Compaction, the standard answer to a full context, is exactly that change. Summarise the old turns and you have rewritten the middle of the string; the engine throws away the state for everything downstream. A session running near its limit does this every few turns, which is the worst case: as Semenov and Dorofeev observe of eviction schemes, you keep paying to write the cache and never stay still long enough to read it. Hermes, the open-source agent I run daily, has had a string of bug reports of precisely this shape: a timestamp refreshed after compression, a memory note re-injected into the system prompt, and each time the whole prefix silently recomputed.
The deeper bind
It gets worse, because of how attention works. Where something sits in the context shapes how much it matters. The first tokens act as anchors for everything that follows (attention sinks), and material buried in the middle of a long context is the most likely to be overlooked (lost in the middle). A long-lived session must keep learning: new facts arrive and some of them need to matter as much as the system prompt does. The way to make them matter is to put them early. And putting them early is the most expensive edit there is.
Position is importance. Position is also the cache key.
A string-shaped contract gives the agent exactly one way to say "this is important", which is to move it, and moving it destroys the state. The two sides are being forced to communicate something rich through a channel that cannot carry it. That is a leaky abstraction in its purest form.
The sign: the interface is getting chatty
Look at what has happened to that simple interface. It has grown cache breakpoints you place by hand, with lifetimes you choose. Routing keys so that a session lands where its cache lives. Context-editing strategies, including a setting for the minimum amount to clear at once, because every clearing invalidates the cached prefix and has to be worth it. Server-side compaction. And, from one provider, compaction items that are encrypted and "not intended to be human-interpretable": state the client must carry back and forth but cannot read.
Every one of those is a sensible feature. Together they are a symptom. Each knob exposes a little of the state the contract pretends is not there. I think this generalises, and it resolves what looks like a contradiction with Part 1, which said to keep contracts narrow and move them up, not make them wider:
A narrow contract that starts growing knobs is a layer about to move.
Chattiness is not the destination. It is the transition state: what a boundary looks like in the period after it has stopped being true and before it has been redrawn.
The next important trend
So the near future is a finer-grained contract between agent and engine, and I think it is the most important trend in inference that is not about the models themselves. The shape is not hard to guess. Segments with stable identities, so the two sides can talk about parts of a context rather than resending all of it. A way to mark importance that is not position. Edits that come with a quoted cost, so the agent can decide whether a compaction is worth it. Visibility into what is warm, and lifetime hints in the other direction ("I will be back in ten minutes"). Snapshot, fork and restore as first-class operations. Compaction performed by the engine on its own state rather than by rewriting text, which is where research on Cartridges and neural cache compaction is heading. And reuse that is not tied to the prefix at all, the goal of CacheBlend and EPIC.
Today each of those arrives in one vendor's dialect, and is exercised first by that vendor's own harness. Every major model provider now ships one, and at least one publishes its agent benchmarks as measured inside its own. That is not a conspiracy. It is what a transition state looks like: whoever owns both sides of a leaking boundary can route around the leak first. For everyone building open engines and open agents, the lesson of Part 1 applies directly. The valuable artefact is not any one implementation of this richer interface. It is an agreed definition of it, and the conformance suite that proves an engine and an agent honour it.
Where the layer ends up
But a chatty contract is still a transition, and I do not think it is where this settles. The statement that is true of everything an agent does is not "messages in, text out", and it will not be "segments, pins and snapshots" either. It is "here is a task". When the contract moves there, harness and engine sit beneath it as a single implementation with a single owner of the state, free to cache, compact, reorder and forget in whatever way the hardware rewards, because nobody above can see. That is Part 1's monolith, one storey taller, and just as disposable.
I think that move is only a matter of time. Until then, watch the knobs. They are the abstraction telling you, as plainly as it can, that it is in the wrong place.
Larger attended contexts, biological density and selection—and the freedom to build the intelligence that acts for you.
The law of leaky abstractions in the age of the one-time monolithic application.
Further reading
- Joel Spolsky, The Law of Leaky Abstractions (2002)
- Yichao Ji, Context Engineering for AI Agents: Lessons from Building Manus (2025)
- Semenov and Dorofeev, Beyond Compaction: Structured Context Eviction for Long-Horizon Agents (2026)
- Xiao et al., Efficient Streaming Language Models with Attention Sinks (2023)
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)
- Yao et al., CacheBlend (2024); Hu et al., EPIC: Efficient Position-Independent Caching (2024)
- Eyuboglu et al., Cartridges (2025); Baseten, Neural KV cache compaction (2026)