Agentic workloads re-read the same long conversation prefix on every turn. When the GPU's KV cache cannot hold every active conversation, each returning turn recomputes its prefix from scratch and the whole fleet slows to the speed of prefill. We added a write-back cache of evicted conversation snapshots in host RAM to the DS41RT engine, for the hybrid AFD fleet (an RTX 5090 coordinator plus DGX Spark expert ranks): at 32 concurrent coding agents the same RTX 5090 + four DGX Spark fleet served 3.1× the turns at 6.1× lower average response time, recomputing a quarter as many prompt tokens; with no memory pressure the cache costs nothing measurable.
DS41RT is tpurtell's open-source engine for DeepSeek-V4.1-Flash on an RTX 5090 plus DGX Spark fleet at 1M context; this page adds a cache tier to it and measures what changes. Part 2 of a two-part series. Part 1, the fleet itself (14 September 2026): DeepSeek-V4.1-Flash at 1M context on four DGX Sparks plus one RTX 5090 — hardware, the 5090 port, and the baseline prefill, decode and 1M-context numbers this page builds on.
5f487c14;7c35f3e5.| Shape (fleet measurements, §7) | Cache off | Cache on | What changed |
|---|---|---|---|
| No memory pressure: one or two conversations, cold prompts of 15k–650k tokens | prefill 3.5–5.6k tok/s | the same: 0.99× on average, within ±5% per rung, arms alternated on the same engine state | Nothing. The cache only moves bytes when the GPU evicts a snapshot. |
| Many small sessions thrashing the GPU's retained-prefix banks (48 sessions of ~10k tokens, 6 users) | 174 revisits in the window, TTFT p50 2.3 s | 485 revisits, TTFT p50 0.5 s | Revisits restore in about 1 ms instead of re-prefilling 10k tokens. |
| Long sessions beyond the GPU pool (24 sessions of 300k tokens, 6 users; 7.2M tokens of working set) | 6 revisits, TTFT p50 160 s | 454 revisits, TTFT p50 0.9 s | The fleet stops being a prefill engine and goes back to serving turns. |
| 32 concurrent coding agents, each on its own growing 30k+ token session | 84 turns, 86 s average per turn, 43% of prompt tokens from cache | 264 turns, 14 s average, 96% from cache | Three times the work, a quarter of the recompute, with the load model retrying the front door's refusals the way a real agent does. The first pass, which counted refusals as errors, read 2.3× and 23 s. |
| A 1,000,007-token conversation returning after the GPU evicted its KV pages | 270.9 s re-prefill | 1.09 s restore from host RAM (893 MB) | The largest single conversation: 249× faster, output identical at temperature 0. |
| Active contexts that outgrow the GPU pool itself (three clients on 679k-token sessions) | With the cache on, three active ~700k contexts hit the engine's admission wall within one 600 s window; the pool cannot hold them with the cache off either, the cache just reaches the wall sooner because its turns are faster | An engine limit, not a cache effect (§7.4, §9). Fixed upstream in DS41RT v6. | |
K concurrent coding agents on the workload of §3, 240 s per point, same fleet, cache off versus on. The shaded zone is where the agents' working set no longer fits the GPU and the cache kicks in.
Left: below pressure the lines coincide; at 32 agents the average turn takes 14 s with the cache and 86 s without. Right: the same 32-agent window as a share of prompt tokens; recomputed tokens are GPU time somebody pays for. Sections 5 and 6 read both in full. The 48-agent point was run separately, because the ramp stops at its first failing point: 272 vs 81 turns, 28 s vs 131 s average, 94% vs 13% of prompt tokens from cache.
A transformer answers a turn in two phases. Prefill reads the whole prompt and writes a key/value (KV) entry per token into GPU memory; decode then produces the reply one token at a time, reading those entries. For an agent on a long task the prompt is the entire conversation so far: the system prompt, the code it has read, every tool result. On our own agent sessions that prefix is 50k–160k tokens and grows by a few hundred to a few thousand tokens per turn (§3). If the KV entries for the prefix are still on the GPU, the next turn only prefills the new tokens and starts replying in well under a second. If they are gone, the engine recomputes the whole prefix: 8 s for 55k tokens, 57 s for 320k, 4.5 minutes for 1M on this fleet.
The cliff is real: below it a turn costs a few hundred new tokens of prefill; above it every returning turn re-reads the whole conversation, and the same hardware does a fraction of the work.
The GPU cannot keep every conversation's entries. On a 32 GB coordinator the KV pool holds about 2.2M tokens for everything at once: active requests plus retained prefixes. With a handful of long sessions or a few dozen agents, the engine evicts the prefix of whichever conversation was idle longest, which is exactly the one about to come back. Above that threshold every returning turn pays a full re-prefill, the prefill lane saturates, and the fleet's throughput collapses to a fraction of what the same hardware does below it. That cliff is what KV-cache tuning is about, and it is why the same hardware can do double the work with the right cache in front of it.
DS41RT already retains completed prompts and turns as snapshots in two GPU-resident radix banks
that share the KV pool with active requests; a returning conversation whose snapshot survived is served
from it. We added a second tier behind those banks. When the engine retains a snapshot, the cache copies
its pages to pinned host memory in the background (write-back: the request path never waits for the
copy). When the engine later evicts the snapshot to make room, the host copy is what remains. When the
conversation returns, the cache restores the pages through the engine's own fill path in milliseconds and
the turn proceeds as if the eviction had never happened. The crate is ds41rt-hostcache; the
engine binding is three hooks (on retain, before evict, before prefill) and a CUDA copy engine.
This builds directly on two earlier pieces of our work: the DS41RT fleet report (the RTX 5090 coordinator plus four DGX Spark expert ranks this page measures) and the no-tax write-behind disk tier for GLM-5.3-Flash, whose design decisions (eviction-driven stores, the engine's own cache as the source of truth, an idle engine writes nothing, everything unit-tested engine-free) were carried over and adapted to a snapshot-based engine and a RAM tier.
We designed the benchmark's agent workload from our own usage rather than from a synthetic profile. The numbers below are mined from one day of our coding-agent traffic through the fleet (2,003 requests from 35 agent threads, anonymised; no message content was read, only token counts and timestamps).
Two rarer events matter for a cache. Compaction: about one turn in 200 the agent summarises its context and the prefix shrinks (to ~72% of its size in our sessions), so the next turn shares only part of the old prefix. Cold sessions: a new thread starting with a large prompt, or an old one returning after a long idle; 8 of our 35 threads started at 8k tokens or more and 9 resumed after ten-minute-plus gaps. Concurrency in our own use is modest, a median of 4 requests in flight and a p90 of 14, so the benchmark ramps from 8 to 48 agents to bracket it. A second, smaller client on the same fleet (an assistant with an 18k-token fixed system prefix) shows the same shape at a smaller scale: 74% warm continuations, 23% cold starts, and, on an unloaded engine, 90% of all its prompt tokens served from cache, rising above 96% once prompts exceed 64k tokens.
The benchmark's agent loop (§7.5) therefore models: a 30k-token code base as the starting prefix, a ~1.6k-token tool result plus the ~100-token reply appended per turn, a 5–30 s tool gap between turns during which the agent holds no request slot, and K such loops running concurrently on the same engine.
The tier has one governing rule: with no memory pressure the cache must be invisible, and under pressure the cost it adds must always be smaller than the cost it saves. Everything else follows from that rule and from a second one we learned the hard way on the disk tier: never make the engine wait for the cache, and never let the cache fail a request.
cudaMemcpyBatchAsync
of a few dozen extents), not as tens of thousands of per-segment calls; on this fleet the difference is
17 GB/s versus 57 GB/s over the same PCIe link.From the user's chair the only number is the wall-clock time until the reply is complete: the queueing, the prefill of whatever the engine had to recompute, and the reply itself. The chart is the average of that time over every served turn in a 240-second window, for K concurrent coding agents running the §3 loop.
At 8 and 16 agents the two lines sit on top of each other (2.8 vs 3.0 s and 10.2 vs 8.4 s): the GPU holds every agent's prefix, nothing is evicted, and the cache has nothing to do. At 32 agents the working set no longer fits. Without the cache the average turn takes 86 s, because most turns are re-prefilling a 30k+ token prefix behind a saturated prefill lane; with it the average is 14 s and the fleet serves 3.1× as many turns. That is the same hardware doing three times the work; the remaining 14 s is decode plus queueing at the engine's 16-slot front door, not prefill (TTFT p50 1.11 s; §7.4). These are the rerun's numbers, with the load model retrying the front door's refusals the way a real agent does (105 retries off, 47 on, zero errors); the first pass, which counted refusals as errors, read 2.3× and 23 s. At 48 agents the gap widens again: 272 turns vs 81, an average of 28 s vs 131 s, with the baseline down to 13% cache hits; a handful of turns on each arm (16 off, 5 on) exhausted the 20-retry backpressure budget at the front door and count as errors (§7.4).
Source: the B5 agentic ramp cell, steady segments, served turns only (§7.5). "Average" is the arithmetic mean of client wall time per turn; medians and percentiles are in the §7.5 table.
Every prompt token is either served from cache, a lookup and a byte restore, or recomputed, a full pass through the model on the prefill lane. Recompute is GPU time: billed as uncached input tokens by a provider, paid as fleet capacity by a self-hoster. The share of prompt tokens served from cache is the economic number.
Hit share = Σ prompt_cache_hit_tokens ÷ Σ prompt_tokens from the engine's usage accounting, steady segment.
Every cell ran the same engine build with the cache on (24 GiB host quota) and off (quota 0, the same image), so the comparison isolates the cache. Real prose and real source code, temperature 0, streaming; TTFT is the client-observed first content delta; decode is completion tokens over the wall time after the first token; the engine's counters bracket every segment; each arm switch is a coordinator relaunch and the expert ranks stayed up throughout.
| Cell | Question | Shape | Result |
|---|---|---|---|
| B1 · no overhead | Does the cache cost anything when nothing is evicted? | Cold prompt ladders 15k–650k tokens, prose and code; decode at 1, 3 and 6 streams; each arm from a fresh launch | No measurable overhead: matched A/B/A ladders put prefill at 0.99× (on / off) on average, within ±5% per rung, and decode higher with the cache on (§7.3a) |
| Pilot · thrash | Many small sessions cycling through the 24 retained banks | 48 sessions of ~10k tokens, 6 users, 15 min per arm | 485 vs 174 revisits; TTFT p50 0.51 vs 2.28 s; decode per stream 19.7 vs 7.2 tok/s; 0 errors |
| B2 · steady rotation R1–R5 | Sustained revisits at growing working sets | R1 10k×48; R2 170k×12; R3 300k×12; R4 679k×6 (+2k/visit); R5 300k prose+code×24; 3–6 users | See the table below: 5–64× the revisits once the working set exceeds the GPU pool; neutral where it fits (R2) |
| B3 · recovery | Does the engine come back clean after pressure stops? | The sessions of the R3 pressure cell revisited after 60 s idle, then again after a 600 s observe window | Every primed session revisited after 60 s restored from host RAM (TTFT 0.25–0.6 s; the six revisits of sessions the harness had never primed prefilled cold, as they must); after the 600 s window all 12 hit (TTFT p50 0.73 s, restores mean 15.7 ms, 99.8% hits); 0 errors; host-cache footprint 5.9→6.2 GB of the 24 GiB quota |
| B4 · practical ceiling | How far can one coordinator be pushed? | Sessions ramp at 300k (6, 8 users); context ramp at 679k (2, 3 users) | 6 users at 300k (relaxed-bar rerun, decode bar 8 tok/s): TTFT p95 1.6 s, 12.5 tok/s per stream, 0 errors; the plan's 20 tok/s bar stops the ramp at 6 users on an engine decode ceiling no cache changes. 8 users and three active 679k contexts hit the GPU-pool wall (§7.4) |
| B5 · agentic ramp | The §3 workload at 8, 16, 32 (48) concurrent agents | 30k-token code base, +1.6k tokens per turn, 5–30 s tool gaps, 240 s per point | §5 and §6. Comparison of record = the rerun with the load model retrying the front door's refusals; the first pass (refusals as errors) is the lower bound. 48 agents run separately as one point on both arms (§7.5). |
| 1M rung | The largest single conversation | 1,000,007-token prompt, cold then after eviction | 270.9 s cold; 1.09 s restored (893 MB), 100% hit |
| Equivalence | Is a restored conversation the retained one? | 21k, 72k, 309k tokens: cold vs restored, temperature 0 | Identical output; restores 20, 68, 301 ms against 5.4, 13.8, 65.5 s cold |
| Conversation | Cold prefill | Restore from host RAM | Ratio |
|---|---|---|---|
| ~10k tokens (rotation revisits, p50, includes the new turn) | 2.28 s | 0.51 s | 4.5× |
| 54,620 tokens | 8.27 s | 46 ms | 180× |
| 320,840 tokens | 57.0 s | 359 ms | 159× |
| 1,000,007 tokens | 270.9 s | 1.09 s (893 MB) | 249× |
| Cell | Working set | Revisits off | Revisits on | TTFT p50 off / on | TTFT p95 off / on | Decode p50 off / on (tok/s) | Hit share on | Restore mean |
|---|---|---|---|---|---|---|---|---|
| R1 | 48 × ~10k, 6 users (thrashes the 24 banks) | 112 | 544 | 2.16 / 0.59 s | 4.38 / 1.02 s | 7.3 / 17.3 | 98.6% | 1.1 ms |
| R2 | 12 × 170k, 3 users (fits the GPU pool) | 170 | 499 | 0.63 / 0.64 s | 1.22 / 1.18 s | 29.7 / 34.8 | 96.0% | 12.0 ms |
| R3 | 12 × 300k prose, 6 users (3.6M tokens, beyond the 2.2M-token pool) | 7 | 448 | 122 / 0.90 s | 288 / 1.69 s | 0.6 / 15.1 | 98.2% | 18.6 ms |
| R4 | 6 × 679k over 3 users (4.1M tokens; three active at once), growing by 2k per visit | 3 | 216 | 356 / 1.44 s | 532 / 2.53 s | 0.5 / 17.4 | 97.2% | 41.6 ms |
| R5 | 24 × 300k prose and code, 6 users (7.2M tokens, beyond the 24 GiB host quota) | 6 | 454 | 160 / 0.87 s | 331 / 1.62 s | 0.6 / 14.9 | 98.3% | 17.8 ms |
Off windows are 300 s and on windows 600 s by plan (the off arm's revisits are minutes long); R2's off arm counts a shorter window too, so read its revisit columns as "the same" rather than 3×. R4-on served 135 of its 216 steady visits; the other 81 were refused by the engine's admission once the three active contexts outgrew the GPU pool (§7.4). No cell recorded a restore timeout or an uncached drop; the only restore failures were R4's 86 abandons for want of device room, counted, after which the turn prefilled cold and failed admission exactly as it would with the cache off.
| Corpus | Prompt tokens | TTFT off | TTFT on | on / off | Prefill off (tok/s) | Prefill on (tok/s) |
|---|---|---|---|---|---|---|
| prose | 19,734 | 3.61 s | 4.50 s | 1.25 | 5,468 | 4,593 |
| prose | 69,343 | 13.02 s | 17.01 s | 1.31 | 5,325 | 4,856 |
| prose | 134,578 | 35.25 s | 37.27 s | 1.06 | 3,818 | 4,459 |
| prose | 650,943 | 173.03 s | 172.51 s | 1.00 | 3,762 | 3,715 |
| code | 14,725 | 2.75 s | 3.02 s | 1.10 | 5,362 | 5,237 |
| code | 65,775 | 11.71 s | 11.19 s | 0.96 | 5,619 | 5,501 |
| code | 133,819 | 34.56 s | 28.03 s | 0.81 | 3,872 | 5,045 |
| code | 510,215 | 128.97 s | 130.89 s | 1.01 | 3,956 | 4,044 |
Single samples per rung from fresh launches. The 0.81–1.31 spread at the small rungs is the engine's run-to-run variance after churn, in both directions. Aggregate decode at six streams: prose 105 vs 110 tok/s, code 77 vs 129 tok/s (off / on); the cache arm was the fresher engine state in the code run.
Two further passes alternated off and on on the same, already-churned engine (a fresh cold ladder and three single-stream decode samples per arm per pass), so both arms see the same device state. Prefill throughput is within ±5% per rung in both directions and averages 0.99× (on / off) over the eight rung pairs; decode is higher with the cache on in both passes. There is no measurable overhead.
| Pass | Prompt tokens | Prefill off (tok/s) | Prefill on (tok/s) | on / off | TTFT off | TTFT on |
|---|---|---|---|---|---|---|
| 2 | 18,776 | 3,767 | 3,459 | 0.92 | 4.98 s | 4.91 s |
| 2 | 69,469 | 5,224 | 5,241 | 1.00 | 13.30 s | 15.49 s |
| 2 | 159,764 | 5,070 | 5,000 | 0.99 | 31.51 s | 31.84 s |
| 2 | 676,809 | 3,747 | 3,949 | 1.05 | 180.63 s | 138.15 s |
| 3 | 16,592 | 3,502 | 3,639 | 1.04 | 4.74 s | 5.15 s |
| 3 | 91,383 | 5,325 | 5,242 | 0.98 | 17.16 s | 13.93 s |
| 3 | 167,018 | 5,257 | 5,057 | 0.96 | 31.77 s | 33.34 s |
| 3 | 650,439 | 3,866 | 3,841 | 0.99 | 168.23 s | 175.82 s |
| Single-stream decode, mean of three samples per arm: pass 2 off 45.6 / on 70.8 tok/s; pass 3 off 63.6 / on 68.9 tok/s. The off arm of pass 2 ran first after the heaviest cells and shows the post-churn decode dip that both arms exhibit (§9). | ||||||
Each pass draws a fresh salted prompt per rung, so the token counts differ between passes for the same character size. Prefill tok/s = prompt tokens ÷ TTFT for a single cold stream. A first A/B/A pass was discarded: an orchestration fault had left the cache on during its "off" arm, so it measured on against on (§9, operational traps); passes 2 and 3 above ran after the fix with the arm verified at each switch.
Two engine limits set the ceiling in these runs, both reached sooner with the cache because it makes the turns before them fast. First, admission under GPU-pool over-commit: when the sum of the active contexts exceeds the pool (three 700k-token sessions, or eight 300k sessions on the B4 ramp), the engine has nothing left to evict and fails the request instead of queueing it. Second, the HTTP front door: bursts above the engine's request concurrency are refused outright rather than queued, which at 32 agents cost each arm 30 refusals, and admission queueing at that 16-slot limit is part of the cache-on arm's 23 s. Neither depends on the cache; both are patched in our fork and reported upstream (§8, §9). The admission patch has been through one fleet smoke on the same R4 shape (three clients on 679k-token sessions, 600 s): 287 turns served with zero HTTP 500 and 57 admissions deferred instead of refused, against 135 served and 81 refused on the unpatched build; TTFT p50 1.41 s, p95 3.0 s, restores mean 31 ms, 97.8% hit share, output equivalence 3/3. The front-door patch (429 with Retry-After after a bounded wait) passed its 134 unit tests and a fleet overflow burst: 200 concurrent requests against the 16-slot front door, 97 served, 103 answered 429 with Retry-After after the 30 s bounded wait, no 503, no 500. Neither patch is part of the measured candidate (rc4); both are on rc6.
| Agents | Arm | Turns served (240 s) | Avg s per turn | TTFT p50 | TTFT p95 | Hit share | Decode p50 (tok/s) | Backpressure retries | Turn latency p95 (incl. backoff) |
|---|---|---|---|---|---|---|---|---|---|
| 8 | off | 103 | 2.76 | 0.52 s | 1.5 s | 96.4% | 50.0 | 0 | — |
| 8 | on | 102 | 2.96 | 0.48 s | 2.2 s | 96.3% | 49.5 | 0 | — |
| 16 | off | 145 | 10.20 | 0.67 s | 9.0 s | 90.4% | 15.0 | 0 | — |
| 16 | on | 164 | 8.41 | 0.67 s | 4.0 s | 96.2% | 13.0 | 0 | — |
| 32 | off | 84 | 86.31 | 43.52 s | 56.0 s | 42.7% | 2.1 | 105 | 173 s |
| 32 | on | 264 | 14.04 | 1.11 s | 13.1 s | 96.0% | 8.6 | 47 | 38 s |
| 48 | off | 81 | 130.72 | 73.20 s | 83.9 s | 12.9% | 1.4 | 501 | 250 s |
| 48 | on | 272 | 28.19 | 13.60 s | 23.7 s | 94.2% | 6.9 | 334 | 69 s |
Rerun on the production candidate with the load model retrying the front door's refusals (jittered backoff; 105 retries off, 47 on; zero errors). The plan's bar (revisit TTFT p95 ≤ 10 s) still fails at 32 on both arms: the off arm at 56 s on prefill queueing (every revisit is a full re-prefill), the on arm at 13.1 s on admission queueing at the engine's 16-slot front door. Neither is a cache effect. The first pass, which counted the refusals as errors (30 per arm), read 87 vs 199 turns and 85.8 vs 23.4 s; its raw cells are kept beside the rerun's. 48 agents: run separately as a single point on both arms (the ramp stops at 32): 272 vs 81 turns, 28.2 vs 130.7 s average, 94.2% vs 12.9% hit share; 16 turns off and 5 on exhausted the 20-retry backpressure budget and are counted as errors.
Everything needed is public. The engine and the cache are upstream DS41RT; the cache has been included since v6; the harness, corpora builder and report assembler are in our recipes repository.
| What | Where |
|---|---|
| The engine | tpurtell/ds41rt v2 (revision 9477b6e), the base of every build here; the expert-rank image is upstream's, unmodified. |
| The cache | Upstream in tpurtell/ds41rt: merged as PR #4 on 17 September 2026 and released in v6. This row originally named our fork's branch hostcache/rc4, the production candidate measured in §5–§7: crate rust/crates/ds41rt-hostcache, binding ds41rt-daemon/src/v41_native_serve/prefix/host_cache.rs, CUDA runtime shim ds41rt-ffi/src/cuda_runtime.rs. The rc6 additions, the admission requeue (issue #2) and the bounded front-door queue (issue #3), are upstream in v6 too. Additive: no upstream file changes behaviour with the cache off. |
| Upstream issues | #2 admission fails with HTTP 500 under GPU-pool over-commit instead of queueing; #3 the HTTP front door refuses with 503 instead of queueing with backpressure. Both are fixed upstream in v6 (commits 5f487c14 and 7c35f3e5) and closed. |
| The fleet and the 5090 port | The DS41RT fleet report: build, probe, preflight and launch scripts, gates and runbook (hughmadden/ds41rt-rtx5090, release 1.0.0). |
| The harness | hughmadden/ds41rt-rtx5090, bench/hostcache/ at release 1.0.0 (MIT): hc-rotate.py (rotating and agentic sessions, 503/429 backpressure), hc-ladder.py (cold/reuse ladders), hc-equiv.py (cold-vs-restored equivalence), hc-matrix.sh (the resumable matrix driver with per-point bars), hc-build-corpus.py (deterministic prose and code corpora from real text), hc-report.py / hc-ladder-report.py / hc-final-report.py (reducers), hc-mock-server.py + test-hc-rotate.sh (offline mock and the harness's own 125-check suite), and PLAN.md, the bench plan as run. Addresses and paths are placeholders; nothing else differs from what ran. The raw JSONL cells are available on request. |
# coordinator (the cache arm); the off arm is the same image with the quota unset or 0
DS41RT_HOST_CACHE_BYTES=25769803776 \ # 24 GiB pinned host quota
DS41RT_HOST_CACHE_COPY_BUDGET_MS=1000 \ # eviction waits up to 1 s for an in-flight store
<launch coordinator> --prefix-cache-entries 24 --concurrency 16 --max-context 1048576 \
--prefill-batch-tokens 1024 --memory-reservation 97% # dSpark on, FP4 KV
# restore copy budget: --host-cache-restore-budget-ms 500 (default); store pacing: --host-cache-store-pace-ms 0 (default)
# counters: GET /v1/stats -> host_cache.{restores,restore_failures,restore_timeouts,evict_drops_uncached,
# copy_submissions,device_evictions,prefill_holds,...} and host_cache_config
# corpora (once), then the cells; every cell writes JSONL with the /v1/stats brackets per segment
python3 scripts/hc-build-corpus.py --out bench/corpus
scripts/hc-matrix.sh off b1 ; scripts/hc-matrix.sh on b1 # no-overhead ladders + decode rungs
scripts/hc-matrix.sh off b2-r3 ; scripts/hc-matrix.sh on b2-r3 # 24 x 300k prose, 6 users (R1..R5 likewise)
scripts/hc-matrix.sh off b5-agentic ; scripts/hc-matrix.sh on b5-agentic # K = 8,16,32,48 agent loops
python3 scripts/hc-equiv.py --base <coordinator> --sizes 21k,72k,309k --out bench/equiv.jsonl # exit 3 on divergence
python3 scripts/hc-final-report.py --draft report-draft.md --pilot pilot-report.md --out report.md
# measure: temperature 0, streaming, stream_options.include_usage; TTFT = first content delta;
# decode = completion_tokens / (wall - TTFT); hit share = prompt_cache_hit_tokens / prompt_tokens
This tier was designed on 14 September and was running on the fleet the same night, built by a small swarm of coding agents working from a design memo, with a reviewing model gating every packet, and it only reached the fleet after every suite passed engine-free. If you are porting it to another engine, this is what to copy and what to avoid.
No packet touched the fleet until it had passed against stubs. The crate ships a stub copy engine on a virtual clock (asymmetric bandwidths, wedged streams, error injection) and a simulator of the engine's device model (page pools, lanes, an interleaver that plays scheduler ticks as choices, the §3 workloads). Every packet added four suites against them, functional, concurrency, stability and soak, and the merged tree runs about 200 crate tests in seconds with no GPU, no model and no engine. That is what let independent agents deliver the pool, the snapshots, the copy engine, the cache facade, the simulator, the instrumentation and the two performance fixes in parallel, each cross-reviewed by a different model before the merge. The fleet's role was to confirm, not to discover; the one thing it did discover (below) was a pressure rule the simulator had not modelled, and the fix came back with a simulator test for it.
The stop criterion for an autonomous benchmark is the feature's goal, not the error count: a cache that is neutral where it must win has failed, even at zero errors.
null before its first sample (normalise it); an orchestration edit that dropped a file's
execute bit so every relaunch silently failed; a rotation harness invoked without its output flag so a
healthy build "failed" its smoke. Dry-run every harness invocation, and make the settle step assert that
the container actually restarted with the intended quota.The cache design, the crate, the engine binding, the harness and every measurement on this page are Turquoise Bay AI's work, carried out by a swarm of coding agents under a coordinating model. Sydney timestamps throughout.