No-tax disk persistence for GLM-5.3-Flash

Headlines

1 · What this is

GLM-5.3-Flash (320B MoE) in EXL3 TR3 4bpw, tensor-parallel across four DGX Spark nodes (GB10, 121.7 GiB unified memory each), vLLM 0.28.1rc1.dev617 on a pinned upstream commit (83252ea899) plus a twelve-patch persistence series, DFlash2 speculative decoding at k=7, --enforce-eager, fp8 KV, MAX_NUM_SEQS=4. The disk tier is per-rank NVMe with a sqlite index and per-rank coordinators. The serving configuration is unchanged from the published TP4 playbook; everything described here is additive.

2 · No disk vs this deployment

All cells measured with unique salted prompts, streaming TTFT, medians of two reps or better; engine variance ±5–10%. Protocols and probe scripts are in the repo.

3 · The five decisions that made it no-tax

The first version of this tier stored eagerly — every new chunk written on the request path, responses held for the store drain, synchronous all-rank admissions. It worked, and it taxed exactly the traffic it should protect: −11…−38% prefill, −37% decode at 256k, 36× write amplification. The no-tax property came from five decisions, each taken after measurement:

1. Stores are eviction-driven, never request-driven (patch 0009). The GPU prefix cache is the source of truth for what deserves durability: blocks copy to disk only when the cache evicts them, through a deferred-free sink that holds blocks outside the free queue while background copies drain. Nothing is offered, held, or admitted on any request path — zero store bytes measured at every context size. Demand-aware holding (decline and release under free-queue pressure) keeps the sink from ever stalling allocation — a lesson paid for by a real flood crash.
2. Reads are RAM-first with a size gate. The connector is consulted only beyond the GPU cache boundary; requests under 32k prompt tokens skip the disk entirely (a small RAM-miss recomputes cheaper than the scan costs). The gate keys on request size, never miss size, and its skips are counted, not silent.
3. Lookup misses must be nearly free (patches 0010/0010b). The remaining tax was the lookup scan: per-key index reads on four ranks on the scheduler's critical path, −22% prefill at 64k. An in-RAM durable-key index (authoritative for negatives, maintained on every durability transition) plus in-process dispatch for the rank-local coordinator leg took the residual to ~0% (2,026 vs 2,028 tok/s OFF). Unit tests pin both: a counting connection wrapper proves zero index I/O on the fast negative.
4. An idle engine writes nothing (patch 0011). A free-pool watermark gates the sink: above it, holds are declined. Census-verified zero bytes during a full prefill/decode smoke at OFF-parity speeds; the only measured cost under a full pool is −4.6% decode while copies drain. Stale-content durability moved to its own mechanism rather than weakening this gate.
5. Stale content flushes in idle time, at decode-zero cost (patch 0012). APC blocks untouched for a configurable age (1 hour default) are written during decode-free steps only — prefill windows and finish drains — as in-place copies fenced against reuse. A decode step never pays for persistence by construction. This is what gives cold sessions durability without memory pressure.

Two engineering disciplines carried the campaign: the patch series is hash-pinned (before/after hashes per file, an AST suite binding the patched methods and running them engine-free), and every measured claim above has a committed probe. The flood-crash and fence-cleanup bugs found in bring-up were caught by that suite or its live smoke — the suite is part of the design, not an afterthought.

4 · Measured evidence

4.1 Prefill and decode across modes

4.2 The flood and the restore

400 unique 9k sessions (3.6M tokens) against the 3.39M-token pool, then a re-request of an early session while the engine is busy:

4.3a · Pressure bench on a small pool (2026-09-14)

KV pool deliberately capped to 944,871 tokens (12 GiB, ~28% of normal) so eviction is continuous. A 200k session (21% of the pool) was seeded, then the fleet carried ten minutes of sustained C=4 production-shape load (rotating unique 8k–32k prefills, 300-token decodes) followed by a 16×120k eviction flood and a cold restart:

Read together with §4.2: whole-session disk restore is proven at 9k (1.42 s) and as mid-prefill partials; at 200k under sustained pressure the honest result is partial-to-none, by the design’s bounded-write economics — the fix is the flush command (§6) or sized windows, not a defect. Concurrency behaviour itself was clean throughout: no stalls, no errors, decode at parity under eviction load.

4.3 Standard benchmark and production smoke

Local Inference Labs standard bench (C1–C6, 700-token outputs, salted prompts): 38.3 / 73.4 / 51.6 / 52.9 / 64.3 / 69.2 aggregate tok/s, zero failures — within variance of the published no-persistence baseline; the tier does not disturb the standard bench.

Production smoke (agentic ladder, 300-token outputs): 32k → 15.4s TTFT, 2,030 prefill tok/s; 128k → 66.6s, 2,008; 256k → 141s, 1,897; 400k (404,745 tokens, the largest single request this fleet has served) → 261s, 1,549 tok/s, decode healthy; a 4×32k concurrent burst ran clean at 393–1,396 prefill tok/s per stream. The configured max_model_len of 850k is refused at admission in 1.1s — the practical single-request ceiling is ~405k tokens, set by unified memory, independent of persistence; whether the baseline playbook shares that wall has not been measured and should be (same ladder, persistence off).

5 · Open item, stated plainly

Full cold restore of large sessions from disk. Disk restore is proven at 9k (1.42 s) and as a 423,936-token partial inside a live 800k prefill, and cache retrieval is proven at 200k (3.0 s). A complete 200k+ session restored cold from disk after a restart recomputes today: the lookup traces show the earliest chunks absent, and the census shows why — under a heavy flood the bounded capture window (4,096 held blocks, 8 keys/step drain) lands gigabytes, not the whole evicted set, and the mamba groups' chunks are visible to the store but not to the flusher's scan population. The dials exist (eviction_hold_cap, eviction_store_per_step, idle_flush_per_scan) and the mechanism is proven in both directions; whole- session coverage at flood scale is engineering, not architecture. Until it lands, quote disk restore at the tested sizes.

6 · Reproduce

# engine: vLLM 83252ea899 + patches/persistence-flash/0001..0013 (hash-pinned manifest;
# AST suite test_native.py, 23 tests, runs engine-free)
# package: persistence/recipe_persistence (bind-mount overlay)
# configs: PERSISTENCE=on  PERSIST_STORE_MODE=evict_only
#   PERSIST_MIN_DISK_LOOKUP_TOKENS=32768  PERSIST_LOOKUP_KEYS_PER_STEP=2048
#   PERSIST_EVICTION_STORE_HIGH_WATERMARK=<~20% pool blocks>   # pressure gate
#   PERSIST_IDLE_FLUSH_STALE_SECONDS=3600 / _PER_SCAN=32 / _SCAN_SECONDS=5
# measure:
python3 benchmark/bench_c1c6.py --url http://<head>:8888 --model <name> --rounds 3
python3 benchmark/ab_matrix.py --modes off,evict --sizes 8192,65536,262144 --conc 1,4
python3 benchmark/probe_evict_refill.py --n 400 --tokens 9000 --settle 45

Public lane: github.com/hughmadden/GLM-5.3-Flash-NVFP4-1M-KV-4x-DGX-Spark under exl3-persistence/. Sydney timestamps throughout.

7 · Credits

• Tony (d2)Wild — the TP4 DGX-Spark serving playbook this recipe extends (launcher, memory ritual, baseline configuration).
• Mia AI Lab — the GLM-5.3-Flash EXL3 TR3 4bpw build and engine lineage (NoPE sparse-MLA, EXL3 kernels).
• brandonmusic — author of the EXL3 TR3 4bpw quantisation recipe this fleet serves (the quant the Mia-AiLab build packages).
• Local Inference Labs — Spark-optimised GLM-5.3 builds, the LMCache design lineage that informed the lookup-overlap decisions, and the standard benchmark.
• The vLLM project and community — the engine and the community fixes carried into the series.

Full per-component attribution in the repository's CREDITS.md. The persistence tier — patches, package, probes, this report — is Turquoise Bay AI's additive work, committed with hash-pinned provenance.