Your AI provider recycles other customers' GPU memory and owes you the design doc

If you call a hosted model, your prompt is very likely written into GPU memory that held someone else’s prompt a moment earlier. It is then read back through bookkeeping you have never seen. Cloudflare’s write-up of its GLM-5.3 incident is the clearest account I have read of what happens when that bookkeeping slips. I came away convinced that a provider’s cache isolation design belongs in public documentation, next to the rate limits and the price per million tokens.

Start with the mechanics. Between 27 August and 9 September 2026, 155 responses from @cf/zai-org/glm-5.3 on Workers AI contained fragments of other customers’ requests. The model advertised a 1,048,576-token context window. The KV cache pool on the prefill GPUs held 681,280 tokens. SGLang sizes that pool at startup from whatever memory the weights leave behind, and context length is a separate setting, so nothing forced the two to agree. Cloudflare says it knew about the gap before launch and assumed the engine would refuse what it could not hold. In disaggregated mode, SGLang instead swapped the oversized prompt for a one-token placeholder and prefilled that. It shipped one page to the decode node and reported the transfer complete. Decode had reserved pages for the full prompt and attended over all of them. Every page except the first still held whatever earlier requests had left there, because pages are not zeroed between requests. Cloudflare describes that practice as common among inference engines.

What struck me was how polite the failure was. The client got HTTP 200, a correct prompt_tokens count and a coherent answer. The only tell was that the reply arrived faster than a prompt that size should allow. Cloudflare’s own pre-launch tests checked status codes and throughput, saw green and shipped. Another user had already filed the bug upstream as SGLang issue #30233 on 6 July. It took Gadi Evron of Knostic, with a reproduction script sent through HackerOne, to get it looked at.

Cloudflare makes a decent case that this was a one-off. It says the traffic looked like ordinary long-context use rather than probing and that it found no evidence of deliberate exploitation. It also says every other disaggregated model it runs has pools larger than its context length, so nothing else could reach the broken path. The response was quick: about 40 minutes from declaring the incident to taking the model offline, with a fix and a restore the same day. Add the Stanford audit by Gu et al. at ICML 2025, which found global prompt-cache sharing across users at seven API providers including OpenAI. After disclosure, at least five of them changed something, such as disabling cross-organisation sharing. On that reading, the system works: researchers find things and vendors fix them.

I read the same evidence the other way. In both cases an outsider had to discover, by measurement, a property the provider could have stated in a paragraph. The Stanford authors inferred sharing from timing because providers that advertised caching often did not say whether it was per user, per organisation or global. Evron found Cloudflare’s leak by sending prompts long enough to fall off a cliff that Cloudflare’s own logs printed at every startup. OpenAI’s Redis bug in March 2023, which may have exposed payment details for 1.2% of ChatGPT Plus subscribers active during a nine-hour window, belongs to the same family. Each involved shared infrastructure and a bug in tracking who owned which data. Across several vendors and several years, I think that pattern is a property of the architecture. An engine that reuses unzeroed pages across tenants for speed depends entirely on correct lifetime tracking across nodes, aborts, offloads and RDMA races. Cloudflare’s post lists several earlier fixes in exactly that class.

I think publishing the design costs a provider very little. Cloudflare has now effectively done it. Its post covers disaggregated prefill and decode over Mooncake, pages reused without zeroing and a per-transfer checksum flag. It also describes a page-tagging prototype that cost under 1% in throughput before it was shelved for reliability problems, plus a draft proposal to zero pages on release. A customer can weigh all of that, which is impossible with a provider that publishes nothing.

Before your next contract renewal, put two questions to your inference vendor in writing. First, at what scope is any cache shared? Second, does the engine verify page ownership before decode reads it, or does it trust the scheduler? If you run SGLang yourself, compare the KV pool size in your startup log with your configured context length on every prefill and decode node today. If the pool is smaller, upgrade to v0.5.22, released on 19 October, and lower the context length until it fits.

Prompted by Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang, Cloudflare Blog.