Shared, Dedicated or On-Prem AI After Cloudflare's KV Cache Leak

What actually went wrong with GLM 5.3

According to Cloudflare's incident report, between August 27 and September 9, 2026, some customers using the GLM 5.3 model on Workers AI received responses containing data from other customers' requests made to the same model shortly before theirs. In total, 155 responses were affected, all of them answers to prompts longer than 681,280 tokens. Cloudflare says it found no evidence of deliberate exploitation.

Two problems combined. Cloudflare deployed the model with a 1,048,576-token context window, while the KV cache pool on its prefill GPUs had room for only 681,280 tokens, a mismatch the company says it knew about before launch. Then, in SGLang's disaggregated prefill/decode mode, a prompt too large for the pool was marked as rejected and swapped for a one-token placeholder instead of being aborted. The decode node received a 'transfer complete' message and assumed the previous cache had been overwritten, when most of the earlier user's state was still there.

The uncomfortable detail is that the engine bug was already public. Another user had reported it as SGLang issue #30233 on July 6, 2026, and Cloudflare only found that report while investigating. Once the problem reached Cloudflare, the response was quick: a HackerOne report arrived on September 8 at 23:11, an incident was declared on September 9 at 13:22, Product Security reproduced the leak with two independent accounts within 25 minutes, and the model was offline within 40 minutes. Service returned the same day with a corrected configuration and a patched engine.

Why a shared cache is a shared risk

The KV cache stores intermediate attention state so the model does not recompute it for every token. As Unite.AI explains, Cloudflare's setup hands the cache from prefill GPUs to decode GPUs over RDMA via Mooncake, and the cache is a shared pool of fixed-size pages that are not zeroed between requests. The engine alone keeps track of which request owns which page. When that bookkeeping fails, the next reader gets someone else's data.

Bugs are not the only route. Stanford researchers point out that cached prompts are processed faster than uncached ones, so if a cache is shared across users, an attacker can spot cached prompts from response times and learn about other people's inputs. They built statistical audits that detect prompt caching in real-world APIs. A later paper, CacheProbe, says most providers have moved to per-account or per-organization caching, but warns that gateways such as OpenRouter add a layer that could get around provider-level isolation, and that measuring time-to-first-token can rebuild cached prompts token by token.

Engine-level mitigations exist, but they are partial. vLLM's design docs describe hash-based prefix caching with optional per-request salting to keep cache reuse separate. A gateway security patch quotes the vLLM security guide calling prefix-cache reuse a timing side channel (CVE-2025-46570) and describing cache_salt as 'not an isolation boundary.' My reading: salting lowers the odds of accidental sharing between tenants, but it is a convention, not a wall.

What the big API providers promise, and what that covers

The major providers document their cache scope clearly. OpenAI's documentation says cached states live on individual machines and are not shared across organizations or reused across regional processing boundaries. Anthropic's documentation says different organizations never share caches, even with identical prompts, and that caches are isolated by workspace on the Claude API, Claude Platform on AWS and Microsoft Foundry, while Bedrock and Google-managed Claude on Google Cloud isolate only at the organization level.

That workspace versus organization distinction matters more than it looks. If your HR team and your sales team sit inside one organization on a platform that isolates only at that level, they share a cache scope. For many companies that is acceptable. For a company where personnel files and customer pitches must never meet, it is a design decision someone should make on purpose.

What no documentation can promise is the absence of bugs. Cloudflare's cache was not meant to be shared across customers either; an engine defect made it so. Knostic, which reported the issue, notes that the anomalous response came back as HTTP 200 with success: true and was billed in full at 103,604.64 neurons. Nothing in the response signalled a failure. A leak of this kind looks like a slightly odd answer, and most users would dismiss it as a hallucination.

Shared serverless inference: the cheap default with a ceiling

Shared serverless inference is the right answer for a lot of work. You skip capacity planning, pay per use, get access to large models immediately and let the provider patch the engine. For drafting marketing copy, summarizing public documents or coding help on repositories without secrets, I would not argue for anything heavier.

The ceiling is that your isolation depends on the provider's configuration discipline and on upstream engine code you cannot see. Cloudflare says GLM 5.3 was the only affected model because every other disaggregated Workers AI deployment has KV pools larger than its context length. That is reassuring, and it is also the point: tenant isolation hinged on one sizing number being right.

There is an irony in where this incident struck. Only very long prompts were affected, and long-context use is exactly where confidential material piles up: a full contract bundle, a quarter of board minutes, hours of transcripts. If a single leaked response from your workload would trigger a regulatory notification, a client conversation or a lawsuit, shared inference should not be your only option.

Dedicated cloud instances: isolated from others, not from yourself

A dedicated instance or private endpoint means only your organization's requests reach the inference engine. If a Cloudflare-style bug hits, the worst case is that one of your users sees another of your users' data. For most threat models that is a large improvement, because the external tenant is gone from the picture.

It is not zero risk. If HR and sales share one endpoint, the same class of bug crosses departments. You also inherit the operational decisions Cloudflare got wrong: matching advertised context length to KV cache capacity, pinning engine versions and tracking upstream fixes. In this case the relevant SGLang fixes were PR #36988 in v0.5.20, which retires aborted prefill requests before they reach decode, and PR #38935 in v0.5.21, which rejects them at admission. Someone on your side, or your managed provider's side, has to notice releases like these.

On cost, I could not find reliable sourced figures, so I will not quote any. Structurally, you pay for reserved capacity whether you use it or not, so the economics favour steady, predictable volume. A team that sends a few dozen sensitive documents a week will usually find a dedicated endpoint expensive per request; a team running continuous transcription or document processing is more likely to justify it.

On-premise local models: the strongest boundary and the heaviest lift

Running a model on your own hardware means data does not leave your network. The tenant boundary becomes your building and your access controls. For material that contracts or regulation forbid sending to third parties, this is often the only clean answer.

The price is operational. You need hardware, people who understand GPUs and inference engines, monitoring and a patching routine. The engines are the same ones the cloud uses, so the SGLang bug would exist on your server too if you ran the affected version and configuration. Open models you can run locally are often smaller than the largest hosted ones; for narrow tasks like summarization, extraction or transcript analysis that is frequently fine, for open-ended complex reasoning the gap can show.

On-prem is not for teams with nobody to own the stack. An unpatched local server running an old engine with a known cache bug is not safer than a well-run cloud service. Moving inference in-house puts the risk under your control; it does not make the risk disappear.

Matching the data to the deployment

The practical mistake is choosing one deployment model for the whole company. A better question for each workload: what would a single leaked response contain, and who would you have to tell? The research behind this piece did not cover specific regulations, so treat the list below as a practitioner's starting point and check it against your own legal obligations.

Meeting data is a useful test case because it is unstructured and people speak freely. A routine standup and a restructuring discussion pass through the same transcription and summarization pipeline, and transcripts are long, which pushes them toward the large-context territory where this incident occurred. Products such as Meeting AI offer cloud or on-premise deployment with a local LLM option, which lets an organization route sensitive meetings locally while sending routine ones to the cloud, instead of making one choice for everything.

• Public and marketing content: shared serverless is fine.
• Internal but low-sensitivity material (product docs, code without secrets): shared inference with organization or workspace-level cache isolation.
• Routine meeting recordings and transcripts: shared or dedicated, depending on who attends and what is discussed.
• Board, M&A, investor and personnel meetings: dedicated at minimum, on-premise where possible.
• Legal files, privileged advice and contracts under NDA: dedicated at minimum, on-premise where clients require it.
• HR material such as performance reviews, health information and disciplinary records: on-premise, or dedicated endpoints separated per department.

Questions to put to any inference provider

Whichever route you take, the Cloudflare case gives you a concrete checklist. These questions apply to hosted APIs, managed dedicated endpoints and your own internal platform team alike.

One more criterion that is easy to overlook: how a provider behaves after something breaks. Cloudflare reproduced, contained and published the root cause in detail, including its own prior knowledge of the sizing mismatch. A provider willing to write that kind of report is one you can reason about. A provider that has never published an incident is not necessarily safer; you just know less.

• At what level are caches scoped: organization, workspace, API key or individual request?
• Do requests pass through a gateway, and does that gateway preserve the underlying provider's isolation?
• Which inference engine and version do you run, and how quickly do you adopt upstream security fixes?
• Does the advertised context length fit within KV cache capacity on every node?
• How is KV cache page ownership tracked and tested, given pages may not be zeroed between requests?
• How would you detect a cross-tenant leak when the affected response returns as a normal success?
• Do you run a bug bounty, and what is your public incident disclosure policy?