Prompt-pool privacy: what it means and what it doesn't

What is the K3 guarantee?

K3 is CloakAPI's cross-tenant isolation guarantee at the inference pool level. In shared AI infrastructure, multiple tenants' requests are often batched together for efficiency — a standard technique in transformer inference that allows GPU compute to be amortised across concurrent requests. Batching is invisible to the application layer but creates a potential surface for cross-tenant information leakage.

The K3 guarantee is: no CloakAPI request from Tenant A is ever included in the same inference batch as a request from Tenant B. Each tenant's requests are dispatched to the AI provider in isolated batches. The provider's scheduler may internally batch requests from different CloakAPI tenants if those requests happen to arrive at the provider simultaneously — K3 operates at the CloakAPI gateway layer, not at the model provider's inference layer.

This distinction matters and we return to it below.

What K3 protects against

Side-channel inference via shared batch state

Research into transformer batch processing (notably work by Nasr et al. on membership inference in LLM inference services, and follow-on work on timing-based side channels in shared GPU inference) has demonstrated that co-location of requests in a shared batch can create measurable side channels. Timing differentials, cache hit patterns, and in some implementations shared attention state can leak information about co-located requests.

By ensuring Tenant A and Tenant B are never in the same CloakAPI-level batch, K3 eliminates the primary surface for batch-level side channels at our gateway layer. An attacker who controls one CloakAPI tenant cannot use batch co-location to infer properties of another tenant's requests.

Cross-tenant cache poisoning

Some AI providers implement KV-cache sharing across requests with matching prefixes (prompt caching). If Tenant A's system prompt partially matches Tenant B's, and they share a cache, a carefully crafted prompt from Tenant A could poison the cache entry that Tenant B subsequently reads. K3 prevents CloakAPI from dispatching requests in a way that would cause our batching layer to create this situation — we do not share prefix-cache state across tenants at our layer.

Again: the provider's internal KV-cache may still be shared across all CloakAPI requests (since from the provider's perspective, all CloakAPI traffic originates from the same API key by default). This is addressed under "what K3 does not protect against" below.

What K3 does not protect against

Being explicit about limitations is more important than listing protections. Here are the threat classes that K3 does not address:

Same-tenant cache hits

K3 isolates across tenants. Within a single tenant, KV-cache reuse is permitted and expected — this is a performance feature that reduces cost and latency for repeated similar prompts. If your tenant has multiple users and those users issue similar requests, their requests may share cache state. This is a within-tenant privacy question, not a cross-tenant one, and it is governed by your application's data governance policies rather than CloakAPI's isolation model.

Model-level memorisation

A large language model trained on a corpus can memorise training data and reproduce it in generated outputs. If your data was part of a model's pre-training or fine-tuning dataset, the model may regurgitate it regardless of how the inference request is structured. CloakAPI's tokenisation ensures that sensitive spans in your prompts are not transmitted to the model. It does not and cannot affect what the model already knows from its training data.

Mitigation: use models whose training data provenance is auditable for your use case. Avoid sending output that might contain memorised sensitive data back to storage without scrubbing.

Provider-level cross-tenant inference

As noted above: K3 operates at the CloakAPI gateway layer. The AI provider receives all CloakAPI tenant traffic on the same API key (unless you use BYOK — see below). The provider's internal inference infrastructure may batch requests from different CloakAPI tenants. The K3 guarantee does not extend to the provider's internal scheduling decisions.

This is an inherent limitation of using a shared API key to a third-party provider. The solutions are BYOK (your own API key, which the provider associates with a single account) or complete-local mode (inference runs in your infrastructure).

Prompt injection via shared corpus

If two tenants use CloakAPI to process documents from a shared corpus (for example, a publicly available dataset), a malicious document in the corpus could contain embedded prompt-injection payloads. The injection targets the model's behaviour, not CloakAPI's isolation. K3 does not protect against this because the attack vector is the content of the data, not the batching structure.

Model fine-tuning on tenant prompts

This is the honest residual risk we are most uncomfortable with.

CloakAPI's DPA prohibits AI providers from using tenant request data for model training. Every AI provider in our routing table has agreed to an equivalent prohibition in their terms of service or via executed data processing agreements. However: the model provider controls their training pipeline. If they were to violate their DPA and fine-tune on CloakAPI tenant data, they would have access to the tokenised form of your requests — meaning they would see placeholder tokens, not PHI, but they would see the non-sensitive spans of your prompts and the structure of your requests.

How to read the K3 guarantee

Customer mitigations

Bring Your Own Key (BYOK)

BYOK lets you supply your own API credentials for each AI provider in the routing table. CloakAPI uses your credentials for your requests, so from the provider's perspective your traffic is a single-tenant account — your data is never co-mingled with other CloakAPI customers' data at the provider layer. BYOK also means the provider's data processing terms apply directly to your account, not mediated through CloakAPI's commercial relationship with the provider.

BYOK is available on every CloakAPI account at the flat 5% BYOK rate — there are no plan tiers. Configuration is in Settings → Providers → BYOK in the dashboard.

Complete-local mode

Complete-local mode routes inference to a model running in your own infrastructure — either a self-hosted open-weight model or a dedicated instance provisioned in your cloud account. No request leaves your environment. CloakAPI acts as a signing and receipt-issuing proxy only; the inference compute is yours.

Complete-local mode eliminates provider-layer risks entirely. It requires provisioning inference infrastructure (GPU compute) and accepting the operational overhead. For regulated industries with strict data residency requirements, it is frequently the appropriate architecture.

Complete-local mode is available on every CloakAPI account at the flat 5% local rate — there are no plan tiers. Email hello@cloakapi.io for architecture guidance.

How we think about this

K3 is a meaningful guarantee within a clearly bounded scope. We think it is worth having, and we think it is worth being explicit about its boundaries rather than presenting it as a comprehensive solution to inference privacy. The goal is to give customers an accurate picture of what they are buying so they can decide whether it is sufficient for their threat model — and if not, what additional measures (BYOK, complete-local, full tokenisation) they need to layer on top.

Privacy tooling that overstates its guarantees is worse than no privacy tooling at all, because it creates false confidence. We would rather lose a sale to a customer who needs stronger guarantees than we currently provide than win a sale from a customer who misunderstands what they bought.