Caching

The gateway keeps a workspace-scoped response cache so repeat requests don't re-hit an upstream provider. A cache hit costs $0 and never reaches the provider.

Reach for it through the SDK rather than hand-written headers — the options are typed, validated at the call site, and the outcome comes back parsed:

import { AiSetu, promptCacheStats } from '@ai-setu/client';

const client = new AiSetu({
  cache: { policy: 'support-faq', namespace: 'tenant-42' },
  onCache: (info) => metrics.increment(info.hit ? 'cache.hit' : 'cache.miss'),
});

const res = await client.chat.completions.create({
  model: 'openai/gpt-4o-mini',
  messages: [{ role: 'user', content: 'What is AI Setu?' }],
});

client.lastCache; // { status: 'hit', hit: true, semantic: false, ageSeconds: 42 }
promptCacheStats(res); // { inputTokens: 1000, cachedTokens: 3000, writeTokens: 0, ratio: 0.75 }

Per request, client.withCache(...) takes the same options and merges them over the client defaults:

await client.chat.completions.create(
  { model: 'openai/gpt-4o-mini', messages },
  client.withCache({ ttl: '2h', tags: ['sku:123'] }),
);

An invalid SDK option throws rather than being sent: the gateway drops a malformed policy name or tag silently, which leaves caching that appears configured but does nothing.

Which cache to reach for

Two different mechanisms share the word "cache". They stack, and they are worth adopting in this order:

Prompt caching discounts the unchanging prefix of a prompt while still running the model — no staleness risk at all, and typically the largest saving on agent traffic. Response caching substitutes a stored answer for the provider call entirely. Reach for semantic matching last; see its caveats below.

The rest of this page documents the headers themselves, for callers on a raw HTTP client or a language without an SDK:

curl https://gateway.aisetu.ai/v1/chat/completions \
  -H "Authorization: Bearer $AI_SETU_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-ai-setu-cache-policy: support-faq" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "What is AI Setu?"}]
  }'

Check the outcome on the response header x-ai-setu-cache.

Prompt caching

Distinct from everything else on this page: prompt caching does not substitute a stored answer. The model still runs and the output is still fresh — the provider just skips re-prefilling an unchanged prefix (system prompt, tool definitions, settled conversation) and discounts those tokens. That makes it safe in a way response caching is not, and it needs no opt-in.

It is also easy to miss, because the saving shows up only in the usage counters:

import { promptCacheStats } from '@ai-setu/client';

const res = await client.chat.completions.create({ model, messages });
promptCacheStats(res);
// { inputTokens: 1000, cachedTokens: 3000, writeTokens: 0, ratio: 0.75 }

ratio is the share of the prompt that avoided a full-price prefill. The helper reads both the OpenAI shape (usage.prompt_tokens_details.cached_tokens) and the Anthropic shape (usage.cache_read_input_tokens), so callers do not branch on provider.

To keep the discount, keep the stable part of the prompt first and unchanged. A timestamp, a request id, or a shuffled tool list at the top of the system prompt breaks the prefix and costs the discount on every call. On the Anthropic path the gateway places cache breakpoints for you — at the last tool definition, the last system block, and the second-to-last message, so the growing tail of a conversation stays outside the cached prefix. A caller who places their own cache_control markers is left alone.

Exact-match caching (default)

By default, a request is cacheable when temperature is unset or 0 and it carries no tools. A request with a non-empty tools array is an agent step and is cached only when you opt in — see Caching inside agent loops. The cache key is derived from the workspace, an optional namespace, and a hash of the model plus the full request body — so identical requests hit, and anything different (a different model, a different message, a different tool list) misses. Cache entries default to a 600 second TTL and can be set up to a 24 hour maximum via a request header.

An unrecognized x-ai-setu-cache value fails open to default behavior rather than erroring.

Semantic caching (opt-in — read the caveats first)

Setting x-ai-setu-cache: semantic adds a similarity search on top of exact matching: on an exact miss, the gateway embeds the latest user message and serves the best stored response whose cosine similarity clears a threshold. Everything else in the request — prior conversation, model, temperature, tools, response format, workspace, namespace — must still match exactly, so paraphrased questions can hit but different contexts never cross-match. Semantic mode implies caching even at temperature > 0; no force header needed.

You can also turn it on without the header by setting semantic: true on a cache policy. The two are equivalent — a policy just means call sites do not each have to send the header. x-ai-setu-cache: off overrides both.

The threshold defaults to 0.8. A policy may set its own similarityThresholdBp (basis points, so 9000 = 0.90, minimum 5000), which takes precedence. x-ai-setu-cache-threshold adjusts it per request, and the header may only ever raise the bar — in every keying mode, whether or not a policy set one. Lowering it is what lets a loosely-related answer through, so that decision stays with whoever operates the workspace and never with whoever sends the request.

A few things never enter the semantic store: non-text content (images, audio) and messages over roughly 2,000 characters — these degrade to exact-match-only caching instead. Under stable-context keying the non-text rule is stricter still: any non-text content block anywhere in the conversation excludes the turn, not just in the question being matched. See below.

Conversations containing tool calls or tool results are also excluded by default, because a tool result carries live, caller-specific state. Detection is structural rather than by field name, so it holds across the shapes below whichever endpoint you post them to:

It is not a claim about every shape any provider has ever defined. A Bedrock-Converse body (toolUse / toolResult blocks) posted to /v1/messages is not recognised as tool traffic under strict keying; under stable-context it is excluded anyway, because the content-block allowlist below rejects block types the gateway does not recognise.

That one exclusion is relaxable, but only under strict keying, and only for the OpenAI tools shape: if every tool invoked in the turn is declared { "safe": true } (per-tool declarations), the turn becomes eligible — and the tool result still forms part of the exact prefix match, so it can only ever be reused for a caller whose conversation was identical. The Anthropic and legacy-function tool shapes have no way to express that declaration, so turns using them are excluded.

Under stable-context keying the declaration is ignored and tool turns are always excluded; see below for why. The deployment default (GATEWAY_SEMCACHE, on) can disable the embedding step entirely, in which case semantic degrades to plain exact-match caching. Embedding cost is platform-absorbed — a semantic hit costs $0, same as an exact hit.

Semantic-cache quality depends on the embedding model behind it, which is currently English-leaning; non-English paraphrase matching may be weaker than exact-language matching. Exact-match caching is unaffected by this and works the same regardless of language.

Relaxed keying for voice traffic

By default, two requests only count as "the same conversation" if every preceding message is identical. That is the safe rule, and for voice it is usually the wrong one: a speech transcript differs on every call — names, hesitations, recognition noise — so no two callers ever share a conversation prefix, and semantic hits collapse to the first turn.

A policy can opt into semanticKeying: stable-context. The conversation history is then left out of the match entirely, and what identifies the conversation becomes:

• your system (or developer) messages,
• your tools array, and
• whatever you send in x-ai-setu-cache-context.

curl https://gateway.aisetu.ai/v1/chat/completions \
  -H "Authorization: Bearer $AI_SETU_API_KEY" \
  -H "x-ai-setu-cache-policy: voice-faq" \
  -H "x-ai-setu-cache-context: brand:acme.support.en" \
  ...

Two details worth knowing:

• Entries stored under one keying mode are never served to a request using the other, so switching a policy's mode cannot mix the two. It does mean the entries written under the old mode stop being used and expire on their own.
• x-ai-setu-cache-context is required in this mode. If it is missing, blank, or something the gateway cannot parse (it must be lowercase letters, digits, ., _, - or :, up to 128 characters), the request silently falls back to strict keying. There is no shared "no context" bucket — landing in the widest bucket has to be something you asked for by name.
• A policy using this mode must set similarityThresholdBp explicitly. With the transcript out of the cache key, the similarity bar is the only remaining barrier, so it is not something to inherit from a platform default.

Tool turns are always excluded in this mode, including tools you declared { "safe": true }, in every wire shape listed above — OpenAI tool_calls and role: "tool", legacy function_call and role: "function", and the Anthropic tool_use / tool_result blocks (the last of which ride on a role: "user" message, so "no tool role in the conversation" is not the same thing as "no tool traffic"). That declaration was made under strict keying, where it means "the presence of this tool doesn't disqualify the turn" — the tool result itself was still part of the match. Under stable-context the result is not in the cache key at all, so honouring the declaration would let two callers who looked up different orders be served each other's answers. Tool turns fall back to exact-match-only caching here.

So is anything else the cache key does not cover. Only the content types the gateway can key on are allowed through in this mode, which today means text. A block of any other type — an image, an audio clip, a document, a thinking block from extended thinking, or a type introduced by a provider after your gateway was built — outside your system messages excludes the turn, because stable-context drops the conversation from the key and those bytes would be invisible to it. So does a content field that is neither a string nor an array of blocks. New block types therefore cost you a cache hit rather than leaking one.

If you run extended thinking, expect thinking blocks to exclude most multi-turn conversations from this mode.

Everything else still applies unchanged: model, temperature, tools, workspace and namespace remain exact matches.

Cache policies

A cache policy is a named, workspace-configured set of cache settings. Rather than each call site hardcoding a duration, a request names a policy and an operator changes the numbers in one place.

curl https://gateway.aisetu.ai/v1/chat/completions \
  -H "Authorization: Bearer $AI_SETU_API_KEY" \
  -H "x-ai-setu-cache-policy: stock" \
  ...

A policy carries:

Policies are scoped to a workspace or shared tenant-wide, and a workspace-scoped policy takes precedence over a tenant-wide one of the same name.

The TTL ceiling

x-ai-setu-cache-ttl still works, but is now capped by the policy's maxTtlSeconds. This matters: cache duration is a judgement about how stale an answer may safely be, and that judgement belongs to whoever operates the workspace — not to whichever call site sent the header.

A policy for volatile data can set a ceiling of, say, 60 seconds, and a caller asking for 24 hours simply gets 60. Without a ceiling, one careless integration could serve day-old stock or pricing to a customer.

Choosing durations

Pick per kind of content, not per endpoint:

• Reference content — policies, hours, shipping zones — tolerates hours or days.
• Catalogue data tolerates minutes.
• Stock, price and order status tolerate seconds, and should carry a low ceiling.

If nothing is configured, the built-in default applies: 600 seconds fresh, no stale window, and a 24 hour ceiling — the behaviour the gateway has always had.

Serving stale while refreshing

A policy's swrSeconds adds a window after a response stops being fresh during which it may still be served. Inside that window the gateway answers instantly from cache instead of making the caller wait on a full provider call, and reports stale rather than hit so you can tell a guaranteed-current answer from a tolerated one.

Exactly one caller per window refreshes the entry. That caller does not receive the stale copy — it goes upstream, waits, and gets a fresh answer like any other cache miss. Everyone else is served immediately. A lock ensures only one refresh happens, so a popular entry going stale does not send a burst of identical requests to the provider at once.

Use it where a slightly old answer is better than a slow one — reference content, FAQs, catalogue text. Leave swrSeconds at 0 for anything where a stale answer would mislead: stock, price, order status, account balances. It is off by default.

Dependency tags

A TTL answers "how long do I trust this if nobody tells me anything". Tags answer the other half: "the underlying data just changed, drop it now". Only your application knows when that happened, so you declare it.

Attach tags to a request:

curl https://gateway.aisetu.ai/v1/chat/completions \
  -H "Authorization: Bearer $AI_SETU_API_KEY" \
  -H "x-ai-setu-cache-tags: sku:123,catalog" \
  ...

Then, when that data changes, revalidate the tag through the control plane:

mutation {
  revalidateCacheTags(workspaceId: "...", tags: ["sku:123"]) {
    invalidated
  }
}

Every cached response that declared sku:123 stops being served from that moment. Responses that did not declare it are untouched — unlike a purge, which drops everything.

A policy's defaultTags are attached automatically to every response under it, so a call site does not have to repeat a dependency that is true of the whole policy.

Notes

• Tag names are slugs: lowercase letters, digits, and . _ - :, starting with a letter or digit. sku:123 is the idiomatic shape.
• A request may declare up to 16 tags; a revalidation call may name up to 64.
• Revalidation requires the same permission as purging, and is audit-logged. Invalidating a hot tag forces fresh upstream calls, so it is a spending decision as much as a correctness one.
• Invalidation takes effect within a few seconds on a busy gateway, which is the same bound that already applies to purging.

Caching inside agent loops

Agent traffic is usually the most expensive and the least cacheable: a tool result embeds live data, so an answer derived from it can be wrong in a way a stale FAQ answer is not.

A request that carries a non-empty tools array is not eligible for the exact or the semantic response cache unless you opt in. There is no read and no write, and the response says so (x-ai-setu-cache: bypass with x-ai-setu-cache-bypass-tool: undeclared-tools). The default exists because a scripted or repeated agent conversation can be byte-identical to an earlier one while the right answer has changed, and replaying the old answer is a silent failure. You opt in one of three ways:

• Per tool — declare a granting x-ai-setu-cache on the tool definition (withToolCache(tool, { cache: true }), { policy } or { ttl }; the raw forms are {"safe": true} / {"cache": true}). Every tool the turn has already invoked must be declared; if nothing was invoked yet (the first turn of a loop), one declared tool is enough.
• Per request — send x-ai-setu-cache-force: 1.
• Per request — send an explicit x-ai-setu-cache mode other than off (no-store, force-refresh or semantic).

temperature > 0 and {"cache": false} still bypass as before; the force header does not override a cache: false declaration.

The decision cannot be made per request — an incoming message is arbitrary user text, and classifying it would cost the very call you are trying to avoid. But it does not need to be. The developer who added a tool knows what it does, so the policy travels with the tool:

import { withToolCache } from '@ai-setu/client';

const tools = [
  withToolCache(getCatalog, { policy: 'catalog', tags: ['catalog', 'sku:{sku}'] }),
  withToolCache(getBrandStory, { policy: 'brand-voice' }),
  withToolCache(getAvailableSlots, { policy: 'scheduling', tags: ['calendar:{date}'] }),
  withToolCache(addToCart, { cache: false }),
  withToolCache(checkout, { cache: false }),
];

The turn's cacheability is then derived from the tools it actually invoked. On the wire this is a key inside the tool definition, so any HTTP client can send it:

{
  "type": "function",
  "function": { "name": "getCatalog", "parameters": {} },
  "x-ai-setu-cache": { "policy": "catalog", "tags": ["catalog", "sku:{sku}"] },
}

cache: true, policy and ttl each opt the tool in. tags and swr alone do not: they shape a cached turn, they do not decide that one is allowed, so pair them with one of the above.

Prefer naming a policy over inline durations: the numbers stay in the console where an operator owns them, and retuning them never touches your code.

How several tools combine

A turn is one answer, so the tools it called must resolve to one policy:

• Any tool declaring cache: false bypasses the whole turn — no read, no write, neither layer.
• TTL and SWR take the minimum across participating tools. A turn that consulted the catalogue (60s) and the brand story (24h) is only as fresh as the catalogue. The x-ai-setu-cache-ttl header is held to the same rule, so it cannot be used to escape a declaration.
• Merging only ever narrows. A tool asking for longer than the request-level policy does not get it, and everything declared is capped by that policy's ceiling.
• Tags union, up to the usual 16.

A tool with no declaration does not narrow the TTL, but it does keep its turn out of the cache: if the turn invoked it, the whole turn is bypassed with undeclared-tools unless you used the force header or an explicit mode. Declare cache: false explicitly on anything that writes, charges, sends, or returns a value that changes on its own, so the reason reported names the tool.

Tag templates interpolate from that call's own arguments — sku:{sku} on a getProduct(sku: '9f2') call becomes sku:9f2 — so revalidating sku:9f2 drops precisely the cached turns that depended on that product. Argument values are model output and go through the same validation as caller-supplied tags.

Declarations are stripped before the request reaches the provider, and are excluded from the cache key — retuning a policy does not invalidate warm entries.

Seeing when a turn was bypassed

An uncached turn and a cold turn look identical unless you are told, so a bypassed turn says so:

client.lastCache;
// { status: 'bypass', hit: false, semantic: false, bypassedByTool: 'addToCart' }

On the wire: x-ai-setu-cache: bypass plus x-ai-setu-cache-bypass-tool. The header carries the blocking tool's name for a cache: false declaration, or the fixed value undeclared-tools when the request carried tools with no opt-in. The same split is on the ai_setu_gateway_cache_bypass_total metric.

Response headers

From the SDK these arrive parsed as client.lastCache (or the onCache callback) — status, a hit boolean that is true for hit / semantic-hit / stale, plus ageSeconds and similarity. lastCache stays undefined when the request was not cacheable at all, which is distinct from a miss: nothing was looked up.

Streaming responses (stream: true) are cached the same way — an SSE stream is stored on completion and replayed verbatim on a later hit. Tool order is normalized before hashing, so clients that reorder a tools array between otherwise-identical calls still hit.

Purging

Caching is always scoped to your workspace — entries are never served across workspaces, and entries written under a namespace are only served under that same namespace. If your underlying content changes (an FAQ update, a policy change) and you don't want to wait out the TTL, purge programmatically rather than guessing:

// @ai-setu/admin
await admin.cache.purge({ workspaceId, namespace: 'support-bot' });

Common reasons a cache never hits

• Bursty traffic on the default TTL. 600 seconds fresh, no stale window, means entries expire between bursts. A policy with a longer TTL and a stale window fixes this, and tags keep the longer TTL safe. Start here — it is the most common cause and the cheapest fix.
• The model string varies. zai.glm-4.7-flash and @mantle/zai.glm-4.7-flash reach the same model but are different cache keys. Pin one spelling.
• temperature > 0. Not cacheable unless you opt in with force.
• Tools on the request. Not cacheable unless a tool declares a granting x-ai-setu-cache, or you opt in with force or an explicit mode. Check x-ai-setu-cache-bypass-tool: undeclared-tools.
• Long agent context. A 3,000-token prompt carrying per-run state never repeats byte-for-byte, so exact matching cannot hit. Prompt caching still discounts its stable prefix — check promptCacheStats(response).ratio before concluding nothing is being saved. Tool cacheability declarations are the next lever; semantic matching is the last one, with the caveats above.