Caching

The gateway keeps a workspace-scoped response cache so repeat requests don't re-hit an upstream provider. A cache hit costs $0 and never reaches the provider.

curl https://gateway.aisetu.ai/v1/chat/completions \
  -H "Authorization: Bearer $AI_SETU_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-ai-setu-cache: semantic" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "What is AI Setu?"}]
  }'

Check the outcome on the response header x-ai-setu-cache.

Exact-match caching (default)

By default, a request is cacheable when temperature is unset or 0. The cache key is derived from the workspace, an optional namespace, and a hash of the model plus the full request body — so identical requests hit, and anything different (a different model, a different message, a different tool list) misses. Cache entries default to a 600 second TTL and can be set up to a 24 hour maximum via a request header.

An unrecognized x-ai-setu-cache value fails open to default behavior rather than erroring.

Semantic caching (opt-in)

Setting x-ai-setu-cache: semantic adds a similarity search on top of exact matching: on an exact miss, the gateway embeds the latest user message and serves the best stored response whose cosine similarity clears a threshold — default 0.8, overridable per request via x-ai-setu-cache-threshold (clamped to 0–1). Everything else in the request — prior conversation, model, temperature, tools, response format, workspace, namespace — must still match exactly, so paraphrased questions can hit but different contexts never cross-match. Semantic mode implies caching even at temperature > 0; no force header needed.

A few things never enter the semantic store, by design and not configurable: conversations containing tool calls or tool results, non-text content, and messages over roughly 2,000 characters — these degrade to exact-match-only caching instead. This deployment default (GATEWAY_SEMCACHE, on) can disable the embedding step entirely, in which case semantic degrades to plain exact-match caching. Embedding cost is platform-absorbed — a semantic hit costs $0, same as an exact hit.

Semantic-cache quality depends on the embedding model behind it, which is currently English-leaning; non-English paraphrase matching may be weaker than exact-language matching. Exact-match caching is unaffected by this and works the same regardless of language.

Cache policies

A cache policy is a named, workspace-configured set of cache settings. Rather than each call site hardcoding a duration, a request names a policy and an operator changes the numbers in one place.

curl https://gateway.aisetu.ai/v1/chat/completions \
  -H "Authorization: Bearer $AI_SETU_API_KEY" \
  -H "x-ai-setu-cache-policy: stock" \
  ...

A policy carries:

Policies are scoped to a workspace or shared tenant-wide, and a workspace-scoped policy takes precedence over a tenant-wide one of the same name.

The TTL ceiling

x-ai-setu-cache-ttl still works, but is now capped by the policy's maxTtlSeconds. This matters: cache duration is a judgement about how stale an answer may safely be, and that judgement belongs to whoever operates the workspace — not to whichever call site sent the header.

A policy for volatile data can set a ceiling of, say, 60 seconds, and a caller asking for 24 hours simply gets 60. Without a ceiling, one careless integration could serve day-old stock or pricing to a customer.

Choosing durations

Pick per kind of content, not per endpoint:

• Reference content — policies, hours, shipping zones — tolerates hours or days.
• Catalogue data tolerates minutes.
• Stock, price and order status tolerate seconds, and should carry a low ceiling.

If nothing is configured, the built-in default applies: 600 seconds fresh, no stale window, and a 24 hour ceiling — the behaviour the gateway has always had.

Serving stale while refreshing

A policy's swrSeconds adds a window after a response stops being fresh during which it may still be served. Inside that window the gateway answers instantly from cache instead of making the caller wait on a full provider call, and reports stale rather than hit so you can tell a guaranteed-current answer from a tolerated one.

Exactly one caller per window refreshes the entry. That caller does not receive the stale copy — it goes upstream, waits, and gets a fresh answer like any other cache miss. Everyone else is served immediately. A lock ensures only one refresh happens, so a popular entry going stale does not send a burst of identical requests to the provider at once.

Use it where a slightly old answer is better than a slow one — reference content, FAQs, catalogue text. Leave swrSeconds at 0 for anything where a stale answer would mislead: stock, price, order status, account balances. It is off by default.

Dependency tags

A TTL answers "how long do I trust this if nobody tells me anything". Tags answer the other half: "the underlying data just changed, drop it now". Only your application knows when that happened, so you declare it.

Attach tags to a request:

curl https://gateway.aisetu.ai/v1/chat/completions \
  -H "Authorization: Bearer $AI_SETU_API_KEY" \
  -H "x-ai-setu-cache-tags: sku:123,catalog" \
  ...

Then, when that data changes, revalidate the tag through the control plane:

mutation {
  revalidateCacheTags(workspaceId: "...", tags: ["sku:123"]) {
    invalidated
  }
}

Every cached response that declared sku:123 stops being served from that moment. Responses that did not declare it are untouched — unlike a purge, which drops everything.

A policy's defaultTags are attached automatically to every response under it, so a call site does not have to repeat a dependency that is true of the whole policy.

Notes

• Tag names are slugs: lowercase letters, digits, and . _ - :, starting with a letter or digit. sku:123 is the idiomatic shape.
• A request may declare up to 16 tags; a revalidation call may name up to 64.
• Revalidation requires the same permission as purging, and is audit-logged. Invalidating a hot tag forces fresh upstream calls, so it is a spending decision as much as a correctness one.
• Invalidation takes effect within a few seconds on a busy gateway, which is the same bound that already applies to purging.

Caching inside agent loops

By default, a request carrying tool calls or tool results is never semantically cached. A tool result embeds live data, so an answer derived from it can be wrong in a way a stale FAQ answer is not.

That default is safe but blunt: it means agent loops — usually the most expensive traffic — get no benefit. If a tool is genuinely read-only, say so:

{
  "type": "function",
  "function": { "name": "getOrderStatus", "parameters": { ... } },
  "x-ai-setu-cache": { "safe": true, "tags": ["order:{orderId}"] }
}

A turn becomes eligible for caching when every tool it invoked is declared safe. One undeclared or unsafe tool and the whole turn is excluded, as before.

The tags entries become dependency tags, with {placeholder} interpolated from that call's arguments — so a getOrderStatus call for order 9f2 depends on order:9f2. Revalidate that tag when the order changes and the cached turn is dropped precisely when it stops being true.

Declarations are stripped from the request before it reaches the provider, so they never interfere with the upstream call.

Response headers

Streaming responses (stream: true) are cached the same way — an SSE stream is stored on completion and replayed verbatim on a later hit. Tool order is normalized before hashing, so clients that reorder a tools array between otherwise-identical calls still hit.

Purging

Caching is always scoped to your workspace — entries are never served across workspaces, and entries written under a namespace are only served under that same namespace. If your underlying content changes (an FAQ update, a policy change) and you don't want to wait out the TTL, purge programmatically rather than guessing:

// @ai-setu/admin
await admin.cache.purge({ workspaceId, namespace: 'support-bot' });