Conversational Caching

If you build a voice or chat agent, this page decides your hit rate. It assumes you have read Caching for the mechanics — modes, policies, tags, headers.

A turn is not one cacheable unit

One user utterance normally fans out into several provider calls, and they are not equally cacheable:

Ask one prompt to do all three and the whole turn inherits the cacheability of its least cacheable part — always the reply, which is unique per customer per turn. The cache then does nothing, and the hit rate reads as a threshold to tune when it is really a decomposition to do.

Layers 2 and 3 are the ones that hit. They exist only if you issue them as their own calls.

Intent classification

The highest hit rate available, and the easiest to split out. A static label set, one utterance, no customer context.

import { AiSetu, normalizeUtterance, sharedCacheOptions } from '@ai-setu/client';

const classifier = new AiSetu().withCache(sharedCacheOptions({ policy: 'intent' }));

await classifier.chat.completions.create({
  model,
  temperature: 0,
  messages: [
    { role: 'system', content: INTENT_LABELS }, // frozen — interpolate nothing
    { role: 'user', content: normalizeUtterance(transcript) },
  ],
});

normalizeUtterance is load-bearing. The exact cache canonicalises the request body — sorted keys, normalised numbers — but never rewrites message text. Without it, "Cancel my order" and "cancel my order" are two entries, and a question a thousand customers ask can miss a thousand times.

Every customer who says a common phrase shares one entry. Nothing customer-specific is in the prompt, so there is nothing to leak.

Knowledge and FAQ lookup

Impersonal by construction: a question, the retrieved article, an answer.

const faq = new AiSetu().withCache(
  sharedCacheOptions({
    mode: 'semantic',
    policy: 'faq',
    tags: ['kb:returns-policy'],
  }),
);

Tag anything derived from content you publish. When the article changes, one revalidateCacheTags(["kb:returns-policy"]) drops every cached answer that depended on it — which is what makes a long TTL safe. Without event-based invalidation you are forced into a short TTL and lose the hits you split the call out to get.

Keep these calls shallow — a system message and one user message. The semantic path's adaptive depth valve caps searches at a learned conversation depth, and a two-message sub-call is always under the cap. A second, independent reason not to run the lookup inside the conversation.

Deflection answers, and the splice

Deflection copy reads as personalised but usually is not. This is the highest-leverage change available, and it needs no SDK at all:

Cache the impersonal body. Interpolate the personal fields after the response returns.

// Uncacheable: one entry per customer, hit rate ~0.
messages: [{ role: 'user', content: `Reply to ${customer.name} about their delayed order.` }];

// One entry shared by every customer with a delayed order.
const body = await faq.chat.completions.create({
  model,
  messages: [{ role: 'user', content: 'Explain a delayed order and the options available.' }],
});
const reply = `${customer.name}, ${text(body)}`;

The rule that decides it: if the model does not need to reason over the personal field, it does not belong in the prompt. Names, order numbers, dates and balances are almost always splice candidates. A tier that changes the answer's substance is not.

The reply itself — prompt caching

The conversational reply cannot be response-cached. Every whole-prefix scheme fails on it for two independent reasons: no two customers share a prefix, and within one conversation the prefix grows every turn. That is structural, not a threshold.

Prompt caching is the lever that applies. It substitutes nothing — the model runs, the output is fresh — the provider just skips re-prefilling an unchanged prefix and discounts those tokens. It needs the prefix stable across the turns of one conversation, which a per-customer prompt already is.

Your job is to keep the prefix stable. Anything volatile invalidates everything after it:

A per-turn flag interpolated into the system prompt is the common killer — it sits ahead of everything and invalidates the whole prefix every turn. Move it after the history and the same prompt starts hitting from turn two.

Verify rather than assume:

import { promptCacheStats } from '@ai-setu/client';

const { ratio } = promptCacheStats(response);
// ratio 0 across repeated turns ⇒ something volatile is still in the prefix

Traps

namespace partitions; it does not organise. An entry written under a namespace is only served under that same namespace. Setting a per-customer namespace on a shareable sub-call — the instinct when the calling code is already customer-scoped — produces a zero hit rate with no error anywhere. sharedCacheOptions throws rather than let it through. Use namespaces for isolation you actually want, never as a filing system.

temperature > 0 skips the cache unless forced. sharedCacheOptions sets force for you, and mode: 'semantic' implies it.

A tool result in the window disqualifies the turn, unless the tool is declared safe. Live data must not be served to someone else. See caching inside agent loops.

Long questions fall back to exact-only. Past the embed cap the gateway skips embedding silently; isSemanticEligibleText makes the check explicit.

Adoption order

1. Split classification out — smallest change, highest hit rate, no leak surface.
2. Split knowledge lookups out and tag them, then raise the TTL.
3. Splice personalisation out of deflection copy.
4. Fix the prompt shape so the reply's prefix stops moving, and confirm with promptCacheStats.

Steps 1-3 are entirely yours. Step 4 needs prompt caching enabled for your workspace.