Routing

When you send a bare model id, AI Setu decides which provider serves the request. This page covers how that decision is made, what happens when a provider fails, and the optional load-balancing layer on top.

const res = await client.chat.completions.create(
  { model: 'claude-sonnet-4-5', messages: [{ role: 'user', content: 'Hi' }] },
  // optional per-call override of the provider order:
  { headers: { 'X-AI-Setu-Provider-Order': 'bedrock,anthropic' } },
);

console.log(res.headers?.['x-ai-setu-provider']); // which provider actually served

Resolution precedence

For a bare model id (no <provider>/ or @<slug>/ prefix), the gateway picks a provider in this order — each step only applies if the previous one didn't already decide:

1. Explicit prefix — <provider>/<model> or @<slug>/<model> always hard-overrides everything below.
2. X-AI-Setu-Provider-Order request header — a comma-separated, case-insensitive provider list (e.g. anthropic,openai) that reorders and filters the candidates for this one request. Setting this also disables the load-balancing reshuffle below, for this request.
3. Workspace precedence — a per-provider priority you set in the console (Settings → Providers & routing), stored on your BYOK bindings.
4. BYOK-first — a provider you have your own credential for is preferred over the platform key, each group kept in catalog order.
5. Catalog static order — the model's default provider precedence.

After ordering, providers with no usable credential (no BYOK binding and no platform key configured for that provider on this deployment) are dropped. The first survivor serves the request; the rest become failover candidates.

Failover

If the serving provider returns a retryable error — HTTP 5xx, HTTP 429, or a connection/timeout failure — and no response byte has reached the client yet, the gateway transparently retries the next candidate. Up to 3 attempts total (primary + two fallbacks). Any other 4xx is returned as-is immediately, since the request would fail identically on every provider.

A failed attempt is fully refunded before the next attempt reserves again — failover never double-bills. Once the first byte of a streamed response has been sent, the request is committed to that provider; there is no failover after that point. The X-AI-Setu-Provider response header always names whichever provider actually served the request.

This is controlled by the GATEWAY_FAILOVER kill-switch (on by default). When off, only the first candidate is attempted — no cross-provider failover.

Load balancing

On top of the ordering above, an opt-in load-balancing layer can reshuffle the availability-filtered candidate list before failover runs — it never adds, removes, or duplicates candidates, only reorders them. It's gated by the GATEWAY_LOAD_BALANCE kill-switch (off by default) and is skipped entirely if you sent an explicit X-AI-Setu-Provider-Order header for that request.

Four strategies:

Load-balance configuration resolves most-specific-first: a (workspace, model) config beats a (workspace, any-model) config, which beats a (tenant-wide, model) config, which beats a (tenant-wide, any-model) config. Ask your workspace admin whether load balancing is enabled on your deployment before relying on a particular strategy.

Verifying what served a request

Every successful response carries:

@ai-setu/client surfaces the same information synchronously via client.lastRouting ({ provider, byok, connectionSlug }) after each call.