Load balancing

Spread traffic across a request's candidate providers with round-robin, weighted, latency, or cost strategies.

Load balancing is opt-in on each deployment (the GATEWAY_LOAD_BALANCE kill-switch, off by default). Ask your platform operator whether it's enabled before configuring a strategy — with the switch off, no reordering ever runs.

Routing already produces an ordered list of provider candidates for a request (BYOK-first, workspace precedence, catalog order). Load balancing sits on top of that list and reshuffles it per request — it never changes which providers are eligible, only which one goes first.

Configuring a strategy

mutation SetStrategy {
  upsertLoadBalanceConfig(
    tenantId: "..."
    input: {
      workspaceId: "..." # omit for a tenant-wide default
      model: null # omit/null to apply to every model
      strategy: WEIGHTED
      weightsJson: "{\"openai\":3,\"anthropic\":1}"
      enabled: true
    }
  ) {
    id
    strategy
    weightsJson
    enabled
  }
}

Or from the console: Settings → Load balancing. strategy is one of ROUND_ROBIN, WEIGHTED, LATENCY, COST. weightsJson is only consulted for WEIGHTED; a provider missing from the map defaults to weight 1.

What gets reordered

The candidate list for a request is the set of providers your workspace can actually serve that model from — typically a BYOK connection plus the platform key as fallback, or several BYOK connections across providers. Load balancing reorders that list; it does not select among multiple credentials you may have configured for the same provider — a per-provider multi-credential spread exists in the gateway's load-balance package (SelectKey) but is not yet called from the request path, so configuring more than one connection to the same provider today does not get spread traffic between them.

Strategies

Strategy	Behavior
`round_robin`	Rotates the candidate list by an atomically-incremented counter keyed on tenant + workspace + model, so a different candidate leads on each call. After n requests every candidate has led exactly once.
`weighted`	Picks the primary candidate by weighted random draw (`weightsJson`, missing entries default to `1`), then orders the rest by descending weight. Ties keep their original candidate order.
`latency`	Orders ascending by each provider's observed latency — an exponentially-weighted moving average (EWMA, α = 0.2) updated after every call. A provider with no samples yet sorts first, so it gets traffic and the tracker learns its latency.
`cost`	Orders ascending by price-per-call from the rate card. A provider with no known price sorts last, so traffic isn't sent to an unpriced candidate by default.

Resolving which config applies

A request's (tenant, workspace, model) is matched against configured rows in this order — most specific wins:

(workspace, model) — this workspace, this exact model
(workspace, any model) — this workspace, all models
(tenant-wide, model) — every workspace in the tenant, this exact model
(tenant-wide, any model) — the tenant's default

No matching row (or any resolution error) means load balancing is disabled for that request — resolution fails open rather than blocking the call. Config changes propagate through an in-process cache (10s) and a shared cache (60s), so a change can take up to a minute to reach every gateway instance.

The no-double-bill invariant

Load balancing only reorders the candidate slice; it never adds, removes, or duplicates a candidate. That matters because failover refunds a failed attempt's reservation before trying the next candidate — reordering the same set of candidates keeps that refund-then-retry accounting intact. Reshuffling who goes first cannot cause a request to be billed twice, and cannot cause a candidate to be tried that wouldn't otherwise have been eligible.

Interaction with failover and manual ordering

Load balancing runs immediately before the failover loop, and only when all of the following hold:

the kill switch is on,
a config resolves for this request with a non-empty strategy,
there is more than one candidate to reorder, and
you did not send an explicit X-AI-Setu-Provider-Order header on this request — a manual order always wins and disables the reshuffle for that call.

Once the list is ordered, normal failover applies: a retryable upstream error (5xx, 429, connection/timeout) advances to the next candidate in the (possibly reordered) list, up to 3 attempts, with no failover once the first response byte has reached you.

Verifying what served a request

X-AI-Setu-Provider on the response always names whichever candidate actually served the request, regardless of strategy — see Routing for the full set of routing response headers.