AI Gateway

The AI Gateway is the provider-agnostic abstraction every Polyant component uses to call a large language model. The supervisor does not import @ai-sdk/openai. The memory extractor does not import @ai-sdk/anthropic. They both call chat({ tier: "standard", messages, ... }) and the gateway picks the right provider, the right model, and threads logging, tracing, and cost accounting through the call.

This page covers the tier model, the per-instance provider override, the capability gates, the separate embeddings gateway, the call-type tagging used for cost analytics, and the structured logging that lands every call in ai_logs.

Tier abstraction, not model lock-in

Every call site declares a workload class, not a model:

• fast - cheap classification work. Memory extraction, webhook event matching, title generation, conversation summarisation.
• standard - the main conversation tier. The supervisor uses this for user-facing turns.
• heavy - deep reasoning. Its consumers today are the semantic governance gates (prompt injection, PII, topic guardrail), which each parse a JSON verdict and each fail open: a heavy model that is unavailable in the region stops the gates from gating, with one warning line and no refusal.

The tier tables, and the per-model capabilities and rates behind them, live in one place: packages/engine/src/ai-gateway/model-catalog.ts. Four providers are built in: openai, anthropic, bedrock, and nebius (Nebius Token Factory, an OpenAI-compatible endpoint serving open-weight models). The managed-services build registers three more at boot, all EU-hosted and OpenAI-compatible: tensorx (TensorX), scaleway (Scaleway Generative APIs) and regolo (Regolo.ai).

A provider that does not ship with the engine is added through a registry rather than by editing the maps every request reads: registerAiProvider() takes the adapter, the catalogue (tiers, models, rates, capabilities) and the secret that holds the API key as one unit, before the first request is served. It refuses a built-in name, so a registration can never silently re-route every agent on openai. Embedders have the twin registerEmbeddingProvider(), described below.

For the exact model ids each tier resolves to today, read the catalogue - pinning them here would go stale on the next model release. The shape is one { fast, standard, heavy } triple per provider, plus one catalogue entry per model carrying its input/output/cache rates and its capability flags (reasoning, reasoning control and levels, vision, temperature, cache).

A deployment owner can swap the entire fleet to another provider by setting each instance’s provider, or pin a single agent to a specific model - and no application code changes. Every workload that asked for standard continues to ask for standard.

Un-catalogued model ids (a fresh Bedrock or Nebius id, say) fall back to per-provider heuristics, and the fallback is logged: an entry in the catalogue is the supported way to add a model.

Bedrock is the one provider whose tiers deliberately avoid Anthropic. Claude on Bedrock sits behind a per-account use-case grant, and an account without it failed every tier at once - including the service jobs that resolve through the tier table and have no per-agent override to rescue them. fast and standard run on Amazon Nova, the only non-Anthropic Bedrock family with both prompt caching and vision; heavy runs on an open-weight reasoning model. Two costs come with that choice and are worth stating: Nova Pro does not reason, so a Bedrock agent with thinking enabled and no explicitly pinned model loses reasoning silently, and the heavy id is a plain on-demand model rather than a cross-region inference profile, so its availability is per-region. An agent that needs Claude on Bedrock pins it explicitly, once the account holds the grant.

Which providers and models a tenant may run

Managed services. Provider admissibility is available with Polyant managed services.

Being wired into the catalogue is not the same as being allowed. A provider reaches an agent only when both tenant tiers agree: a platform admin admits the provider to the organization, and an organization admin assigns it to the agent’s workspace. Either half missing and the agent cannot run on it.

Within an admitted provider, a single model can be switched off - and here the rule inverts. Admission is a grant, so it intersects; switching a model off is a veto, so it unions. A model the organization disabled stays disabled even if the workspace never touched it, and the other way round.

The intersection is computed in one place (ai-providers/admissibility.ts), so the runtime and the panel cannot drift apart on the same question. It fails closed: an agent whose organization or workspace cannot be resolved is admitted nothing, not everything.

Three consequences are worth knowing before you rely on it:

• An agent with no provider of its own is judged on the deployment default. It used to be exempt, on the reading that there was no tenant decision to apply to a provider nobody had chosen - but the gateway resolves it to a real provider either way, so the least-configured agents were the ones running whatever the installation defaults to, admitted or not.
• A tier slides rather than stops. Internal work asks for a tier, not a model, and title generation, memory extraction, transcription and webhook matching all run on fast. If the model a tier points at is switched off, the tier moves onto the first admitted model of the same provider and the call is logged as slid. Only when the provider has nothing left to run does the call actually stop - exempting tiers altogether was the other option, and it would have kept sending every message to a provider the organization banned for jurisdiction.
• On upgrade nothing changes. The migration admits every provider to every organization and every workspace, so a running deployment comes out of it behaving exactly as before. The narrowing is a decision someone has to make. The same holds for the three providers available with managed services: the upgrade that introduced them admitted them for existing organizations and workspaces.
• Embeddings and transcription are bound too. Admission applies to every call that sends the tenant’s content to a provider, not only to chat: memories, knowledge documents and search queries going to an embedder, and voice notes going to speech-to-text. An agent embedding or transcribing on a provider its workspace does not admit is refused. The speech services (Deepgram, ElevenLabs, Typesafe) are outside the provider catalogue, so no admission applies to them; the connection rules below still do.

Where these are edited: the organization half is in the Admin Console, under the organization’s row, and belongs to a platform admin alone - an org admin who could widen it would be lifting the allow-list they are subject to. The workspace half is in the workspace tier of Settings, under workspace.assignment:manage. Both write to the management audit trail.

Whoever configures an agent reads the result with GET /api/instances/{slug}/providers, under agent:read: the providers, models and prepared connections the agent’s workspace admits. It needs none of the governance permissions, which belong to the people who decide what a workspace runs, so a Member’s model picker offers only what the agent may actually use.

Provider connections, a credential picked by name

Managed services.

With managed services every provider credential is a named connection, owned by an organization and lent to workspaces, owned by one workspace and usable by its agents alone, or private to one agent. Every task that calls a provider binds a role to one connection: the agent’s conversation (chat), its embedder (embedding), voice-note transcription (stt), the parts of a phone call, and the organization’s library embedder. Whoever builds an agent picks a connection by name; the key never reaches the panel. A provider catalog says what each provider can do, and a role accepts only a connection that can serve it. Provider connections and roles describes the connections, the roles, the managed-only rule and the Bedrock grant for the deployment’s AWS identity.

A role that cannot run is reported, never repaired: the call is refused rather than served by another key, because a revoked credential resolved by quietly falling back onto another one would move the spend onto a contract nobody chose. The credential refusals are, in the order they are checked:

Admission comes after the credential: provider_not_admitted when the provider the turn would reach is not admitted by both tiers, model_not_admitted when the pinned model is switched off by one of them. The credential is checked first because it is the more specific fact: an agent bound to a withdrawn connection has a concrete, fixable problem, and reporting “provider not admitted” instead would send an admin to the wrong screen.

On upgrade the provider keys found in agent and organization secrets were carried into connections and bindings, so no agent changed credential that day. See Upgrading to 1.2.0.

Per-instance override - provider AND specific model

Each agent has two columns on the instances row that drive model selection: provider (openai / anthropic / bedrock / nebius, or a registered provider such as tensorx) and model (any model id that provider supports). Both are editable from the admin panel under Model for the agent. Either can be left null.

How the gateway combines them with the caller’s tier:

final.provider = request.provider     (instance row, any registered provider)  ?? DEFAULT_PROVIDER ("openai")
final.model    = request.model        (instance row)  ?? resolveModel(provider, tier)

The exact model the user picked wins over the tier table - tier becomes a label-only hint when model is set. If model is null, the tier table picks the model (the matrix above). So:

• Operator sets provider="anthropic", model="claude-sonnet-5" → the supervisor’s standard call goes to that exact Sonnet build, regardless of what the tier table says.
• Operator sets provider="anthropic" and leaves model null → the supervisor’s standard call resolves to whatever the tier table says is Anthropic’s standard today.
• Operator leaves both null → the gateway falls back to DEFAULT_PROVIDER (“openai”) and the tier table.

Where the explicit model is and isn’t honoured

This is the part the resolution rule alone hides: not every call site passes the instance’s model. The supervisor does, for the user-facing turn. Background workloads do not.

The asymmetry is intentional: a cheap classifier should stay cheap. If you pin Sonnet on your agent, you do not want every memory-extraction pass to also use Sonnet - that would silently 10× the cost of fire-and-forget background work. The tier table keeps fast-tier calls on the cheap model for the chosen provider, even when the user-facing turn runs on the premium model the operator picked.

If you genuinely need background work to follow the explicit model, the right knob is the tier table in ai-gateway/model-catalog.ts, not the instance row.

Capability gates apply to the resolved model

Several per-agent settings are only meaningful on some model builds, and every gate reads the catalogue entry for the resolved model - the instance’s explicit model, or the tier-table standard model when none is pinned. No model-id regex decides behaviour on the request path.

• Thinking. thinkingEnabled is dropped when the resolved model has no reasoning support: a stale true after switching models has no runtime effect, nothing crashes, the feature is simply off. A model that always reasons has no off-switch - only its effort is tunable, and the panel locks its toggle on (Claude Fable 5 and 5.1, Claude Opus 5.5, gpt-6-astra among them).
• Thinking off is sent, not omitted. Some models reason by default, so leaving the parameter out keeps them reasoning and billing for it. For those the catalogue records the off-switch and the gateway sends it: reasoning_effort: "none" for gpt-6-sol and gpt-6-luna, thinking disabled for Claude Opus 5 and Sonnet 5 (on Bedrock too). Claude Sonnet 5.5 refuses that switch, so off sends its lowest setting, thinking only between tool calls.
• Reasoning control and levels. How thinking is expressed on the wire (an effort level, a token budget, or the adaptive shape) comes from the catalogue, as does the exact set of levels that model accepts. Both the API and the admin panel validate against that set, so no request goes out with an effort the provider would reject with a 400.
• Temperature. Omitted entirely for models that do not accept the parameter, rather than sent and rejected.
• Vision. Image and file parts are stripped for models that cannot take them.
• Cache. The prompt-cache marker is injected only for families where it is accepted and actually discounted. See Prompt → Prompt caching.

Per-instance secrets

On the open-source build the provider keys are agent secrets, encrypted in instance_secrets: openai_api_key, anthropic_api_key, nebius_api_key, and for AWS-backed services bedrock_api_key plus the dedicated aws_provider_access_key_id / aws_provider_secret_access_key / aws_provider_region namespace - deliberately separate from the generic aws_* secrets a tool may declare, so the AI provider and a tool can use different AWS accounts.

With managed services these keys are held by connections and are no longer read from the secrets: the secret routes refuse to write them, and copies left there by the upgrade reach no tool, hook or plugin.

The provider adapters read the key from the resolved config and never touch process env directly. Without the right key, the call fails fast at the adapter - there is no fallback to a process-wide key, and none for the region either: a Bedrock agent, chat or embeddings, with no region is refused with a message naming the setting.

Why tier abstraction earns its keep

The same supervisor binary serves many instances at once. Some run on Anthropic, some on OpenAI, some on Bedrock. The OpenAI tiers moved from gpt-4o to the gpt-6 family in one release. None of these changes touch supervisor code, prompt templates, or tool implementations.

The tier names also document intent. A grep for tier: "fast" instantly tells you which call paths are cost-sensitive (extractors, classifiers) versus user-facing (standard). When a new optimisation lands - caching, batching, a cheaper model - the operator can roll it out to every “fast” caller in one config change.

Embeddings have their own gateway and their own provider

Embeddings do not follow the chat provider. They go through a separate embeddings gateway with its own per-agent embeddingProvider column. Two embedders are built in, and the managed-services build registers two more:

An embedder added with registerEmbeddingProvider() is an OpenAI-compatible /v1/embeddings endpoint described as data: base URL, model id, the secret holding its key, and the dimensions it actually returns, which decide the vector column its rows land in. The agent’s embedder selector lists what the server serves (GET /api/instances/models returns them as embedders), not a fixed pair. An embedder name the server does not serve fails instead of falling back to OpenAI, and an import that names one is refused.

Anthropic, Nebius and Scaleway have no embedder here, so an agent chatting on one of them still needs one of the above - but it does not have to be OpenAI. An agent on Bedrock can embed on Bedrock with no OpenAI key at all, and one on TensorX or Regolo.ai can keep its embeddings with the same EU provider. Credentials are resolved for the embedder, independently of which model the agent chats with; with managed services that is the connection bound to the agent’s embedding role.

New agents default to 1024 dimensions. memories and knowledge_chunks each carry parallel 1024- and 1536-dimension vector columns with a database check constraint ensuring exactly one is populated per row, so both spaces can coexist across agents in one deployment.

Changing the embedder is destructive. Vectors are provider-specific and are deliberately not re-embedded: re-embedding is slow, costly, and easy to get half-right. Instead the old space is abandoned - every memory and the whole knowledge base for that agent (documents, chunks, raw content) are deleted in one transaction and the dimension is realigned to the new provider’s default. Conversations are untouched, so keyword search over raw messages keeps working; only extracted memories go.

Changing only the chat provider never touches embeddings.

Call-type tagging: conversation vs. service

Every gateway call carries a callType in ChatCallOptions:

• "conversation" - user-visible turns. The supervisor handling an inbound message.
• "service" - background work. Memory extraction, title generation, webhook event matching, room event matching, summarisation.

The tag flows through to ai_logs.call_type. Analytics queries can then split billable conversation cost from infrastructure service cost - answering questions like “what does it cost me to keep memory enabled for this instance?” without confusing the per-conversation cost reports.

The supervisor’s main path passes "conversation". Every fire-and-forget extractor passes "service". The convention is enforced by code review, not by the type system.

Logging: every call lands in ai_logs

After every chat() or chatStream(), successful or failed, and after every embedding call, the gateway writes one row into ai_logs (packages/engine/src/ai-gateway/logger.ts):

A failed call writes its row with outcome = error, the error class, and the usage and cost of the steps it completed before failing: a multi-step call that dies at step five was billed for four. A turn cancelled and restarted because the user sent a follow-up writes its row with outcome = aborted and what its model calls were billed. Embedding calls (knowledge ingestion, memory extraction, retrieval) write a row with their tokens and cost; a registered OpenAI-compatible embedder is logged with no cost. Cost and token sums in analytics count every row, and cost totals are summed in double precision so long periods keep their precision; call counts and response times count only ok model calls.

Cache writes are priced per model. Anthropic reports how many tokens it wrote at each cache lifetime, and each is billed at its own rate: 2× the input rate for a 1-hour write, 1.25× for a 5-minute one. A write with no reported split is billed at the 1-hour rate. OpenAI’s gpt-5.6 and gpt-6 models carry a cache-write rate of 1.25× input in the catalogue. A model with no cache-write rate bills writes at its input rate.

Some models are priced by prompt length, and the threshold is judged per call, so each call of a tool loop is priced on its own prompt. A call to Claude Haiku 5.5 (on Anthropic or Bedrock) whose prompt passes 100K tokens pays 5× on every rate, and a call to GPT-5.4, GPT-5.6 or GPT-6 with more than 272K input tokens pays the long-context rates for the whole request. GET /api/instances/models returns the higher rates as costLongPrompt. Amazon Nova is priced at the eu-south-1 (Milan) rates, because an eu.* inference profile is billed at the price of the Region that calls it.

The logger buffers in memory and flushes every 10 entries or every 5s. On flush failure entries are re-queued; the buffer is capped at 1000 to bound memory.

ai_logs is the source of truth for the admin panel’s cost dashboard and the per-instance analytics pages. It is distinct from pipeline_traces, which records end-to-end pipeline latency including non-LLM phases (context prep, tool building, persistence) - see Architecture.

How it works

+----------------------------------------------------+
| caller declares workload class                     |
|   supervisor.run()      tier: "standard"           |
|   extractMemories()     tier: "fast", call=service |
|   webhookMatcher()      tier: "fast", call=service |
|   titleGenerator()      tier: "fast", call=service |
+--------------------------+-------------------------+
                           |
                           v
+----------------------------------------------------+
| AI Gateway (ai-gateway/index.ts)                   |
|   resolveCallConfig(request, options)              |
|     provider  <- request.provider | instance | default
|     modelId   <- request.model    | resolveModel() |
|     buildLangSmithProviderOptions(...)             |
+--------------------------+-------------------------+
                           |
       +-------------------+-------------------+
       v                   v                   v
   OpenAIProvider   AnthropicProvider  BedrockProvider  NebiusProvider  (+ registered)
   (providers/openai.ts) (...anthropic.ts) (...bedrock.ts) (...nebius.ts)  (registry.ts)
       |                 |                 |                |
       +--------+--------+-----------------+----------------+
                 |
                 v
        Vercel AI SDK (generateText / streamText)
                 |
                 v
   logAndRecordUsage()
       pipelineLog.llmResponse(...)
       estimateCost(provider, model, tokens)
       aiLogger.log({ provider, model, tier, tokens,
                      cost, durationMs, conversationId,
                      instanceId, callType })
                 |
                 v
            ai_logs table (buffered flush every 10 entries / 5s)

Code reference

• packages/engine/src/ai-gateway/index.ts - chat(), chatStream(), resolveCallConfig(), logAndRecordUsage().
• packages/engine/src/ai-gateway/model-catalog.ts - Per-provider tier mapping and per-model rates + capability flags.
• packages/engine/src/ai-gateway/config.ts - resolveModel(), estimateCostBreakdown(), isThinkingCapable(), reasoningControlFor(), reasoningLevelsFor().
• packages/engine/src/ai-gateway/types.ts - ChatRequest, ChatResponse, ChatStreamResult, TierMapping, ProviderAdapter.
• packages/engine/src/ai-gateway/providers/openai.ts - OpenAI adapter (Vercel AI SDK @ai-sdk/openai).
• packages/engine/src/ai-gateway/providers/anthropic.ts - Anthropic adapter (@ai-sdk/anthropic).
• packages/engine/src/ai-gateway/providers/bedrock.ts - Bedrock adapter (@ai-sdk/amazon-bedrock).
• packages/engine/src/ai-gateway/providers/nebius.ts - Nebius Token Factory adapter (OpenAI-compatible).
• packages/engine/src/ai-gateway/providers/registry.ts - registerAiProvider(): the provider name → adapter and API-key secret map, built-ins seeded, the rest registered at boot.
• packages/engine/src/embeddings-gateway/registry.ts - registerEmbeddingProvider(): embedders added at boot, with the dimensions each returns.
• packages/engine/src/ai-gateway/logger.ts - ai_logs schema + buffered AILogger.
• packages/engine/src/ai-gateway/langsmith.ts - LangSmith tracing provider options.
• packages/engine/src/instances/config-resolver.ts - Per-instance provider + secrets resolution (30s TTL cache).
• packages/engine/src/embeddings-gateway/ - The separate embeddings gateway: provider resolution, per-provider models and dimensions, and the destructive reset on an embedder switch.

Spend caps

Managed services.

Before each model call the gateway asks the spend guard whether the agent may still spend. An agent can carry caps per calendar window, per contact and per conversation; when one is reached the call is refused and the agent answers with its stop message, or, mid-turn, the tool loop stops and the message is appended to what it had said. The caps are soft: they are checked against committed spend, calls already in flight may finish, and no overshoot bound is guaranteed. Configuration, semantics and the API are on Spend cap.

See also

• Architecture - where the gateway sits in the request pipeline.
• Memory - uses tier: "fast" for extraction.
• Agents - the supervisor consumes the gateway via tier: "standard".
• Tools - some tools also call the gateway through their ToolContext.