AI Gateway

The AI Gateway is the provider-agnostic abstraction every Polyant component uses to call a large language model. The supervisor does not import @ai-sdk/openai. The memory extractor does not import @ai-sdk/anthropic. They both call chat({ tier: "standard", messages, ... }) and the gateway picks the right provider, the right model, and threads logging, tracing, and cost accounting through the call.

This page covers the tier model, the per-instance provider override, the capability gates, the separate embeddings gateway, the call-type tagging used for cost analytics, and the structured logging that lands every call in ai_logs.

Tier abstraction, not model lock-in

Every call site declares a workload class, not a model:

• fast — cheap classification work. Memory extraction, webhook event matching, title generation, conversation summarisation.
• standard — the main conversation tier. The supervisor uses this for user-facing turns.
• heavy — deep reasoning. Used sparingly today; reserved for tasks where the operator opts into reasoning models (o3, Claude Opus, etc.).

The tier tables, and the per-model capabilities and rates behind them, live in one place: packages/engine/src/ai-gateway/model-catalog.ts. Four providers are wired: openai, anthropic, bedrock, and nebius (Nebius Token Factory, an OpenAI-compatible endpoint serving open-weight models).

For the exact model ids each tier resolves to today, read the catalogue — pinning them here would go stale on the next model release. The shape is one { fast, standard, heavy } triple per provider, plus one catalogue entry per model carrying its input/output/cache rates and its capability flags (reasoning, reasoning control and levels, vision, temperature, cache).

A deployment owner can swap the entire fleet to another provider by setting each instance’s provider, or pin a single agent to a specific model — and no application code changes. Every workload that asked for standard continues to ask for standard.

Un-catalogued model ids (a fresh Bedrock or Nebius id, say) fall back to per-provider heuristics, and the fallback is logged: an entry in the catalogue is the supported way to add a model.

Per-instance override — provider AND specific model

Each agent has two columns on the instances row that drive model selection: provider (openai / anthropic / bedrock / nebius) and model (any model id that provider supports). Both are editable from the admin panel under Settings → Model for the agent. Either can be left null.

How the gateway combines them with the caller’s tier:

final.provider = request.provider     (instance row)  ?? DEFAULT_PROVIDER ("openai")
final.model    = request.model        (instance row)  ?? resolveModel(provider, tier)

The exact model the user picked wins over the tier table — tier becomes a label-only hint when model is set. If model is null, the tier table picks the model (the matrix above). So:

• Operator sets provider="anthropic", model="claude-sonnet-4-5-20250929" → the supervisor’s standard call goes to that exact Sonnet build, regardless of what the tier table says.
• Operator sets provider="anthropic" and leaves model null → the supervisor’s standard call resolves to whatever the tier table says is Anthropic’s standard today.
• Operator leaves both null → the gateway falls back to DEFAULT_PROVIDER (“openai”) and the tier table.

Where the explicit model is and isn’t honoured

This is the part the resolution rule alone hides: not every call site passes the instance’s model. The supervisor does, for the user-facing turn. Background workloads do not.

The asymmetry is intentional: a cheap classifier should stay cheap. If you pin Sonnet on your agent, you do not want every memory-extraction pass to also use Sonnet — that would silently 10× the cost of fire-and-forget background work. The tier table keeps fast-tier calls on the cheap model for the chosen provider, even when the user-facing turn runs on the premium model the operator picked.

If you genuinely need background work to follow the explicit model, the right knob is the tier table in ai-gateway/config.ts, not the instance row.

Capability gates apply to the resolved model

Several per-agent settings are only meaningful on some model builds, and every gate reads the catalogue entry for the resolved model — the instance’s explicit model, or the tier-table standard model when none is pinned. No model-id regex decides behaviour on the request path.

• Thinking. thinkingEnabled is dropped when the resolved model has no reasoning support: a stale true after switching models has no runtime effect, nothing crashes, the feature is simply off. A model that always reasons has no off-switch — only its effort is tunable.
• Reasoning control and levels. How thinking is expressed on the wire (an effort level, a token budget, or the adaptive shape) comes from the catalogue, as does the exact set of levels that model accepts. Both the API and the admin panel validate against that set, so no request goes out with an effort the provider would reject with a 400.
• Temperature. Omitted entirely for models that do not accept the parameter, rather than sent and rejected.
• Vision. Image and file parts are stripped for models that cannot take them.
• Cache. The prompt-cache marker is injected only for families where it is accepted and actually discounted. See Prompt → Prompt caching.

Per-instance secrets

Per-instance API keys live encrypted in instance_secrets: openai_api_key, anthropic_api_key, nebius_api_key, and for AWS-backed services bedrock_api_key plus the dedicated aws_provider_access_key_id / aws_provider_secret_access_key / aws_provider_region namespace — deliberately separate from the generic aws_* secrets a tool may declare, so the AI provider and a tool can use different AWS accounts.

The provider adapters read these from the resolved config and never touch process env directly. Without the right key, the call fails fast at the adapter — there is no fallback to a process-wide key. (The Bedrock embedder is the one documented exception: it falls back to the engine’s AWS_REGION for the region when the per-agent secret is unset.)

Why tier abstraction earns its keep

The same supervisor binary serves many instances at once. Some run on Anthropic, some on OpenAI, some on Bedrock. Some operators upgrade their standard tier from gpt-4o to gpt-4.1 overnight. None of these changes touch supervisor code, prompt templates, or tool implementations.

The tier names also document intent. A grep for tier: "fast" instantly tells you which call paths are cost-sensitive (extractors, classifiers) versus user-facing (standard). When a new optimisation lands — caching, batching, a cheaper model — the operator can roll it out to every “fast” caller in one config change.

Embeddings have their own gateway and their own provider

Embeddings do not follow the chat provider. They go through a separate embeddings gateway with its own per-agent embeddingProvider column, and there are two implementations:

Anthropic and Nebius have no embedding API, so an agent chatting on either still needs one of the two above — but it does not have to be OpenAI. An agent on Bedrock can embed on Bedrock with no OpenAI key at all. Credentials are resolved for the embedder, independently of which model the agent chats with.

New agents default to 1024 dimensions. memories and knowledge_chunks each carry parallel 1024- and 1536-dimension vector columns with a database check constraint ensuring exactly one is populated per row, so both spaces can coexist across agents in one deployment.

Changing the embedder is destructive. Vectors are provider-specific and are deliberately not re-embedded: re-embedding is slow, costly, and easy to get half-right. Instead the old space is abandoned — every memory and the whole knowledge base for that agent (documents, chunks, raw content) are deleted in one transaction and the dimension is realigned to the new provider’s default. Conversations are untouched, so keyword search over raw messages keeps working; only extracted memories go.

Changing only the chat provider never touches embeddings.

Call-type tagging: conversation vs. service

Every gateway call carries a callType in ChatCallOptions:

• "conversation" — user-visible turns. The supervisor handling an inbound message.
• "service" — background work. Memory extraction, title generation, webhook event matching, room event matching, summarisation.

The tag flows through to ai_logs.call_type. Analytics queries can then split billable conversation cost from infrastructure service cost — answering questions like “what does it cost me to keep memory enabled for this instance?” without confusing the per-conversation cost reports.

The supervisor’s main path passes "conversation". Every fire-and-forget extractor passes "service". The convention is enforced by code review, not by the type system.

Logging: every call lands in ai_logs

After every successful chat() or chatStream(), the gateway writes one row into ai_logs (packages/engine/src/ai-gateway/logger.ts):

The logger buffers in memory and flushes every 10 entries or every 5s. On flush failure entries are re-queued; the buffer is capped at 1000 to bound memory.

ai_logs is the source of truth for the admin panel’s cost dashboard and the per-instance analytics pages. It is distinct from pipeline_traces, which records end-to-end pipeline latency including non-LLM phases (context prep, tool building, persistence) — see Architecture.

How it works

+----------------------------------------------------+
| caller declares workload class                     |
|   supervisor.run()      tier: "standard"           |
|   extractMemories()     tier: "fast", call=service |
|   webhookMatcher()      tier: "fast", call=service |
|   titleGenerator()      tier: "fast", call=service |
+--------------------------+-------------------------+
                           |
                           v
+----------------------------------------------------+
| AI Gateway (ai-gateway/index.ts)                   |
|   resolveCallConfig(request, options)              |
|     provider  <- request.provider | instance | default
|     modelId   <- request.model    | resolveModel() |
|     buildLangSmithProviderOptions(...)             |
+--------------------------+-------------------------+
                           |
       +-------------------+-------------------+
       v                   v                   v
   OpenAIProvider   AnthropicProvider  BedrockProvider  NebiusProvider
   (providers/openai.ts) (...anthropic.ts) (...bedrock.ts) (...nebius.ts)
       |                 |                 |                |
       +--------+--------+-----------------+----------------+
                 |
                 v
        Vercel AI SDK (generateText / streamText)
                 |
                 v
   logAndRecordUsage()
       pipelineLog.llmResponse(...)
       estimateCost(provider, model, tokens)
       aiLogger.log({ provider, model, tier, tokens,
                      cost, durationMs, conversationId,
                      instanceId, callType })
                 |
                 v
            ai_logs table (buffered flush every 10 entries / 5s)

Code reference

• packages/engine/src/ai-gateway/index.ts — chat(), chatStream(), resolveCallConfig(), logAndRecordUsage().
• packages/engine/src/ai-gateway/model-catalog.ts — Per-provider tier mapping and per-model rates + capability flags.
• packages/engine/src/ai-gateway/config.ts — resolveModel(), estimateCostBreakdown(), isThinkingCapable(), reasoningControlFor(), reasoningLevelsFor().
• packages/engine/src/ai-gateway/types.ts — ChatRequest, ChatResponse, ChatStreamResult, TierMapping, ProviderAdapter.
• packages/engine/src/ai-gateway/providers/openai.ts — OpenAI adapter (Vercel AI SDK @ai-sdk/openai).
• packages/engine/src/ai-gateway/providers/anthropic.ts — Anthropic adapter (@ai-sdk/anthropic).
• packages/engine/src/ai-gateway/providers/bedrock.ts — Bedrock adapter (@ai-sdk/amazon-bedrock).
• packages/engine/src/ai-gateway/providers/nebius.ts — Nebius Token Factory adapter (OpenAI-compatible).
• packages/engine/src/ai-gateway/logger.ts — ai_logs schema + buffered AILogger.
• packages/engine/src/ai-gateway/langsmith.ts — LangSmith tracing provider options.
• packages/engine/src/instances/config-resolver.ts — Per-instance provider + secrets resolution (30s TTL cache).
• packages/engine/src/embeddings-gateway/ — The separate embeddings gateway: provider resolution, per-provider models and dimensions, and the destructive reset on an embedder switch.

See also

• Architecture — where the gateway sits in the request pipeline.
• Memory — uses tier: "fast" for extraction.
• Agents — the supervisor consumes the gateway via tier: "standard".
• Tools — some tools (e.g. verifyDocument) also call the gateway through their ToolContext.