Observe with LangSmith

LangSmith is Polyant’s supported tracing backend. Every supervisor turn is exported as a root span, with sub-spans for each tool invocation and reasoning step — so you can replay a conversation, inspect prompts, and measure latency at every layer.

LangSmith config is per-instance and DB-only. There are NO LANGSMITH_* env vars that the engine reads — everything lives on the instance row and in instance_secrets. This guide walks through the setup, what gets traced, and the trade-offs.

Prerequisites

A LangSmith account at smith.langchain.com .
An admin session on Polyant.
An instance already provisioned (e.g. my-assistant).

Step 1 — Create a LangSmith project and API key

Sign in at smith.langchain.com .
Create a new Project (e.g. polyant-prod-my-assistant). One project per Polyant instance is the common pattern — it keeps trace search tidy.
Go to Settings → API Keys → Create API Key. Copy the key (it starts with ls_ or lsv2_). LangSmith will not show it again.

Step 2 — Configure the instance

You can do this from the admin panel or via the API.

Via the admin panel

Open the instance → Settings tab.
Toggle LangSmith Tracing on.
Paste the API key into the LangSmith API Key field.
Enter the LangSmith Project name exactly as it appears in LangSmith.
Click Save.

Via the API


# 1. Toggle tracing on + set the project
curl -s -X PATCH http://localhost:4000/api/instances/my-assistant \
  -H "Content-Type: application/json" \
  -H "Cookie: authjs.session-token=$TOKEN" \
  -d '{ "langsmithEnabled": true, "langsmithProject": "polyant-prod-my-assistant" }'
 
# 2. Store the API key
curl -s -X PUT http://localhost:4000/api/instances/my-assistant/secrets \
  -H "Content-Type: application/json" \
  -H "Cookie: authjs.session-token=$TOKEN" \
  -d '{ "secrets": [{ "key": "langsmith_api_key", "value": "lsv2_..." }] }'

The config-resolver caches per-instance secrets for 30 seconds, so the next message you send (or wait up to 30s) will start emitting traces.

Step 3 — Send a test message and watch the trace

Use the Playground, or hit the API directly:


curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-assistant",
    "messages": [{ "role": "user", "content": "what tools do you have?" }]
  }'

Open your project on smith.langchain.com — a new root run should appear within a few seconds.

What gets traced

The engine wires up tracing via wrapAISDK (in ai-gateway/langsmith.ts) and passes the project per call through providerOptions.langsmith (in ai-gateway/index.ts). The result is:

One root span per supervisor turn at LLM tier standard — covers the planning step that produced the final user-visible reply.
Sub-spans for each tool invocation — name, input args, output, duration.
Sub-spans for any reasoning steps the SDK emits (e.g. multi-step tool loops).
A separate -service thread suffix for background calls (memory extraction, title generation). They are routed through their own thread name so they do not pollute the per-conversation view in LangSmith.

What is NOT in the LangSmith trace:

Token usage and model metadata are explicitly omitted at the LangSmith integration layer to avoid double-pricing in LangSmith’s own UI; the AI SDK middleware captures usage on the child runs instead. Use Polyant’s ai_logs + pipeline_traces tables, or LangSmith’s run-level metrics, for cost analysis.
The Room scheduler and webhook ingestion paths trace under separate conversation IDs (room:{instanceId}:{timestamp}) — useful for keeping proactive runs distinct from user chats.

Step 4 — Disable tracing for one instance


curl -s -X PATCH http://localhost:4000/api/instances/my-assistant \
  -H "Content-Type: application/json" \
  -d '{ "langsmithEnabled": false }'

The toggle is read on every supervisor turn (subject to the 30s config cache). No engine restart needed. The API key stays in instance_secrets so re-enabling later is one PATCH away.

Trade-offs

Latency: tracing adds roughly 50–150 ms of overhead per turn (network export to LangSmith). For latency-sensitive workloads, keep it on in staging and disable selectively in production.
Retention: LangSmith retention is governed by your LangSmith plan, not by Polyant. Export traces you care about long-term.
Data exposure: prompts, tool args, and tool outputs are sent verbatim to LangSmith. If your instance handles PII or regulated data, gate the toggle per instance — langsmithEnabled: false keeps the data inside your own database.
Per-instance scope: each instance can point to a different project, with a different API key. There is no global “turn tracing on for all instances” switch — this is deliberate, so compliance choices are explicit per workload.

Verification

GET /api/instances/my-assistant returns .instance.langsmithEnabled: true and the expected .instance.langsmithProject (the instance object is wrapped under an instance key).
GET /api/instances/my-assistant/secrets returns a secrets array; jq '.secrets[] | select(.key == "langsmith_api_key")' should show configured: true.
A fresh message in the Playground produces a new run in LangSmith within ~5 seconds.
The run shows the supervisor span at the root, with one child span per tool call.