Polyant - Architecture

Monorepo Structure

This project is an npm workspaces monorepo with two packages:

The root package.json orchestrates both packages. Infrastructure (Docker Compose, .env) lives at the monorepo root.

Overview

Polyant is an open-source platform for building AI assistants with long-term memory, multi-channel support, and per-instance customization. A single instance of the engine can host any number of independently-configured assistants (prompts, skills, tools, channels, secrets), each addressable as an OpenAI-compatible model. Every user interaction is routed through a Supervisor agent that orchestrates tools, sub-agents, and memory to produce contextual and proactive responses.

Tech Stack

Architecture

+------------------------------------------------------------+
|                   HTTP SERVER (NestJS)                      |
|   /v1/chat/completions  |  /v1/models  |  /health          |
|   /api/instances  |  /api/conversations                     |
|   /api/organizations/:orgSlug/...  |  /api/platform/...     |
|   /a2a/:slug/jsonrpc  |  /a2a/:slug/.well-known/...        |
+------------------------------------------------------------+
|                   AUTH + RBAC GUARDS                        |
|   AuthGuard -> PermissionGuard -> OrgRef -> WorkspaceRef     |
|   Scope resolved most-specific-first; fails closed          |
+------------------------------------------------------------+
|                   CHANNEL LAYER                             |
|   Telegram | Slack | WhatsApp (Twilio) | Web (REST/SSE)     |
|   Room | Scheduled | Agent (in-process)                     |
|        ChannelAdapter abstraction                           |
+------------------------------------------------------------+
|                   AGENT LAYER                               |
|   Supervisor (tier: standard, max 15 steps)                 |
|     Tools: searchMemory | webSearch | curl/httpRequest |    |
|            saveMemory | readSkill | readFile/writeFile |    |
|            searchKnowledge | Slack/WhatsApp out |           |
|            scheduleTask | spawnTask | ...                   |
|   Sub-agents: ad-hoc via spawnTask                          |
|   Agent-to-agent: via `agent` channel adapter (not spawnTask)|
+------------------------------------------------------------+
|                   MEMORY LAYER                              |
|   pgvector (cosine similarity) - extracted memories         |
|   PostgreSQL Full-Text Search (keyword) - conversations     |
|   Hybrid Search: RRF (Reciprocal Rank Fusion)              |
|   Native LLM extraction post-response (fire-and-forget)    |
+------------------------------------------------------------+
|                   AI GATEWAY                                |
|   Tier abstraction: fast | standard | heavy                 |
|   Provider: OpenAI | Anthropic | Bedrock                    |
|   Per-instance provider/model override                      |
|   Logging: tokens, costs, latency -> PostgreSQL             |
|   Tracing: LangSmith                                        |
+------------------------------------------------------------+
|                   DATA LAYER                                |
|   PostgreSQL (conversations, memories, instances,           |
|     instance_secrets, instance_channels, ai_logs,           |
|     pipeline_traces)                                        |
|   Tenancy: organizations, workspaces, memberships,          |
|     roles, role_bindings, organization_secrets,             |
|     organization_plugins, organization_skills               |
|   pgvector extension (memory embeddings)                    |
+------------------------------------------------------------+

Tenancy

The hierarchy is Organization > Workspace > Agent. The agent table is still named instances, and an agent’s workspace_id is a non-nullable foreign key - every agent lives in exactly one workspace of exactly one organization.

A person can belong to several organizations. What they may do is decided in the organization the request addresses, resolved from the URL or the X-Org-Slug header, not from the organization recorded in their session - which is only where they land. Within an organization, a workspace binding is authoritative for workspace-tier permissions: it grants what the organization baseline lacks and revokes what it grants, most-specific-wins in both directions.

The workspace segment of the URL is likewise authoritative rather than advisory. The panel sends X-Workspace-Slug on every call, a guard validates it on every route, and there is no active-workspace cookie - a second notion of “the current workspace” is how a URL and a request come to name different ones.

Two things belong to the installation and reach an organization only when it holds them, rather than being shared: plugins and platform-owned skills. An organization installs a marketplace plugin itself, unless a platform admin has denied it there; a plugin baked into the image and a platform skill need a platform admin grant. A plugin or skill the organization does not hold is invisible to it on the management plane, not merely inert at runtime, and both gates fail closed.

Skills and plugins cross two boundaries, not one

An organization grant or installation makes a skill or a plugin assignable; it does not make it available. A second, per-workspace allow-list decides which of them a workspace may draw on, held in workspace_skills and workspace_plugins, and visibility is the intersection of the two:

• the organization must own the skill or hold the grant, and
• the agent’s workspace must have the row assigned.

The two halves are composed in a single predicate rather than checked separately, so it is not possible to consult the workspace assignment without also consulting the organization grant - which matters because assignment rows deliberately survive an organization-level revoke, exactly as an agent’s own rows do, so that a re-grant restores the previous configuration.

The predicate is evaluated on every use at runtime: when the prompt is built, when a skill is read, and when a skill’s script is executed. It filters the management plane’s reads as well, and those reads report an aggregate count of what is missing rather than naming the rows - the names are organization-tier information, and the agent pages that show the count are reachable with workspace-tier permissions.

A workspace created after this shipped starts empty, and a new organization grant reaches no workspace. The single automatic assignment is createSkill, which assigns the skill it just wrote to the creating agent’s own workspace.

Whether a deployment is single-tenant is read from the data - the number of organizations that exist - not from a configuration flag.

See Roles and permissions for the taxonomy and the resolution order.

Two integration directions

An agent is both a client and a server of other agents’ ecosystems, and the two use different protocols on purpose:

• Outward as a client - MCP. An agent consumes tools from external Model Context Protocol servers; the engine does not expose a generic /mcp tool server. Remote tools are namespaced mcp__<server>__<tool>, descriptions are sanitised, URLs are SSRF-checked, and slow servers are skipped. The public MCP route is the OAuth return point, /mcp/oauth/callback.
• Outward as a server - A2A. External clients discover an agent with GET at /a2a/{slug}/.well-known/agent-card.json and invoke it with POST at /a2a/{slug}/jsonrpc. Discovery is public when A2A is enabled. JSON-RPC follows the agent’s API-key setting like /v1: it requires the key when authentication is enabled and is open when authentication is disabled. The task boundary is the organization.

Neither protocol is the in-process agent channel, which is only how agents inside this deployment call each other. External agent clients use A2A.

Directory Structure

packages/engine/
  src/
    index.ts                              # Boot sequence
    config.ts                             # Configuration (Zod schema)
    workspace/
      index.ts                            # Workspace resolver (per-instance parameterization)
    ai-gateway/
      index.ts                            # Gateway init + chat/chatStream
      types.ts                            # ChatRequest, ChatResponse, ProviderAdapter
      config.ts                           # Model resolution + cost estimation
      model-catalog.ts                    # Tier mappings, per-model rates and capabilities
      logger.ts                           # AILogger (batch write to PostgreSQL)
      langsmith.ts                        # LangSmith tracing setup
      providers/
        openai.ts                         # OpenAI via @ai-sdk/openai
        anthropic.ts                      # Anthropic via @ai-sdk/anthropic
        registry.ts                       # registerAiProvider(): providers added at boot
    agents/
      tools/mcp/                          # MCP client: connect, url guard,
                                          #   tool naming, OAuth provider
      supervisor/
        index.ts                          # supervise() + superviseStream()
        prompt.ts                         # System prompt builder (DB-stored sections + skills + datetime)
      tools/
        registry.ts                       # Self-registration system + loadAllTools()
        tools-sync.ts                     # Catalogue reconciliation (orphan cascade)
        # ~35 *.tool.ts files - see reference/tool-catalog.md for the authoritative list
    activity-stream/                       # In-process pub/sub bus + SSE controller + LLM tap
    attachments/                           # Image/document/audio upload + S3 + signed URLs
    audit/                                 # Per-tool-call audit table and scoped AuditLogger
    knowledge/                             # Per-instance knowledge files (search/read/write tools)
    scheduled-tasks/                       # Cron / interval / one-shot scheduler
    stt-gateway/                           # Per-agent voice-note transcription
      providers/
        openai-whisper.ts                  # OpenAI Whisper provider (whisper-1)
        deepgram.ts                        # Deepgram provider
        aws-transcribe.ts                  # Amazon Transcribe provider
    users/                                 # Credentials provider controller + user-management routes + seed
    auth/                                  # NestJS auth module (guard, JWT decryption) + Auth.js v5 schema (users, accounts, sessions)
    workspace/                             # Workspace resolver (ephemeral per-instance dir)
    analytics/                            # Analytics module logic (no controller - endpoints live in server/analytics/)
      traces.schema.ts                    # Drizzle schema (pipeline_traces table)
      trace.store.ts                      # TraceStore: buffered fire-and-forget pipeline trace writer
      latency.store.ts                    # Latency analytics queries (percentiles, phase breakdown, tool stats)
    memory/
      index.ts                            # Entry point: initMemory() + re-exports
      types.ts                            # Memory, ExtractedFact interfaces
      embedder.ts                         # Embedding calls via the embeddings gateway
      memory-store.ts                     # pgvector upsert + cosine similarity dedup
      hybrid-search.ts                    # Hybrid RRF search (pgvector + PG FTS)
      extractor.ts                        # Native LLM extraction post-response
      schema.ts                           # Drizzle schema (memories table with vector column)
    conversations/
      index.ts                            # Re-exports
      store.ts                            # ConversationStore (messages, summaries, FTS)
      schema.ts                           # Drizzle tables (conversations, conversation_messages)
      types.ts                            # Conversation interfaces
    instances/
      schema.ts                           # Drizzle schema (instances table)
      store.ts                            # Instance CRUD
      skill-env.schema.ts                 # Drizzle schema (instance_skill_env table)
      skill-env.store.ts                  # Encrypted skill env CRUD
      secrets.schema.ts                   # Drizzle schema (instance_secrets table)
      secrets.store.ts                    # Encrypted secrets CRUD
      channels.schema.ts                  # Drizzle schema (instance_channels table)
      channels.store.ts                   # Channel config CRUD
      config-resolver.ts                  # Per-instance config with 30s TTL cache
    skills/
      skills.service.ts                   # Global skill library management
      skills.controller.ts                # /api/organizations/:orgSlug/skills CRUD
    crypto/
      index.ts                            # AES-256-GCM encrypt/decrypt
    channels/
      types.ts                            # ChannelAdapter, IncomingMessage, OutgoingMessage
      channel-manager.ts                  # Adapter orchestrator
      adapters/
        telegram/index.ts                 # grammY webhook
        slack/index.ts                    # @slack/bolt Events API webhook
        whatsapp/
          index.ts                        # WhatsApp adapter (webhook)
          twilio-client.ts                # Twilio Programmable Messaging client
    authz/                                # RBAC: permission taxonomy, tiers, guards,
                                          #   role bindings, workspace bindings,
                                          #   entitlements, authorization audit
    organizations/                        # Tenants, memberships, invitations, bootstrap
    users/                                # Accounts, credentials, password reset
    two-factor/                           # TOTP, recovery codes, mandates, sign-in gate
    impersonation/                        # Read-only platform-admin impersonation
    governance/                           # Policy catalogue, gates, compliance
    retention/                            # Per-agent purge policies and runs
    management-audit/                     # Installation-tier write audit
    plugin-system/                        # Manifests, discovery, per-org grants + visibility
    mail/                                 # SMTP transport and templates
    server/
      main.ts                             # NestJS bootstrap
      server.module.ts                    # Root module
      health/health.controller.ts         # GET /health
      a2a/                                # Agent-to-Agent server: agent card,
                                          #   JSON-RPC endpoint, bounded task store
      openai/
        openai.controller.ts              # /v1/chat/completions, /v1/models
        openai.service.ts                 # Chat completion logic
        openai.types.ts                   # Request/response OpenAI types
        openai.module.ts                  # NestJS module
      instances/instances.controller.ts   # /api/instances CRUD + prompts/tools/skills/secrets/channels
      conversations/conversations.controller.ts  # /api/conversations
      analytics/analytics.controller.ts   # /api/analytics + per-instance analytics
      memories/memories.controller.ts     # /memories CRUD
    database/
      client.ts                           # Drizzle connection
      migrations/                         # Generated migrations
    utils/
      pipeline-logger.ts                  # Structured pipeline logging
      frontmatter.ts                      # YAML frontmatter parser for skills
  workspaces/                             # Per-conversation tool sandboxes ONLY (gitignored)
    <instanceId>/                         # Knowledge lives in PostgreSQL (knowledge_documents + knowledge_chunks) - never here
      conversations/<convId>/             # Ephemeral scratch dir used by readFile / writeFile

packages/web/
  src/
    app/
      globals.css                         # Design tokens
      layout.tsx                          # Root layout (Inter font, ThemeProvider, I18nProvider)
      (auth)/                             # login, forgot-password, reset-password, invitation accept
      (force)/                            # password-change, two-factor-setup (forced full-screen flows)
      (admin)/
        layout.tsx                        # Sidebar + header layout
        account/                          # The signed-in person's own account
        organizations/
          page.tsx                        # Organization list (no slug)
          [orgSlug]/
            page.tsx                      # Organization dashboard
            skills/                       # The organization's skill catalogue
            settings/organization/        # Settings destination (general, people,
                                          #   providers, api-keys,
                                          #   workspaces, audit)
            workspaces/
              [workspaceSlug]/
                instances/                # Agents (list + the agent destination)
                conversations/            # Conversation browsing + search
                playground/               # Playground chat
                memory/                   # Memory management
                activity/                 # Live SSE feed
                governance/               # Workspace governance analytics
                audit-logs/               # Tool audit
        platform/                         # Admin Console destination
          analytics/                      # Cross-organization overview
          settings/                       #   organizations, users, plugins,
                                          #   skills, two-factor, audit
    components/
      layout/                             # Sidebar, breadcrumb, nav destinations,
                                          #   tenant switcher, theme/lang toggles
      analytics/                          # Analytics chart components (KPIs, trends, latency)
      ui/                                 # shadcn/ui components
    lib/
      api.ts                              # API client for engine
      access-visibility.ts                # Permission/feature gating of nav + sections
      nav/                                # Sidebar and agent-section registries (data)
      settings/sections.ts                # Configuration-section registry (data)
      tenant/                             # paths.ts, validate.ts, retarget.ts, hooks
      utils.ts                            # cn() helper
      i18n/                               # Italian/English internationalization
    hooks/                                # use-mobile, etc.

The navigation is data, not markup: the sidebar registry, the configuration-section registry, and the agent-section registry are plain arrays, read by the sidebar, the command palette, and the tenant retargeter alike. Nothing can be offered by one surface and missing from another.

Boot Sequence (packages/engine/src/index.ts)

At startup, the system executes in order:

1. AI Gateway - Initializes logging
2. Trace Store - Initializes pipeline latency trace writer (buffered, periodic flush)
3. Tool Loading - Auto-discovers and registers all *.tool.ts files
4. Memory - Verifies pgvector extension is available
5. NestJS Server - OpenAI-compatible API + Management API on configurable port
6. Channel Adapters - Loads enabled channel configs from DB per active instance, starts adapters dynamically

Request Flow

User message
    |
    v
[Channel Adapter] normalizes to IncomingMessage
    |
    v
[Pre-enrichment]
  |- Load conversation summary (in-memory cache -> PostgreSQL)
  '- Create conversation row if missing (fire-and-forget)
    |
    v
[Conversation History] getRecentMessages(conversationId, 16) from PostgreSQL
    |
    v
[Instance Resolution] resolveInstance(instanceId) -> load config from PostgreSQL (cache 30s)
    |
    v
[Supervisor] (tier: standard, max 15 steps)
  |- System prompt: 7 sections from instance_prompts (per-instance, DB-stored)
  |- Skills: discovered from skills + skill_versions + instance_skills (DB joins)
  |- Last 15 messages + new message
  '- Available tools (filtered per-instance via instance_tools):
       searchMemory   -> hybrid pgvector + PG FTS + RRF search
       webSearch      -> web search (Tavily, optional)
       saveMemory     -> explicit save (only on user request)
       updateSoul     -> modify personality
       updateUserProfile -> update user info
       readSkill      -> load a skill's instructions
       spawnTask      -> delegate to sub-agent
       ...            -> channel-specific and integration-specific tools
    |
    v
[Response] sent to channel
    |
    v
[After Response] (async, fire-and-forget)
  |- 1. traceStore.record() -> pipeline latency trace (phase breakdown + tool timings)
  |- 2. appendMessages() -> save to PostgreSQL conversation_messages
  |- 3. Generate updated summary (tier: fast) -> updateSummary()
  '- 4. extractMemories() -> LLM extraction -> embeddings -> pgvector upsert

Streaming: For the HTTP server, superviseStream() returns an AsyncIterable<string> that gets converted to Server-Sent Events (SSE) in OpenAI format.

AI Gateway

Tier Abstraction

Components don’t request specific models. They request a tier:

Background work (titles, memory, summaries) always resolves through the fast tier, so on Anthropic it runs on Claude Haiku 5.5 whether or not the agent pins a model, with thinking switched off. Haiku 5.5 is also catalogued on Bedrock (EU and global profiles), where an agent pins it explicitly.

The exact mappings, and the per-model capabilities and rates behind them, live in packages/engine/src/ai-gateway/model-catalog.ts. Treat the table above as illustrative - the catalogue is the source of truth, and model ids move. The providers available with managed services (TensorX, Scaleway, Regolo.ai) carry their own tier tables in the catalogue they register at boot.

The AI provider is configured per-instance in the admin panel’s Model section. There is no global AI_PROVIDER env var. Individual instances can also override the model via the Management API.

Logging and Costs

Every LLM call, successful or failed, and every embedding call (call_type = embedding), is logged to the ai_logs table with:

• Provider, model, tier
• Token usage (prompt, completion, total), with prompt-cache reads and writes
• Estimated cost in USD (calculated from per-token pricing; a failed call or a turn preempted by a follow-up message carries the cost of the steps it completed). Some models are priced by prompt length: a call to Claude Haiku 5.5 whose prompt passes 100K tokens, or to GPT-5.4, GPT-5.6 or GPT-6 above 272K input tokens, is priced at the provider’s long-prompt rates
• Duration in ms
• Outcome (ok, error, or aborted for a preempted turn) and, for a failure, the error class
• conversationId and instanceId for correlation, and with managed services the provider connection used and the real contact served

With managed services the same cost feeds the agent’s spend caps, checked before each call.

The logger uses a buffer with periodic flush to minimize DB writes.

LangSmith Tracing

LangSmith tracing is per-instance, configured in the admin panel’s Advanced → LangSmith Tracing card (langsmithEnabled, API key, project name). Uses wrapAISDK from langsmith/experimental/vercel to wrap generateText/streamText at module level. Per-request tracing config is built via buildLangSmithProviderOptions() and passed as providerOptions.langsmith to the AI SDK calls. When providerOptions.langsmith is absent (LangSmith disabled), wrapped functions behave identically to the originals with zero overhead. Produces hierarchical traces: parent run per generateText/streamText call, with child runs for each LLM step and tool execution within maxSteps. Thread grouping via metadata.thread_id = conversationId. Instance filtering via metadata.instance_id. Client instances are cached per API key in-memory.

Agent System

Supervisor

The Supervisor is the system’s decision-making center. It receives the user message, the last 15 messages of history, and the conversation summary.

Configuration:

• Tier: standard
• Max steps: 15 (reasoning cycles)
• System prompt: 7 modular sections stored in the instance_prompts table (per-instance, DB-backed). Defaults are seeded from packages/engine/src/instances/defaults.ts on instance creation
• Available tools: filtered per-instance via the instance_tools table (auto-recomputed when skills change)

Tools shipped with the framework (non-exhaustive - the authoritative list is generated from the registry and lives in reference/tool-catalog.md; the actual set per instance is determined by instance_tools):

Channel-specific tools are also included in the framework and can be enabled per-instance. Third-party integrations - HubSpot, GitHub (repository cloning included), Render, Markdown-to-PDF, S3 file upload - arrive as plugins instead, with namespaced tool names, so an installation carries only the ones it wants.

Sub-Agent System

Sub-agents are isolated agents that the Supervisor can delegate tasks to via spawnTask. They receive all enabled tools except spawnTask, to prevent infinite recursion.

There is no dedicated sub-agents/ directory or SubAgentDefinition type today - the spawnTask tool (packages/engine/src/agents/tools/task.tool.ts) creates ad-hoc sub-agents on the fly with a generic system prompt and the parent’s filtered tool set.

Note: spawnTask is unrelated to agent-to-agent calls between different instances, which go through the agent channel adapter (channels/adapters/agent.adapter.ts). See agents.md for the disambiguation.

Memory System

Architecture

The memory system is fully native - no external services required beyond PostgreSQL with pgvector.

Memory Flow

User conversation <-> Supervisor
         |
         v (fire-and-forget after response)
  extractMemories(conversationId, instanceId)
         |
         v
  Load last 15 messages from PostgreSQL
         |
         v
  Send transcript to LLM (tier: fast)
         |
         v
  LLM returns JSON: [{content, category, importance}]
         |
         v
  Generate embeddings (per-instance embedder)
         |
         v
  Upsert each memory into pgvector:
    - Cosine similarity check against existing memories
    - If similarity > 0.90: update existing memory
    - Otherwise: insert new memory

Categories: preference, fact, event, relationship, decision, general Importance: 1-10 scale (10 = critical life fact, 1 = trivial)

Hybrid Search (RRF)

Search combines two backends via Reciprocal Rank Fusion:

                searchMemory(query)
                       |
           +-----------+-----------+
           |                       |
    pgvector semantic search PostgreSQL FTS
    (extracted memories)     (raw conversations)
    cosine similarity        websearch_to_tsquery
    top 20 results           top 20 results
           |                       |
           +-----------+-----------+
                       |
              Reciprocal Rank Fusion
              score = Σ(1 / (k + rank + 1))
              k = 60
                       |
              Sort + dedup + top N
                       |
              HybridSearchResult[]
              {content, type, score, source, createdAt}

Why RRF: The two backends produce scores on different scales (cosine similarity vs ts_rank). RRF implicitly normalizes based only on ranking position, not absolute values.

Complementarity:

• Semantic (pgvector): finds memories by meaning, even with different words
• Keyword (PG FTS): finds conversations by exact words, proper nouns, dates

Note: the embedder is a per-instance setting, independent of the chat provider, chosen from the embedders the server serves: OpenAI text-embedding-3-small or Amazon Titan Text Embeddings v2, and with managed services TensorX or Regolo.ai (Qwen3-Embedding-8B at 1024 dimensions). Further embedders are registered at boot with registerEmbeddingProvider(). Switching it deletes the instance’s memories and knowledge base, because vectors are not converted between provider spaces. Anthropic, Nebius and Scaleway offer no embedder here. The extraction LLM uses the configured provider via ai-gateway.

Conversation System

PostgreSQL Tables

conversations:

• conversationId (text, unique) - format channelType:channelId
• summary (text, nullable) - updated after each turn
• instanceId (text, nullable)
• createdAt, updatedAt

conversation_messages (with Full-Text Search):

• conversationId (text) - indexed
• role (user|assistant)
• content (text)
• toolCalls (jsonb, nullable)
• search_vector (tsvector) - auto-generated from content (config: simple, managed via SQL migration)
• createdAt - indexed
• GIN index on search_vector

ConversationStore

// Summary management (in-memory cache)
getSummary(conversationId)          // cache hit -> return; miss -> query DB -> cache
updateSummary(conversationId)       // write DB + update cache
ensureConversation(id, instanceId)  // INSERT ... ON CONFLICT DO NOTHING
 
// Message management
appendMessages(conversationId, messages[])      // batch insert
getRecentMessages(conversationId, limit=15)     // last N messages (chronological)
 
// Full-text search
searchConversations(query, {instanceId, limit, offset})  // websearch_to_tsquery + ts_rank
 
// Listing
listConversations({instanceId, limit, offset})  // paginated with instance JOIN
getConversation(conversationId)                 // single conversation detail
deleteConversation(conversationId)              // delete conversation + messages

Summary Generation

After each response, an LLM (tier: fast) generates an updated conversation summary in 2-3 sentences. The previous summary is provided as context. This summary is injected into the Supervisor’s system prompt in subsequent conversations.

Channel Layer

Three adapters with a common ChannelAdapter interface:

Channels are DB-driven and per-instance. Config is stored encrypted in the instance_channels table. The channel manager starts/stops adapters dynamically via admin panel or API. No global env vars for channel credentials.

Webhooks & Event Sources

External systems (HubSpot, custom APIs, etc.) can trigger agent actions via webhooks. A webhook is received at POST /webhooks/:webhookToken (always returns 200 OK immediately, processing is fire-and-forget). The payload (max 64KB) is passed to a webhook matcher LLM (tier: fast) which sequentially evaluates matching definitions (first match wins). Matched events are inserted into the event_backlog table with pending status, where they await processing by the Room scheduler.

Key components:

• webhook-engine.ts - Central dispatch and queue management
• webhook-matcher.ts - LLM-based event classification against priority-ordered definitions
• webhook-sources.store.ts - Event source CRUD (with AES-256-GCM encrypted config)
• webhook-backlog.store.ts - Pending event queue with status lifecycle (pending → processing → completed)
• webhook.validators.ts, webhooks.schema.ts - Payload validation and schemas
• template-renderer.ts, trigger-context.ts - Context building for interpretation
• active-triggers.ts, webhook-logger.ts - Trigger state and audit logging

Rate limit: 60 events/min. Backlog capacity: 100 pending events per instance (excess dropped).

Scheduled Tasks

The scheduler provides time-based and event-based task execution with optional result delivery via outbound channels.

Schema:

• schedule_type: cron (recurring), interval (every N seconds), or oneShot (run once then delete)
• status: enabled, disabled (suspended), error (automatically disabled after N consecutive failures)
• retry_count, retry_delay_ms, max_consecutive_errors - Backoff and auto-disable logic
• result_channel, result_target - Optional outbound channel for delivery (e.g., Slack, WhatsApp, email)

Key components:

• scheduler.service.ts - Singleton scheduler with per-task queuing and tick-based evaluation
• store.ts - Task CRUD and status management
• schedule-utils.ts - Cron parsing and interval calculation
• run-log.store.ts - Execution history (run_id, start_time, duration, status, output)

Tick interval is configurable. Each completed run can send its result to a configured channel (e.g., notification via Slack). Automatic disable occurs after max_consecutive_errors consecutive failures.

Activity Stream (SSE)

The activity stream provides real-time updates to the admin panel via Server-Sent Events (SSE), replacing the prior polling-based feed. Emits structured events from the agent pipeline: inbound/outbound messages, tool calls, memory extraction, scheduled-task fires, webhook matches, and agent-to-agent handoffs.

Components:

• bus-emitter.ts - Event bus with subscription/unsubscription
• event-formatters.ts - Transforms domain events into SSE payloads
• Controller endpoint - GET /api/activity-stream/live with client-managed reconnection

Used by the admin panel’s activity dashboard for live visibility without polling overhead.

Attachment Pipeline

Messages can carry non-text payloads (images, documents, audio) from inbound channels. WhatsApp media (via Twilio MediaUrl) is downloaded, validated, and uploaded to the agent’s own S3 bucket. Outbound attachment references are served via a reverse proxy endpoint or signed S3 URLs to prevent direct exposure.

Data flow:

1. Channel adapter (WhatsApp) extracts MediaUrl and metadata from inbound message
2. Binary blob is uploaded to that agent’s bucket via agent-storage.ts, whose credentials come from the agent’s own secrets (s3_bucket_name, aws_region, aws_access_key_id and aws_secret_access_key; the deployment’s own AWS identity is never used). An agent with no bucket configured stores nothing - the attachment is left as a remote reference
3. Attachment metadata (key, mimeType, size, etc.) is appended to the attachments JSONB column on the corresponding conversation_messages row
4. Outbound delivery uses signed S3 URLs

Key components:

• packages/engine/src/attachments/agent-s3.ts - the single place an agent’s bucket and static keys are resolved from its secrets, and where a leftover s3_use_task_role is reported as no longer supported.
• packages/engine/src/attachments/agent-storage.ts - S3 helpers (upload + read), with a per-agent client cache invalidated on every secret write. No dedicated controller, no message_attachments table.
• Attachment storage - jsonb attachments column on conversation_messages (see packages/engine/src/conversations/schema.ts); each entry holds the S3 key plus metadata.

HTTP Server (NestJS)

OpenAI-Compatible API

Management API - Instances

Management API - Conversations

Management API - Secrets

Management API - Channels

Management API - Analytics

Management API - Memories

Management API - Skills

A skill is owned by an organization or by the platform, never by the deployment at large, so there are two route families:

Reads are the union of what an organization owns and what it has been granted; writes stay owner-exact and answer 403 on a granted platform skill, since the caller can legitimately see the row. On a slug collision the organization’s own skill wins - a grant is additive and must never change what an agent already resolves.

A revoke stops the skill being served, not merely listed: prompt assembly and both skill tools re-check visibility, resolving the organization from the agent’s own workspace and failing closed. The agents’ assignment rows survive, so a re-grant restores the configuration exactly.

The platform routes are gated on the users.is_platform_admin flag rather than a permission - a grantable right to widen a tenant’s capabilities would itself be a privilege-escalation surface. The flag is resolved from the database per check, not read off the session token, so a revocation is not frozen into a live session.

Authentication and authorization

The engine accepts three kinds of caller: an Auth.js session (cookie or bearer), an organization-scoped management API key (X-Polyant-Key), and a per-agent key. A fourth, the ALB gateway identity (x-amzn-oidc-data), exists in the code but AUTH_MODE=alb-oidc is refused at boot - it resolved a tenancy only on a single-organization deployment.

Every management route declares what it requires, and the guard fails closed in both directions: a denied permission is refused, and so is a route that declares nothing at all. Five declarations satisfy it - a public route, an RBAC permission, an installation-role check, “any authenticated human”, and “a service principal only”.

The scope a permission is decided in is resolved most-specific-first: the agent named in the path, else the organization named in the path or the X-Org-Slug header, else the organization recorded in the token. That last one is only a landing preference - see Roles and permissions.

Per-agent auth is separate and unchanged: if authEnabled is set on an agent, calls to /v1/* require Authorization: Bearer <auth_api_key>. If it is false, access is open for that agent.

Infrastructure (Docker Compose)

services:
  postgres:     # PostgreSQL 16 (pgvector), port 5432

Persistent volume: polyant-pgdata (PostgreSQL).

Configuration (.env)

Only infrastructure variables remain in .env. AI provider keys, LangSmith, auth, Tavily, and channel credentials are configured per-instance in the admin panel (the agent’s Model, Channels and Tools sections).

# Database
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=polyant
POSTGRES_USER=polyant
POSTGRES_PASSWORD=changeme
 
# HTTP Server
API_PORT=4000
 
# Encryption
ENCRYPTION_KEY=...              # 32-byte hex key for AES-256-GCM (instance secrets)

There is no default-agent variable: every caller names its agent (the OpenAI-compatible route answers 400 without a model). The deployment’s time zone is the runtime’s TZ; each agent can override it under Advanced → Behaviour overrides.

Design Patterns and Technical Choices

1. Native Memory Extraction

Memory extraction runs entirely in-process: the project’s LLM (tier: fast) extracts structured facts from conversation transcripts, OpenAI generates embeddings, and pgvector stores them with cosine similarity deduplication. No external memory services required.

2. Hybrid Search with RRF

Search combines pgvector (semantic) and PostgreSQL FTS (keyword) via Reciprocal Rank Fusion. This approach implicitly normalizes heterogeneous scores based only on rankings.

3. Real-time Extraction (not Batch)

After each supervisor response, a fire-and-forget process extracts memories from the last 15 messages. No scheduled jobs, no nightly batches.

4. PostgreSQL for Conversation Storage

All messages are saved in PostgreSQL (conversation_messages). The auto-generated tsvector column enables full-text search without additional components.

5. Tier Abstraction (not Model Binding)

Components request a tier (fast, standard, heavy), not a specific model. The tier-to-model mapping is centralized in the model catalogue, packages/engine/src/ai-gateway/model-catalog.ts. Instances can override provider/model for fine-grained control.

6. Pipeline Latency Tracing

Every user message (excluding auto-tasks like Open WebUI title/summary generation) is instrumented with per-phase timing. The TraceStore buffers entries and flushes to pipeline_traces in batches (every 10 entries or every 5 seconds), following the same pattern as AILogger.

Phases tracked: context prep, tool building, LLM call, total. Additional data: individual tool call durations (toolCallTraces JSONB), streaming TTFB, token counts.

The pipeline_traces table is separate from ai_logs: ai_logs tracks individual LLM API calls (including background tasks like summary generation), while pipeline_traces tracks end-to-end pipeline latency for user-facing messages only.

Analytics queries in latency.store.ts use PostgreSQL percentile_cont() for p50/p95/p99 and jsonb_array_elements() for tool call breakdown. Results are served alongside existing analytics via both global and per-instance endpoints.

7. Fire-and-Forget Post-Processing

The afterResponse() function runs asynchronously without blocking the response: saves messages, updates summary, extracts memories. If it fails, the error is logged but the user has already received the response.

8. OpenAI-Compatible API

The server exposes endpoints in OpenAI format (/v1/chat/completions, /v1/models). Compatible with Open WebUI, ChatBox, and any OpenAI-compatible client.

9. Self-Registering Tools

Tools are defined as *.tool.ts files that default-export defineTool() from @polyant-ai/plugin-sdk. The Zod schema is serialized to JSON Schema at module load, so the engine consumes data rather than a live Zod object across a package boundary - which is what lets a third-party plugin resolve its own copy of the SDK safely. A file that does not default-export a defineTool result is skipped at boot; there is no legacy registration form. They are auto-discovered at boot by loadAllTools(). No hardcoded imports or manual wiring needed. Each tool can declare requiredEnv - if the env var is missing, the tool is silently excluded.

10. Instance Personalization (Database-First)

All instance configuration - prompts, skills, tool enablement, secrets, channel credentials, and knowledge documents - is stored in PostgreSQL, not on the filesystem. New instances are seeded from the defaults defined in packages/engine/src/instances/defaults.ts. The workspaces/<instanceId>/ directory exists only as a sandbox root for per-conversation tool work (readFile / writeFile under conversations/<convId>/); it is never the source of truth for any agent configuration.

11. Encrypted Skill Environment Variables

Skill requiredEnv values are scoped per agent and skill in instance_skill_env. Sensitive entries are AES-256-GCM encrypted at rest. readSkill emits {{skill_env.<skill>.<KEY>}}; the runtime substitutes the plaintext only after tool-input validation and immediately before eligible in-process execution, leaving the placeholder in audit arguments and conversation history. Non-sensitive entries may stay inline, while runSkillScript receives its environment only in the spawned child process.

Main Dependencies

Commands

All commands run from the monorepo root and delegate to the appropriate workspace:

# Engine (AI runtime)
npm run dev              # Start engine with tsx watch
npm run dev:engine       # Same as above (explicit)
npm run build:engine     # Build engine only
npm start                # Run engine from dist/
 
# Web (admin panel)
npm run dev:web          # Start Next.js dev server
npm run build:web        # Build web only
 
# All workspaces
npm run build            # Build all packages
npm run lint             # ESLint all packages
npm run typecheck        # TypeScript check all packages
npm test                 # Run all tests
 
# Database (engine)
npm run db:generate      # Generate Drizzle migrations
npm run db:migrate       # Apply migrations
npm run db:studio        # Drizzle Studio GUI
 
# Engine tests
npm run test:unit        # Unit tests only
npm run test:integration # Integration tests only
npm run test:functional  # Functional tests only
 
# Docker
docker compose up -d     # Start postgres + open-webui

Room & Event Sources

The Room system enables proactive, event-driven agent behavior - the agent doesn’t just respond to user messages, it listens to external system events and takes initiative.

Data Flow

External System (HubSpot, etc.)
    │
    ▼
POST /webhooks/:webhookToken        ← always returns 200 OK (fire-and-forget)
    │
    ▼
WebhookController.processEvent()    ← validation cascade:
    │                                   token → source enabled → room enabled
    │                                   → backlog cap (100) → definitions → slug
    ▼
Event Matcher (LLM tier: fast)      ← sequential, first-match-wins
    │
    ▼
Event Backlog (status: pending)     ← queue in PostgreSQL
    │
    ▼
Room Scheduler (30s tick)           ← per-room mutex, parallel across rooms
    │
    ▼
Room Engine (executeRoomCycle)      ← builds synthetic message with:
    │                                   pending events + interpretation prompts
    │                                   + human message (if any)
    ▼
Supervisor (standard LLM call)      ← same supervisor as user messages,
    │                                   memory disabled for room cycles
    ▼
Outbound Channel                    ← Slack / WhatsApp / Telegram
    │
    ▼
Human                               ← reply routes back via triggerImmediate()

Key Design Decisions

• One Room per instance: stored in instance_room (1:1 with instances). Room prompt defines the agent’s mandate for event processing.
• Event Sources are generic: source_type field supports extensible source types (currently hubspot, webhook). Config is AES-256-GCM encrypted.
• Event Definitions are priority-ordered: sequential LLM evaluation, first match wins. Each definition has a matchingPrompt (for the classifier) and an interpretationPrompt (injected into the Room cycle).
• Activity log auto-compacts: daily entries kept 7 days → merged into weekly (4 weeks) → merged into monthly (12 months) → deleted.
• Room scheduler is a singleton: per-room mutex prevents concurrent processing of the same room. Different rooms run in parallel.

Database Tables

API Endpoints

Extension Points