Polyant — Architecture

+------------------------------------------------------------+
|                   HTTP SERVER (NestJS)                      |
|   /v1/chat/completions  |  /v1/models  |  /health          |
|   /api/instances  |  /api/conversations  |  /api/skills     |
+------------------------------------------------------------+
|                   CHANNEL LAYER                             |
|   Telegram | Slack | WhatsApp (Twilio) | Web (REST/SSE)     |
|   Room | Scheduled | Agent (in-process)                     |
|        ChannelAdapter abstraction                           |
+------------------------------------------------------------+
|                   AGENT LAYER                               |
|   Supervisor (tier: standard, max 15 steps)                 |
|     Tools: searchMemory | webSearch | curl/httpRequest |    |
|            saveMemory | readSkill | readFile/writeFile |    |
|            HubSpot suite | GitHub | Slack/WhatsApp out |    |
|            scheduleTask | spawnTask | ...                   |
|   Sub-agents: ad-hoc via spawnTask                          |
|   Agent-to-agent: via `agent` channel adapter (not spawnTask)|
+------------------------------------------------------------+
|                   MEMORY LAYER                              |
|   pgvector (cosine similarity) — extracted memories         |
|   PostgreSQL Full-Text Search (keyword) — conversations     |
|   Hybrid Search: RRF (Reciprocal Rank Fusion)              |
|   Native LLM extraction post-response (fire-and-forget)    |
+------------------------------------------------------------+
|                   AI GATEWAY                                |
|   Tier abstraction: fast | standard | heavy                 |
|   Provider: OpenAI | Anthropic | Bedrock                    |
|   Per-instance provider/model override                      |
|   Logging: tokens, costs, latency -> PostgreSQL             |
|   Tracing: LangSmith                                        |
+------------------------------------------------------------+
|                   DATA LAYER                                |
|   PostgreSQL (conversations, memories, instances,           |
|     instance_secrets, instance_channels, ai_logs,           |
|     pipeline_traces)                                        |
|   pgvector extension (memory embeddings)                    |
+------------------------------------------------------------+

packages/engine/
  src/
    index.ts                              # Boot sequence
    config.ts                             # Configuration (Zod schema)
    workspace/
      index.ts                            # Workspace resolver (per-instance parameterization)
    ai-gateway/
      index.ts                            # Gateway init + chat/chatStream
      types.ts                            # ChatRequest, ChatResponse, ProviderAdapter
      config.ts                           # Model tier mappings + pricing
      logger.ts                           # AILogger (batch write to PostgreSQL)
      langsmith.ts                        # LangSmith tracing setup
      providers/
        openai.ts                         # OpenAI via @ai-sdk/openai
        anthropic.ts                      # Anthropic via @ai-sdk/anthropic
    agents/
      supervisor/
        index.ts                          # supervise() + superviseStream()
        prompt.ts                         # System prompt builder (DB-stored sections + skills + datetime)
      tools/
        registry.ts                       # Self-registration system + loadAllTools()
        tools-sync.ts                     # Catalogue reconciliation (orphan cascade)
        # ~35 *.tool.ts files — see reference/tool-catalog.md for the authoritative list
    activity-stream/                       # In-process pub/sub bus + SSE controller + LLM tap
    attachments/                           # Image/document/audio upload + S3 + signed URLs
    audit/                                 # Per-tool-call audit table and scoped AuditLogger
    knowledge/                             # Per-instance knowledge files (search/read/write tools)
    scheduled-tasks/                       # Cron / interval / one-shot scheduler
    stt-gateway/                           # Voice-note transcription gateway
      providers/
        openai-whisper.ts                  # Default Whisper provider (whisper-1)
        deepgram.ts                        # Alternative provider
        aws-transcribe.ts                  # Alternative provider
    users/                                 # Credentials provider controller + user-management routes + seed
    auth/                                  # NestJS auth module (guard, JWT decryption) + Auth.js v5 schema (users, accounts, sessions)
    workspace/                             # Workspace resolver (ephemeral per-instance dir)
    analytics/                            # Analytics module logic (no controller — endpoints live in server/analytics/)
      traces.schema.ts                    # Drizzle schema (pipeline_traces table)
      trace.store.ts                      # TraceStore: buffered fire-and-forget pipeline trace writer
      latency.store.ts                    # Latency analytics queries (percentiles, phase breakdown, tool stats)
    memory/
      index.ts                            # Entry point: initMemory() + re-exports
      types.ts                            # Memory, ExtractedFact interfaces
      embedder.ts                         # OpenAI embeddings (text-embedding-3-small)
      memory-store.ts                     # pgvector upsert + cosine similarity dedup
      hybrid-search.ts                    # Hybrid RRF search (pgvector + PG FTS)
      extractor.ts                        # Native LLM extraction post-response
      schema.ts                           # Drizzle schema (memories table with vector column)
    conversations/
      index.ts                            # Re-exports
      store.ts                            # ConversationStore (messages, summaries, FTS)
      schema.ts                           # Drizzle tables (conversations, conversation_messages)
      types.ts                            # Conversation interfaces
    instances/
      schema.ts                           # Drizzle schema (instances table)
      store.ts                            # Instance CRUD
      skill-env.schema.ts                 # Drizzle schema (instance_skill_env table)
      skill-env.store.ts                  # Encrypted skill env CRUD
      secrets.schema.ts                   # Drizzle schema (instance_secrets table)
      secrets.store.ts                    # Encrypted secrets CRUD
      channels.schema.ts                  # Drizzle schema (instance_channels table)
      channels.store.ts                   # Channel config CRUD
      config-resolver.ts                  # Per-instance config with 30s TTL cache
    skills/
      skills.service.ts                   # Global skill library management
      skills.controller.ts                # /api/skills CRUD endpoints
    crypto/
      index.ts                            # AES-256-GCM encrypt/decrypt
    channels/
      types.ts                            # ChannelAdapter, IncomingMessage, OutgoingMessage
      channel-manager.ts                  # Adapter orchestrator
      adapters/
        telegram/index.ts                 # grammY long polling
        slack/index.ts                    # @slack/bolt Socket Mode
        whatsapp/
          index.ts                        # WhatsApp adapter (webhook)
          twilio-client.ts                # Twilio Programmable Messaging client
    server/
      main.ts                             # NestJS bootstrap
      server.module.ts                    # Root module
      health/health.controller.ts         # GET /health
      openai/
        openai.controller.ts              # /v1/chat/completions, /v1/models
        openai.service.ts                 # Chat completion logic
        openai.types.ts                   # Request/response OpenAI types
        openai.module.ts                  # NestJS module
      instances/instances.controller.ts   # /api/instances CRUD + prompts/tools/skills/secrets/channels
      conversations/conversations.controller.ts  # /api/conversations
      analytics/analytics.controller.ts   # /api/analytics + per-instance analytics
      memories/memories.controller.ts     # /memories CRUD
    database/
      client.ts                           # Drizzle connection
      migrations/                         # Generated migrations
    utils/
      pipeline-logger.ts                  # Structured pipeline logging
      frontmatter.ts                      # YAML frontmatter parser for skills
  workspaces/                             # Per-conversation tool sandboxes ONLY (gitignored)
    <instanceId>/                         # Knowledge lives in PostgreSQL (knowledge_documents + knowledge_chunks) — never here
      conversations/<convId>/             # Ephemeral scratch dir used by readFile / writeFile / gitCloneRepo

packages/web/
  src/
    app/
      globals.css                         # Design tokens
      layout.tsx                          # Root layout (Inter font, ThemeProvider, I18nProvider)
      (admin)/
        layout.tsx                        # Sidebar + header layout
        page.tsx                          # Dashboard
        instances/                        # Instance management (list, detail, tabs)
        conversations/                    # Conversation browsing + search
        skills/                           # Global skill library CRUD
        playground/page.tsx                # Playground chat page
        memory/page.tsx                   # Memory management (list, search, create/delete)
        settings/page.tsx                 # Global settings
    components/
      layout/                             # Sidebar, header, nav, theme/lang toggles
      analytics/                          # Analytics chart components (KPIs, trends, latency)
      ui/                                 # shadcn/ui components
    lib/
      api.ts                              # API client for engine
      utils.ts                            # cn() helper
      i18n/                               # Italian/English internationalization
    hooks/                                # use-mobile, etc.

User message
    |
    v
[Channel Adapter] normalizes to IncomingMessage
    |
    v
[Pre-enrichment]
  |- Load conversation summary (in-memory cache -> PostgreSQL)
  '- Create conversation row if missing (fire-and-forget)
    |
    v
[Conversation History] getRecentMessages(conversationId, 15) from PostgreSQL
    |
    v
[Instance Resolution] resolveInstance(instanceId) -> load config from PostgreSQL (cache 30s)
    |
    v
[Supervisor] (tier: standard, max 15 steps)
  |- System prompt: 8 sections from instance_prompts (per-instance, DB-stored)
  |- Skills: discovered from skills + skill_versions + instance_skills (DB joins)
  |- Last 15 messages + new message
  '- Available tools (filtered per-instance via instance_tools):
       searchMemory   -> hybrid pgvector + PG FTS + RRF search
       webSearch      -> web search (Tavily, optional)
       saveMemory     -> explicit save (only on user request)
       updateSoul     -> modify personality
       updateUserProfile -> update user info
       readSkill      -> load a skill's instructions
       spawnTask      -> delegate to sub-agent
       ...            -> channel-specific and integration-specific tools
    |
    v
[Response] sent to channel
    |
    v
[After Response] (async, fire-and-forget)
  |- 1. traceStore.record() -> pipeline latency trace (phase breakdown + tool timings)
  |- 2. appendMessages() -> save to PostgreSQL conversation_messages
  |- 3. Generate updated summary (tier: fast) -> updateSummary()
  '- 4. extractMemories() -> LLM extraction -> embeddings -> pgvector upsert

User conversation <-> Supervisor
         |
         v (fire-and-forget after response)
  extractMemories(conversationId, instanceId)
         |
         v
  Load last 15 messages from PostgreSQL
         |
         v
  Send transcript to LLM (tier: fast)
         |
         v
  LLM returns JSON: [{content, category, importance}]
         |
         v
  Generate embeddings (OpenAI text-embedding-3-small)
         |
         v
  Upsert each memory into pgvector:
    - Cosine similarity check against existing memories
    - If similarity > 0.90: update existing memory
    - Otherwise: insert new memory

                searchMemory(query)
                       |
           +-----------+-----------+
           |                       |
    pgvector semantic search PostgreSQL FTS
    (extracted memories)     (raw conversations)
    cosine similarity        websearch_to_tsquery
    top 20 results           top 20 results
           |                       |
           +-----------+-----------+
                       |
              Reciprocal Rank Fusion
              score = Σ(1 / (k + rank + 1))
              k = 60
                       |
              Sort + dedup + top N
                       |
              HybridSearchResult[]
              {content, type, score, source, createdAt}

// Summary management (in-memory cache)
getSummary(conversationId)          // cache hit -> return; miss -> query DB -> cache
updateSummary(conversationId)       // write DB + update cache
ensureConversation(id, instanceId)  // INSERT ... ON CONFLICT DO NOTHING
 
// Message management
appendMessages(conversationId, messages[])      // batch insert
getRecentMessages(conversationId, limit=15)     // last N messages (chronological)
 
// Full-text search
searchConversations(query, {instanceId, limit, offset})  // websearch_to_tsquery + ts_rank
 
// Listing
listConversations({instanceId, limit, offset})  // paginated with instance JOIN
getConversation(conversationId)                 // single conversation detail
deleteConversation(conversationId)              // delete conversation + messages

services:
  postgres:     # PostgreSQL 16 (pgvector), port 5432

# Database
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=polyant
POSTGRES_USER=polyant
POSTGRES_PASSWORD=changeme
 
# HTTP Server
API_PORT=4000
 
# Encryption
ENCRYPTION_KEY=...              # 32-byte hex key for AES-256-GCM (instance secrets)
 
# Instance
DEFAULT_INSTANCE_ID=default     # default instanceId for single-instance setup

# Engine (AI runtime)
npm run dev              # Start engine with tsx watch
npm run dev:engine       # Same as above (explicit)
npm run build:engine     # Build engine only
npm start                # Run engine from dist/
 
# Web (admin panel)
npm run dev:web          # Start Next.js dev server
npm run build:web        # Build web only
 
# All workspaces
npm run build            # Build all packages
npm run lint             # ESLint all packages
npm run typecheck        # TypeScript check all packages
npm test                 # Run all tests
 
# Database (engine)
npm run db:generate      # Generate Drizzle migrations
npm run db:migrate       # Apply migrations
npm run db:studio        # Drizzle Studio GUI
 
# Engine tests
npm run test:unit        # Unit tests only
npm run test:integration # Integration tests only
npm run test:functional  # Functional tests only
 
# Docker
docker compose up -d     # Start postgres + open-webui

External System (HubSpot, etc.)
    │
    ▼
POST /webhooks/:webhookToken        ← always returns 200 OK (fire-and-forget)
    │
    ▼
WebhookController.processEvent()    ← validation cascade:
    │                                   token → source enabled → room enabled
    │                                   → backlog cap (100) → definitions → slug
    ▼
Event Matcher (LLM tier: fast)      ← sequential, first-match-wins
    │
    ▼
Event Backlog (status: pending)     ← queue in PostgreSQL
    │
    ▼
Room Scheduler (30s tick)           ← per-room mutex, parallel across rooms
    │
    ▼
Room Engine (executeRoomCycle)      ← builds synthetic message with:
    │                                   pending events + interpretation prompts
    │                                   + human message (if any)
    ▼
Supervisor (standard LLM call)      ← same supervisor as user messages,
    │                                   memory disabled for room cycles
    ▼
Outbound Channel                    ← Slack / WhatsApp / Telegram
    │
    ▼
Human                               ← reply routes back via triggerImmediate()