Architecture: third-party integration infrastructure

Phase 1 output. Defines the system that lets external tools push artifacts into kongruity and read ranked clusters back out. Scope is the shared infrastructure only — no individual vendor integration is built here.

Constraints inherited from the existing system

Express 4 on Node, ESM, strict TypeScript (backend/tsconfig.json), compiled to dist/.
PostgreSQL through a single pg.Pool in backend/db/index.ts.
No validation libraries permitted (agents.md section 4) — runtime narrowing uses hand-written type predicates.
Today the API is read-only and unauthenticated: backend/app.ts mounts one router exposing GET /v1/notes and POST /v1/notes/cluster.
The notes table already carries a source_meta JSONB column, which becomes the provenance record for ingested artifacts.

Component graph

flowchart TD
  subgraph external [External systems]
    Provider["Provider (Slack, Jira, ...)"]
    Client["REST / Zapier client"]
  end

  subgraph edge [Edge: trust boundary]
    RawBody["express.raw on /v1/webhooks"]
    Signatures["lib/signatures"]
    ApiKey["middleware/apiKey"]
  end

  subgraph core [Core]
    WebhookRoutes["routes/webhooks.routes"]
    IngestRoutes["routes/ingest.routes"]
    Registry["config/providers"]
    Queue["lib/queue"]
    Normalize["services/normalize.service"]
    OAuth["services/oauth.service"]
    HttpClient["lib/httpClient"]
  end

  subgraph data [Data]
    IngestEvents["ingest_events"]
    Integrations["integrations"]
    Notes["notes"]
  end

  Provider --> RawBody --> Signatures --> WebhookRoutes
  Client --> ApiKey --> IngestRoutes
  WebhookRoutes --> IngestEvents
  WebhookRoutes --> Queue
  Queue --> Normalize
  Normalize --> Registry
  Normalize --> Notes
  IngestRoutes --> Notes
  Signatures --> Registry
  OAuth --> Integrations
  ApiKey --> Integrations
  OAuth --> HttpClient
  Notes --> Clustering["services/clustering.service"]

Inbound request flow

Two doors, one destination.

Webhook path. A provider POSTs to /v1/webhooks/:provider. The raw body parser runs first so the exact bytes survive for HMAC comparison. lib/signatures looks the provider up in the registry and verifies the signature plus a timestamp replay window. Challenge and handshake requests short-circuit with the provider's expected response. Everything else records an ingest_events row keyed on (provider, external_id), returns 200 immediately, and enqueues the payload. The queue worker normalizes it into NoteInput[] and writes through the existing createNotes DAO.

REST path. A client POSTs to /v1/notes with a bearer key. middleware/apiKey hashes the presented key and compares it against integrations.api_key_hash in constant time, attaching the resolved integration to the request. The route validates the body with a type predicate and calls createNotes directly. No queue: the caller is synchronous and wants the result.

Trust boundary

Everything left of the core in the graph above is untrusted input. Three rules follow.

Body ordering is load-bearing. express.raw must be mounted on /v1/webhooks before the global express.json() in app.ts. Express runs middleware in registration order, so the webhook router terminates the request before the JSON parser is ever reached. Reversing these two lines silently breaks every signature check, because the raw bytes are gone by the time verification runs.
No unsigned write reaches the database. A webhook without a valid signature, and a REST call without a valid key, are both rejected before any DAO call.
Secrets are encrypted at rest. Access and refresh tokens are stored as AES-256-GCM ciphertext keyed by TOKEN_ENCRYPTION_KEY. API keys are never stored in recoverable form — only a SHA-256 hash, compared with crypto.timingSafeEqual.

Data model

Two new tables alongside the existing notes.

integrations — one row per installed connection. Holds both credentials for a connection: api_key_hash is the inbound credential a client presents to us, access_token_ciphertext and refresh_token_ciphertext are the outbound credentials we present to the provider. Unique on (provider, external_workspace_id).

ingest_events — one row per delivery. Unique on (provider, external_id), which is what makes at-least-once delivery safe. Also carries status, attempts, and last_error, so the queue's in-memory state has a durable shadow and a restart can find work that was interrupted mid-flight.

Technical risks

Queue durability. lib/queue is in-process. A crash between the 200 ack and the insert loses that delivery; the ingest_events row is the only evidence. Acceptable for a proof of concept, and the row makes recovery possible, but this is the first thing to replace with a real broker before production.
Single-tenant assumptions. The existing notes table has no workspace or board column. Multiple installed integrations all write into one flat board. Adding tenancy later means a migration on notes, not just the new tables.
Key rotation. TOKEN_ENCRYPTION_KEY has no versioning in this design. Rotating it invalidates every stored token and forces reinstalls.
Clock skew on replay windows. Signature timestamp tolerance is a fixed window; a badly skewed provider clock will fail verification in a way that looks like an attack.
Cohesion score sensitivity. Duplicate notes from redelivered webhooks do more than clutter the board — they distort the silhouette calculation in backend/services/validation.service.ts, because near-identical vectors compress intra-cluster distance. Deduplication is a correctness requirement, not a tidiness one.

Outbound (export) direction

Not built in this phase, but the shape is fixed now so the inbound work does not foreclose it: lib/httpClient provides retry and 429 backoff, services/oauth.service provides a valid token, and each future export adapter maps ranked clusters to one provider's write API. The registry is the single place an adapter registers itself.

6.1 KiB Raw Blame History