94 lines
6.1 KiB
Markdown
94 lines
6.1 KiB
Markdown
# Architecture: third-party integration infrastructure
|
|
|
|
Phase 1 output. Defines the system that lets external tools push artifacts into kongruity and read ranked clusters back out. Scope is the shared infrastructure only — no individual vendor integration is built here.
|
|
|
|
## Constraints inherited from the existing system
|
|
|
|
- Express 4 on Node, ESM, strict TypeScript (`backend/tsconfig.json`), compiled to `dist/`.
|
|
- PostgreSQL through a single `pg.Pool` in [backend/db/index.ts](backend/db/index.ts).
|
|
- No validation libraries permitted ([agents.md](agents.md) section 4) — runtime narrowing uses hand-written type predicates.
|
|
- Today the API is read-only and unauthenticated: [backend/app.ts](backend/app.ts) mounts one router exposing `GET /v1/notes` and `POST /v1/notes/cluster`.
|
|
- The `notes` table already carries a `source_meta JSONB` column, which becomes the provenance record for ingested artifacts.
|
|
|
|
## Component graph
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph external [External systems]
|
|
Provider["Provider (Slack, Jira, ...)"]
|
|
Client["REST / Zapier client"]
|
|
end
|
|
|
|
subgraph edge [Edge: trust boundary]
|
|
RawBody["express.raw on /v1/webhooks"]
|
|
Signatures["lib/signatures"]
|
|
ApiKey["middleware/apiKey"]
|
|
end
|
|
|
|
subgraph core [Core]
|
|
WebhookRoutes["routes/webhooks.routes"]
|
|
IngestRoutes["routes/ingest.routes"]
|
|
Registry["config/providers"]
|
|
Queue["lib/queue"]
|
|
Normalize["services/normalize.service"]
|
|
OAuth["services/oauth.service"]
|
|
HttpClient["lib/httpClient"]
|
|
end
|
|
|
|
subgraph data [Data]
|
|
IngestEvents["ingest_events"]
|
|
Integrations["integrations"]
|
|
Notes["notes"]
|
|
end
|
|
|
|
Provider --> RawBody --> Signatures --> WebhookRoutes
|
|
Client --> ApiKey --> IngestRoutes
|
|
WebhookRoutes --> IngestEvents
|
|
WebhookRoutes --> Queue
|
|
Queue --> Normalize
|
|
Normalize --> Registry
|
|
Normalize --> Notes
|
|
IngestRoutes --> Notes
|
|
Signatures --> Registry
|
|
OAuth --> Integrations
|
|
ApiKey --> Integrations
|
|
OAuth --> HttpClient
|
|
Notes --> Clustering["services/clustering.service"]
|
|
```
|
|
|
|
## Inbound request flow
|
|
|
|
Two doors, one destination.
|
|
|
|
**Webhook path.** A provider POSTs to `/v1/webhooks/:provider`. The raw body parser runs first so the exact bytes survive for HMAC comparison. `lib/signatures` looks the provider up in the registry and verifies the signature plus a timestamp replay window. Challenge and handshake requests short-circuit with the provider's expected response. Everything else records an `ingest_events` row keyed on `(provider, external_id)`, returns 200 immediately, and enqueues the payload. The queue worker normalizes it into `NoteInput[]` and writes through the existing `createNotes` DAO.
|
|
|
|
**REST path.** A client POSTs to `/v1/notes` with a bearer key. `middleware/apiKey` hashes the presented key and compares it against `integrations.api_key_hash` in constant time, attaching the resolved integration to the request. The route validates the body with a type predicate and calls `createNotes` directly. No queue: the caller is synchronous and wants the result.
|
|
|
|
## Trust boundary
|
|
|
|
Everything left of the core in the graph above is untrusted input. Three rules follow.
|
|
|
|
1. **Body ordering is load-bearing.** `express.raw` must be mounted on `/v1/webhooks` *before* the global `express.json()` in `app.ts`. Express runs middleware in registration order, so the webhook router terminates the request before the JSON parser is ever reached. Reversing these two lines silently breaks every signature check, because the raw bytes are gone by the time verification runs.
|
|
2. **No unsigned write reaches the database.** A webhook without a valid signature, and a REST call without a valid key, are both rejected before any DAO call.
|
|
3. **Secrets are encrypted at rest.** Access and refresh tokens are stored as AES-256-GCM ciphertext keyed by `TOKEN_ENCRYPTION_KEY`. API keys are never stored in recoverable form — only a SHA-256 hash, compared with `crypto.timingSafeEqual`.
|
|
|
|
## Data model
|
|
|
|
Two new tables alongside the existing `notes`.
|
|
|
|
**`integrations`** — one row per installed connection. Holds both credentials for a connection: `api_key_hash` is the inbound credential a client presents to us, `access_token_ciphertext` and `refresh_token_ciphertext` are the outbound credentials we present to the provider. Unique on `(provider, external_workspace_id)`.
|
|
|
|
**`ingest_events`** — one row per delivery. Unique on `(provider, external_id)`, which is what makes at-least-once delivery safe. Also carries `status`, `attempts`, and `last_error`, so the queue's in-memory state has a durable shadow and a restart can find work that was interrupted mid-flight.
|
|
|
|
## Technical risks
|
|
|
|
- **Queue durability.** `lib/queue` is in-process. A crash between the 200 ack and the insert loses that delivery; the `ingest_events` row is the only evidence. Acceptable for a proof of concept, and the row makes recovery possible, but this is the first thing to replace with a real broker before production.
|
|
- **Single-tenant assumptions.** The existing `notes` table has no workspace or board column. Multiple installed integrations all write into one flat board. Adding tenancy later means a migration on `notes`, not just the new tables.
|
|
- **Key rotation.** `TOKEN_ENCRYPTION_KEY` has no versioning in this design. Rotating it invalidates every stored token and forces reinstalls.
|
|
- **Clock skew on replay windows.** Signature timestamp tolerance is a fixed window; a badly skewed provider clock will fail verification in a way that looks like an attack.
|
|
- **Cohesion score sensitivity.** Duplicate notes from redelivered webhooks do more than clutter the board — they distort the silhouette calculation in [backend/services/validation.service.ts](backend/services/validation.service.ts), because near-identical vectors compress intra-cluster distance. Deduplication is a correctness requirement, not a tidiness one.
|
|
|
|
## Outbound (export) direction
|
|
|
|
Not built in this phase, but the shape is fixed now so the inbound work does not foreclose it: `lib/httpClient` provides retry and 429 backoff, `services/oauth.service` provides a valid token, and each future export adapter maps ranked clusters to one provider's write API. The registry is the single place an adapter registers itself.
|