kongruity/ROADMAP.md

Roadmap: integration infrastructure

Phase 2 output. Deconstructs ARCHITECTURE.md into ordered milestones with explicit contracts. Every path is relative to backend/.

Milestone dependency order

flowchart LR
  M0["M0 TypeScript migration (done)"] --> M1["M1 Write path"]
  M1 --> M2["M2 Inbound"]
  M1 --> M3["M3 Identity"]
  M2 --> M4["M4 Transform and deliver"]
  M3 --> M4
  M4 --> M5["M5 Verification"]

M2 and M3 both depend only on M1 and can run in parallel.

Data models

integrations

Column	Type	Notes
id	SERIAL PRIMARY KEY
provider	VARCHAR(32) NOT NULL	registry slug, e.g. slack, rest
external_workspace_id	VARCHAR(128)	provider's own workspace identifier
display_name	VARCHAR(255)	for operator legibility
api_key_hash	CHAR(64)	SHA-256 of the inbound key; null unless the integration accepts pushes
access_token_ciphertext	TEXT	AES-256-GCM, iv:tag:payload base64 triple
refresh_token_ciphertext	TEXT	same encoding
token_expires_at	TIMESTAMPTZ	drives refresh-before-use
scopes	TEXT[]	granted at install
signing_secret_ciphertext	TEXT	per-install webhook secret where the provider issues one
created_at / updated_at	TIMESTAMPTZ DEFAULT NOW()

Unique on (provider, external_workspace_id). Index on api_key_hash.

ingest_events

Column	Type	Notes
id	SERIAL PRIMARY KEY
provider	VARCHAR(32) NOT NULL
external_id	VARCHAR(255) NOT NULL	provider's event or message identifier
integration_id	INTEGER REFERENCES integrations(id)	nullable for unmatched deliveries
status	VARCHAR(16) NOT NULL DEFAULT 'pending'	pending, processing, done, failed
attempts	INTEGER NOT NULL DEFAULT 0
last_error	TEXT
received_at / updated_at	TIMESTAMPTZ DEFAULT NOW()

Unique on (provider, external_id). This constraint is the deduplication mechanism: a redelivery hits the conflict and is dropped.

M1 — Write path

Ticket 1.1 — db/migrate.ts. Add both tables above to the existing up script, CREATE TABLE IF NOT EXISTS in keeping with current style. Verify with npm run db:migrate against a live database.

Ticket 1.2 — middleware/apiKey.ts. Express middleware reading Authorization: Bearer <key>, hashing with SHA-256, looking up integrations.api_key_hash, comparing with crypto.timingSafeEqual, attaching the row to req. Responds 401 with { error: string } on any failure and never distinguishes "no such key" from "wrong key" in the response.

Ticket 1.3 — routes/ingest.routes.ts. POST /v1/notes accepting { notes: NoteInput[] }. Body narrowed by a type predicate, not a schema library. Returns 201 with { inserted: number, notes: Note[] }, 400 on malformed body, 401 from the middleware. Delegates to the existing createNotes in backend/db/notes.dao.ts, which already batches under the Postgres bind-parameter ceiling.

Ticket 1.4 — app.ts wiring. Mount the ingest router. Ordering does not matter yet; it becomes critical in M2.

M2 — Inbound

Ticket 2.1 — db/ingest_events.dao.ts. recordDelivery performing INSERT ... ON CONFLICT (provider, external_id) DO NOTHING RETURNING id, returning null when the row already existed so the caller can drop the redelivery. Plus markProcessing, markDone, markFailed.

Ticket 2.2 — lib/signatures.ts. verifySignature(provider, rawBody, headers, secret) returning a discriminated result rather than throwing. HMAC comparison via crypto.timingSafeEqual on equal-length buffers. Enforces a replay window, default 300 seconds.

Ticket 2.3 — routes/webhooks.routes.ts. POST /v1/webhooks/:provider. Resolves the provider from the registry (404 on unknown), answers challenge requests, verifies the signature (401 on failure), records the delivery (200 and stop on duplicate), enqueues, then returns 200. Must respond within the tightest provider budget, which is Slack's three seconds.

Ticket 2.4 — app.ts ordering. Mount express.raw({ type: '*/*' }) with the webhook router before app.use(express.json()). This is the single most breakable line in the system and needs a regression test of its own.

M3 — Identity

Ticket 3.1 — lib/crypto.ts. encryptSecret and decryptSecret over AES-256-GCM using TOKEN_ENCRYPTION_KEY, node:crypto only. Fails loudly at startup if the key is absent or not 32 bytes.

Ticket 3.2 — db/integrations.dao.ts. CRUD returning decrypted secrets only through explicit accessors, so a careless SELECT * cannot leak plaintext. findByApiKeyHash, findByProviderWorkspace, upsertInstall, updateTokens.

Ticket 3.3 — services/oauth.service.ts. buildAuthorizeUrl, exchangeCode, getValidAccessToken refreshing when token_expires_at is inside a 60-second margin. Provider-specific endpoints come from the registry, not from this file.

Ticket 3.4 — config/providers.ts. The registry. Each entry declares slug, signature scheme, header names, normalizer, OAuth endpoints, and scopes. Adding a provider must touch only this file plus one normalizer.

M4 — Transform and deliver

Ticket 4.1 — services/normalize.service.ts. normalize(provider, payload): NormalizedDelivery returning { externalId: string, notes: NoteInput[] }. Writes provenance into source_meta: provider, external id, permalink, author handle, received timestamp.

Ticket 4.2 — lib/queue.ts. In-process FIFO with concurrency 1, exponential backoff, and a cap on attempts. On terminal failure calls markFailed. Exposes enqueue, size, and a drain promise for deterministic testing.

Ticket 4.3 — lib/httpClient.ts. requestJson with timeout, retry on 429 and 5xx, Retry-After respected, jittered exponential backoff. Used by OAuth exchange today and export adapters later.

M5 — Verification

End-to-end test driving a signed synthetic-provider delivery through the whole chain and asserting the note lands. Explicit cases: valid delivery inserts, replayed delivery is dropped, bad signature is rejected, unknown provider 404s, malformed REST body 400s, missing key 401s, and the raw-body ordering regression.

Definition of done per milestone

npm run type-check, npm run build, and npm test all clean, with new unit tests for every module and no change in behavior to the four pre-existing suites.