1 Commits

Author SHA1 Message Date
KS Jannette
e362f67f46 improve image 2026-08-23 18:03:29 -04:00
3 changed files with 1 additions and 99 deletions

View File

@@ -1,6 +1,6 @@
# kongruity: Signal from noise - 2025-2026 @sjDev - LICENSE: MIT
![kongruity demo image.](kongruity_demo_image.png)
![kongruity demo image.](kongruity_with_frame.jpg)
kongruity pulls in unstructured artifacts of the creative-engineering process, capturing "AHA!" moments scattered across an engineering team's disparate tools: action items or backlog in Atlassian/Jira, user story comments, Slack discussions, retrospective feedback.

View File

@@ -1,98 +0,0 @@
# Roadmap: integration infrastructure
Phase 2 output. Deconstructs [ARCHITECTURE.md](ARCHITECTURE.md) into ordered milestones with explicit contracts. Every path is relative to `backend/`.
## Milestone dependency order
```mermaid
flowchart LR
M0["M0 TypeScript migration (done)"] --> M1["M1 Write path"]
M1 --> M2["M2 Inbound"]
M1 --> M3["M3 Identity"]
M2 --> M4["M4 Transform and deliver"]
M3 --> M4
M4 --> M5["M5 Verification"]
```
M2 and M3 both depend only on M1 and can run in parallel.
## Data models
### `integrations`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | `SERIAL PRIMARY KEY` | |
| `provider` | `VARCHAR(32) NOT NULL` | registry slug, e.g. `slack`, `rest` |
| `external_workspace_id` | `VARCHAR(128)` | provider's own workspace identifier |
| `display_name` | `VARCHAR(255)` | for operator legibility |
| `api_key_hash` | `CHAR(64)` | SHA-256 of the inbound key; null unless the integration accepts pushes |
| `access_token_ciphertext` | `TEXT` | AES-256-GCM, `iv:tag:payload` base64 triple |
| `refresh_token_ciphertext` | `TEXT` | same encoding |
| `token_expires_at` | `TIMESTAMPTZ` | drives refresh-before-use |
| `scopes` | `TEXT[]` | granted at install |
| `signing_secret_ciphertext` | `TEXT` | per-install webhook secret where the provider issues one |
| `created_at` / `updated_at` | `TIMESTAMPTZ DEFAULT NOW()` | |
Unique on `(provider, external_workspace_id)`. Index on `api_key_hash`.
### `ingest_events`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | `SERIAL PRIMARY KEY` | |
| `provider` | `VARCHAR(32) NOT NULL` | |
| `external_id` | `VARCHAR(255) NOT NULL` | provider's event or message identifier |
| `integration_id` | `INTEGER REFERENCES integrations(id)` | nullable for unmatched deliveries |
| `status` | `VARCHAR(16) NOT NULL DEFAULT 'pending'` | `pending`, `processing`, `done`, `failed` |
| `attempts` | `INTEGER NOT NULL DEFAULT 0` | |
| `last_error` | `TEXT` | |
| `received_at` / `updated_at` | `TIMESTAMPTZ DEFAULT NOW()` | |
Unique on `(provider, external_id)`. This constraint is the deduplication mechanism: a redelivery hits the conflict and is dropped.
## M1 — Write path
**Ticket 1.1 — `db/migrate.ts`.** Add both tables above to the existing `up` script, `CREATE TABLE IF NOT EXISTS` in keeping with current style. Verify with `npm run db:migrate` against a live database.
**Ticket 1.2 — `middleware/apiKey.ts`.** Express middleware reading `Authorization: Bearer <key>`, hashing with SHA-256, looking up `integrations.api_key_hash`, comparing with `crypto.timingSafeEqual`, attaching the row to `req`. Responds 401 with `{ error: string }` on any failure and never distinguishes "no such key" from "wrong key" in the response.
**Ticket 1.3 — `routes/ingest.routes.ts`.** `POST /v1/notes` accepting `{ notes: NoteInput[] }`. Body narrowed by a type predicate, not a schema library. Returns `201` with `{ inserted: number, notes: Note[] }`, `400` on malformed body, `401` from the middleware. Delegates to the existing `createNotes` in [backend/db/notes.dao.ts](backend/db/notes.dao.ts), which already batches under the Postgres bind-parameter ceiling.
**Ticket 1.4 — `app.ts` wiring.** Mount the ingest router. Ordering does not matter yet; it becomes critical in M2.
## M2 — Inbound
**Ticket 2.1 — `db/ingest_events.dao.ts`.** `recordDelivery` performing `INSERT ... ON CONFLICT (provider, external_id) DO NOTHING RETURNING id`, returning null when the row already existed so the caller can drop the redelivery. Plus `markProcessing`, `markDone`, `markFailed`.
**Ticket 2.2 — `lib/signatures.ts`.** `verifySignature(provider, rawBody, headers, secret)` returning a discriminated result rather than throwing. HMAC comparison via `crypto.timingSafeEqual` on equal-length buffers. Enforces a replay window, default 300 seconds.
**Ticket 2.3 — `routes/webhooks.routes.ts`.** `POST /v1/webhooks/:provider`. Resolves the provider from the registry (404 on unknown), answers challenge requests, verifies the signature (401 on failure), records the delivery (200 and stop on duplicate), enqueues, then returns 200. Must respond within the tightest provider budget, which is Slack's three seconds.
**Ticket 2.4 — `app.ts` ordering.** Mount `express.raw({ type: '*/*' })` with the webhook router *before* `app.use(express.json())`. This is the single most breakable line in the system and needs a regression test of its own.
## M3 — Identity
**Ticket 3.1 — `lib/crypto.ts`.** `encryptSecret` and `decryptSecret` over AES-256-GCM using `TOKEN_ENCRYPTION_KEY`, node:crypto only. Fails loudly at startup if the key is absent or not 32 bytes.
**Ticket 3.2 — `db/integrations.dao.ts`.** CRUD returning decrypted secrets only through explicit accessors, so a careless `SELECT *` cannot leak plaintext. `findByApiKeyHash`, `findByProviderWorkspace`, `upsertInstall`, `updateTokens`.
**Ticket 3.3 — `services/oauth.service.ts`.** `buildAuthorizeUrl`, `exchangeCode`, `getValidAccessToken` refreshing when `token_expires_at` is inside a 60-second margin. Provider-specific endpoints come from the registry, not from this file.
**Ticket 3.4 — `config/providers.ts`.** The registry. Each entry declares slug, signature scheme, header names, normalizer, OAuth endpoints, and scopes. Adding a provider must touch only this file plus one normalizer.
## M4 — Transform and deliver
**Ticket 4.1 — `services/normalize.service.ts`.** `normalize(provider, payload): NormalizedDelivery` returning `{ externalId: string, notes: NoteInput[] }`. Writes provenance into `source_meta`: provider, external id, permalink, author handle, received timestamp.
**Ticket 4.2 — `lib/queue.ts`.** In-process FIFO with concurrency 1, exponential backoff, and a cap on attempts. On terminal failure calls `markFailed`. Exposes `enqueue`, `size`, and a `drain` promise for deterministic testing.
**Ticket 4.3 — `lib/httpClient.ts`.** `requestJson` with timeout, retry on 429 and 5xx, `Retry-After` respected, jittered exponential backoff. Used by OAuth exchange today and export adapters later.
## M5 — Verification
End-to-end test driving a signed synthetic-provider delivery through the whole chain and asserting the note lands. Explicit cases: valid delivery inserts, replayed delivery is dropped, bad signature is rejected, unknown provider 404s, malformed REST body 400s, missing key 401s, and the raw-body ordering regression.
## Definition of done per milestone
`npm run type-check`, `npm run build`, and `npm test` all clean, with new unit tests for every module and no change in behavior to the four pre-existing suites.

BIN
kongruity_with_frame.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB