5 Commits

Author SHA1 Message Date
a797966f3a Update README.md 2026-09-18 02:20:36 +00:00
31d6729fe6 Update README.md 2026-08-24 00:09:17 +00:00
KS Jannette
1dd17f76f6 hot 2026-08-23 18:19:04 -04:00
df128c3683 Merge pull request 'improve image' (#9) from FEAT-better-demo-image into master
Reviewed-on: #9
2026-08-23 22:04:05 +00:00
KS Jannette
e362f67f46 improve image 2026-08-23 18:03:29 -04:00
3 changed files with 14 additions and 113 deletions

View File

@@ -1,16 +1,13 @@
# kongruity: Signal from noise - 2025-2026 @sjDev - LICENSE: MIT # kongruity: Signal from noise - 2025-2026 @sjDev - LICENSE: MIT
![kongruity demo image.](kongruity_demo_image.png) ![kongruity demo image.](kongruity_with_frame.jpg)
kongruity pulls in unstructured artifacts of the creative-engineering process, capturing "AHA!" moments scattered across an engineering team's disparate tools: action items or backlog in Atlassian/Jira, user story comments, Slack discussions, retrospective feedback. kongruity pulls in unstructured artifacts of the creative-engineering process, capturing "AHA!" moments scattered across an engineering team's disparate tools: action items or backlog in Atlassian/Jira, user story comments, Slack discussions, retrospective feedback.
It stores and synthesizes these into semantically coherent, prioritized clusters that can be incorporated into implementation planning. It stores and synthesizes these into semantically coherent, prioritized clusters that can be incorporated into implementation planning.
In kongruity, artifacts become "sticky notes." A board full looks chaotic. With a click, a RAG-pipeline levearges Models trained to semantically evaluate, group, cluster and add descriptive cluster headers. In kongruity, artifacts become "sticky notes." A board full looks chaotic. With a click, a RAG-pipeline levearges Models trained to semantically evaluate, group, cluster and add descriptive cluster headers.
These are rankable, editable and exportable to sprint project planning tools. These are rankable, editable and exportable to sprint project planning tools.
## Clustering and evaluation: methodology ## Clustering and evaluation: methodology
@@ -36,7 +33,6 @@ Note that: before scoring, structural validation confirms that each note landed
Average silhouette width is a widely-used measure of clustering quality. Higher values indicate: Average silhouette width is a widely-used measure of clustering quality. Higher values indicate:
1. The qualitative semantic cohesiveness of clusters, and: 1. The qualitative semantic cohesiveness of clusters, and:
2. How well-separated each cluster is from its nearest neighboring cluster. 2. How well-separated each cluster is from its nearest neighboring cluster.
Although the coefficient is mathematically bounded by [−1, 1], cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. Because silhouette divides the gap between them by the larger of the two, the practical range on embedding data is roughly [−0.05, 0.10] rather than the full interval. Although the coefficient is mathematically bounded by [−1, 1], cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. Because silhouette divides the gap between them by the larger of the two, the practical range on embedding data is roughly [−0.05, 0.10] rather than the full interval.
@@ -74,11 +70,13 @@ Developers may swap in other LLM SDKs/APIs and alter prompt syntax in `backend/s
The shared infrastructure that third-party integrations are built on. See [ARCHITECTURE.md](ARCHITECTURE.md) for the design, [ROADMAP.md](ROADMAP.md) for milestones, and [LOW_LEVEL_DESIGN.md](LOW_LEVEL_DESIGN.md) for module contracts. The shared infrastructure that third-party integrations are built on. See [ARCHITECTURE.md](ARCHITECTURE.md) for the design, [ROADMAP.md](ROADMAP.md) for milestones, and [LOW_LEVEL_DESIGN.md](LOW_LEVEL_DESIGN.md) for module contracts.
| Endpoint | Auth | Purpose |
| --- | --- | --- | | Endpoint | Auth | Purpose |
| `POST /v1/notes` | `Authorization: Bearer <api key>` | Bulk push notes. Body `{ notes: [{ id, text, author, x?, y?, color? }] }`, max 5000 per request. | | ----------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `POST /v1/webhooks/:provider` | Per-provider request signature | Inbound provider deliveries. Acks immediately, inserts in the background. | | `POST /v1/notes` | `Authorization: Bearer <api key>` | Bulk push notes. Body `{ notes: [{ id, text, author, x?, y?, color? }] }`, max 5000 per request. |
| `POST /v1/slack/commands` | Slack `v0` request signature | The `/sticky` slash command. Form-encoded rather than JSON, and its response body is shown to the user in Slack. | | `POST /v1/webhooks/:provider` | Per-provider request signature | Inbound provider deliveries. Acks immediately, inserts in the background. |
| `POST /v1/slack/commands` | Slack `v0` request signature | The `/sticky` slash command. Form-encoded rather than JSON, and its response body is shown to the user in Slack. |
Registered providers live in `backend/config/providers.ts`. `rest`, `linear`, and `slack` carry payload mappings today; `github` and `jira` declare transport and OAuth endpoints only, so adding one of those integrations means writing a single normalizer rather than new plumbing. Registered providers live in `backend/config/providers.ts`. `rest`, `linear`, and `slack` carry payload mappings today; `github` and `jira` declare transport and OAuth endpoints only, so adding one of those integrations means writing a single normalizer rather than new plumbing.
@@ -98,7 +96,7 @@ Deliveries are deduplicated on `(provider, external_id)` in the `ingest_events`
### Ingestion — pulling tagged artifacts in ### Ingestion — pulling tagged artifacts in
- [x] **Slack** — where decisions actually get made; `/sticky` captures a new note, and an emoji reaction (`:pushpin:` by default) fires an Events API webhook that pulls an existing message in. - [x] **Slack** — where decisions are made; `/sticky` captures a new note, and an emoji reaction (`:pushpin:` by default) fires an Events API webhook that pulls an existing message in.
- [ ] **Microsoft Teams** — same capture gesture for enterprise shops; message extension plus Graph change notifications. - [ ] **Microsoft Teams** — same capture gesture for enterprise shops; message extension plus Graph change notifications.
- [ ] **Jira** — label- or mention-triggered webhook scoped by JQL. (This is where comments typically carry half the backlog's context.) - [ ] **Jira** — label- or mention-triggered webhook scoped by JQL. (This is where comments typically carry half the backlog's context.)
- [ ] **Linear** — engineering-side tickets and threads; label-triggered GraphQL webhook. - [ ] **Linear** — engineering-side tickets and threads; label-triggered GraphQL webhook.
@@ -168,7 +166,7 @@ The key must decode to exactly 32 bytes; the backend refuses to encrypt otherwis
Two capture gestures, both landing in the same ingest pipeline: Two capture gestures, both landing in the same ingest pipeline:
- **`/sticky <your note>`** creates a note from what you type. - `**/sticky <your note>**` creates a note from what you type.
- **Reacting with an emoji** captures the message someone already wrote. The trigger defaults to `:pushpin:` and is set by `SLACK_CAPTURE_REACTION`. - **Reacting with an emoji** captures the message someone already wrote. The trigger defaults to `:pushpin:` and is set by `SLACK_CAPTURE_REACTION`.
### 1. Create the Slack app ### 1. Create the Slack app
@@ -182,7 +180,7 @@ SLACK_CAPTURE_REACTION=pushpin
EOF EOF
``` ```
`SLACK_SIGNING_SECRET` is already in the integration variables above. There is no OAuth flow here on purpose: a single workspace reading credentials from the environment is far less machinery than an install dance, and the OAuth endpoints in the provider registry stay unused until multi-workspace support is actually needed. `SLACK_SIGNING_SECRET` is already in the integration variables above. There is no OAuth flow here on purpose: a single workspace reading credentials from the environment is a simple, and the OAuth endpoints in the provider registry stay unused until multi-workspace support is actually needed.
### 2. Expose backend ### 2. Expose backend
@@ -253,7 +251,7 @@ npm run build
npm run start npm run start
``` ```
The API server starts on **http://localhost:3001** (configurable via the `PORT` environment variable). The API server starts on **[http://localhost:3001](http://localhost:3001)** (configurable via the `PORT` environment variable).
### Start the backend — Development mode ### Start the backend — Development mode
@@ -285,7 +283,7 @@ From the `frontend/` directory:
npm run dev npm run dev
``` ```
The Vite dev server starts on **http://localhost:5173** by default. Open that URL in a browser. The Vite dev server starts on **[http://localhost:5173](http://localhost:5173)** by default. Open that URL in a browser.
## Running tests ## Running tests
@@ -312,3 +310,4 @@ This runs Vitest with jsdom. For watch mode during development:
```bash ```bash
npm run test:watch npm run test:watch
``` ```

View File

@@ -1,98 +0,0 @@
# Roadmap: integration infrastructure
Phase 2 output. Deconstructs [ARCHITECTURE.md](ARCHITECTURE.md) into ordered milestones with explicit contracts. Every path is relative to `backend/`.
## Milestone dependency order
```mermaid
flowchart LR
M0["M0 TypeScript migration (done)"] --> M1["M1 Write path"]
M1 --> M2["M2 Inbound"]
M1 --> M3["M3 Identity"]
M2 --> M4["M4 Transform and deliver"]
M3 --> M4
M4 --> M5["M5 Verification"]
```
M2 and M3 both depend only on M1 and can run in parallel.
## Data models
### `integrations`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | `SERIAL PRIMARY KEY` | |
| `provider` | `VARCHAR(32) NOT NULL` | registry slug, e.g. `slack`, `rest` |
| `external_workspace_id` | `VARCHAR(128)` | provider's own workspace identifier |
| `display_name` | `VARCHAR(255)` | for operator legibility |
| `api_key_hash` | `CHAR(64)` | SHA-256 of the inbound key; null unless the integration accepts pushes |
| `access_token_ciphertext` | `TEXT` | AES-256-GCM, `iv:tag:payload` base64 triple |
| `refresh_token_ciphertext` | `TEXT` | same encoding |
| `token_expires_at` | `TIMESTAMPTZ` | drives refresh-before-use |
| `scopes` | `TEXT[]` | granted at install |
| `signing_secret_ciphertext` | `TEXT` | per-install webhook secret where the provider issues one |
| `created_at` / `updated_at` | `TIMESTAMPTZ DEFAULT NOW()` | |
Unique on `(provider, external_workspace_id)`. Index on `api_key_hash`.
### `ingest_events`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | `SERIAL PRIMARY KEY` | |
| `provider` | `VARCHAR(32) NOT NULL` | |
| `external_id` | `VARCHAR(255) NOT NULL` | provider's event or message identifier |
| `integration_id` | `INTEGER REFERENCES integrations(id)` | nullable for unmatched deliveries |
| `status` | `VARCHAR(16) NOT NULL DEFAULT 'pending'` | `pending`, `processing`, `done`, `failed` |
| `attempts` | `INTEGER NOT NULL DEFAULT 0` | |
| `last_error` | `TEXT` | |
| `received_at` / `updated_at` | `TIMESTAMPTZ DEFAULT NOW()` | |
Unique on `(provider, external_id)`. This constraint is the deduplication mechanism: a redelivery hits the conflict and is dropped.
## M1 — Write path
**Ticket 1.1 — `db/migrate.ts`.** Add both tables above to the existing `up` script, `CREATE TABLE IF NOT EXISTS` in keeping with current style. Verify with `npm run db:migrate` against a live database.
**Ticket 1.2 — `middleware/apiKey.ts`.** Express middleware reading `Authorization: Bearer <key>`, hashing with SHA-256, looking up `integrations.api_key_hash`, comparing with `crypto.timingSafeEqual`, attaching the row to `req`. Responds 401 with `{ error: string }` on any failure and never distinguishes "no such key" from "wrong key" in the response.
**Ticket 1.3 — `routes/ingest.routes.ts`.** `POST /v1/notes` accepting `{ notes: NoteInput[] }`. Body narrowed by a type predicate, not a schema library. Returns `201` with `{ inserted: number, notes: Note[] }`, `400` on malformed body, `401` from the middleware. Delegates to the existing `createNotes` in [backend/db/notes.dao.ts](backend/db/notes.dao.ts), which already batches under the Postgres bind-parameter ceiling.
**Ticket 1.4 — `app.ts` wiring.** Mount the ingest router. Ordering does not matter yet; it becomes critical in M2.
## M2 — Inbound
**Ticket 2.1 — `db/ingest_events.dao.ts`.** `recordDelivery` performing `INSERT ... ON CONFLICT (provider, external_id) DO NOTHING RETURNING id`, returning null when the row already existed so the caller can drop the redelivery. Plus `markProcessing`, `markDone`, `markFailed`.
**Ticket 2.2 — `lib/signatures.ts`.** `verifySignature(provider, rawBody, headers, secret)` returning a discriminated result rather than throwing. HMAC comparison via `crypto.timingSafeEqual` on equal-length buffers. Enforces a replay window, default 300 seconds.
**Ticket 2.3 — `routes/webhooks.routes.ts`.** `POST /v1/webhooks/:provider`. Resolves the provider from the registry (404 on unknown), answers challenge requests, verifies the signature (401 on failure), records the delivery (200 and stop on duplicate), enqueues, then returns 200. Must respond within the tightest provider budget, which is Slack's three seconds.
**Ticket 2.4 — `app.ts` ordering.** Mount `express.raw({ type: '*/*' })` with the webhook router *before* `app.use(express.json())`. This is the single most breakable line in the system and needs a regression test of its own.
## M3 — Identity
**Ticket 3.1 — `lib/crypto.ts`.** `encryptSecret` and `decryptSecret` over AES-256-GCM using `TOKEN_ENCRYPTION_KEY`, node:crypto only. Fails loudly at startup if the key is absent or not 32 bytes.
**Ticket 3.2 — `db/integrations.dao.ts`.** CRUD returning decrypted secrets only through explicit accessors, so a careless `SELECT *` cannot leak plaintext. `findByApiKeyHash`, `findByProviderWorkspace`, `upsertInstall`, `updateTokens`.
**Ticket 3.3 — `services/oauth.service.ts`.** `buildAuthorizeUrl`, `exchangeCode`, `getValidAccessToken` refreshing when `token_expires_at` is inside a 60-second margin. Provider-specific endpoints come from the registry, not from this file.
**Ticket 3.4 — `config/providers.ts`.** The registry. Each entry declares slug, signature scheme, header names, normalizer, OAuth endpoints, and scopes. Adding a provider must touch only this file plus one normalizer.
## M4 — Transform and deliver
**Ticket 4.1 — `services/normalize.service.ts`.** `normalize(provider, payload): NormalizedDelivery` returning `{ externalId: string, notes: NoteInput[] }`. Writes provenance into `source_meta`: provider, external id, permalink, author handle, received timestamp.
**Ticket 4.2 — `lib/queue.ts`.** In-process FIFO with concurrency 1, exponential backoff, and a cap on attempts. On terminal failure calls `markFailed`. Exposes `enqueue`, `size`, and a `drain` promise for deterministic testing.
**Ticket 4.3 — `lib/httpClient.ts`.** `requestJson` with timeout, retry on 429 and 5xx, `Retry-After` respected, jittered exponential backoff. Used by OAuth exchange today and export adapters later.
## M5 — Verification
End-to-end test driving a signed synthetic-provider delivery through the whole chain and asserting the note lands. Explicit cases: valid delivery inserts, replayed delivery is dropped, bad signature is rejected, unknown provider 404s, malformed REST body 400s, missing key 401s, and the raw-body ordering regression.
## Definition of done per milestone
`npm run type-check`, `npm run build`, and `npm test` all clean, with new unit tests for every module and no change in behavior to the four pre-existing suites.

BIN
kongruity_with_frame.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB