59 Commits

Author SHA1 Message Date
dae256f6c6 Merge pull request 'Infrastructure build to support third-party app integrations' (#7) from FEAT-integrations-infrastructure into master
Reviewed-on: #7
2026-08-01 11:46:56 +00:00
KS Jannette
15af3465e2 Infrastructure build to support third-party app integrations 2026-08-01 07:35:01 -04:00
b4666c5439 Merge pull request 'add agents.md' (#6) from update-agents-infra into master
Reviewed-on: #6
2026-08-01 10:00:27 +00:00
KS Jannette
22667b38b8 add agents.md 2026-08-01 06:00:04 -04:00
e547dee979 Merge pull request 'Upgraded embedding model to voyage-3.5, updated README.md' (#5) from FEAT-update-voyage-model into master
Reviewed-on: #5
2026-08-01 09:35:43 +00:00
KS Jannette
362a47f88a Upgraded embedding model to voyage-3.5, updated README.md 2026-08-01 05:35:06 -04:00
274e846909 Merge pull request 'fix minor css issue on stickies' (#4) from minor-css-update into master
Reviewed-on: #4
2026-08-01 08:50:59 +00:00
KS Jannette
c6b07ccb56 fix minor css issue on stickies 2026-08-01 04:50:40 -04:00
KS Jannette
8e79e78006 hotfix 2026-08-01 04:41:31 -04:00
19d4a82c90 Merge pull request 'Re-aligned heuristic, updated readme, added code comment' (#3) from BUG-cohesion-scoreUI-display into master
Reviewed-on: #3
2026-08-01 08:13:29 +00:00
edeadd547a Update README.md 2026-08-01 08:11:10 +00:00
3b707a150b Update README.md 2026-08-01 08:08:34 +00:00
c2d40fa409 Update README.md 2026-08-01 08:08:19 +00:00
KS Jannette
6f09b6ecdc Re-aligned heuristic, updated readme, added code comment 2026-08-01 04:06:39 -04:00
ac75a30b61 Update README.md 2026-08-01 07:42:16 +00:00
f1f2a93e4a Merge pull request 'Updated readme and streams.js' (#2) from FEAT-update-README-with-roadmap into master
Reviewed-on: #2
2026-08-01 07:40:55 +00:00
KS Jannette
68fed1a56e Updated readme and streams.js 2026-08-01 03:39:22 -04:00
e11122ed14 Merge pull request 'FEAT-nonblocking-backend-io-imporvements' (#1) from FEAT-nonblocking-backend-io-imporvements into master
Reviewed-on: #1
2026-08-01 05:43:24 +00:00
KS Jannette
7b47e46852 Cleanup 2026-08-01 01:42:43 -04:00
KS Jannette
62477d009c Updated Claude model 2026-08-01 01:35:27 -04:00
KS Jannette
ee6fa9e576 Add nonblocking/asyn I/O operations 2026-08-01 01:06:19 -04:00
90035b3568 Update README.ms
hotfix
2026-07-31 13:45:35 +00:00
S Jannette
9636be78c9 Fix typos and enhance README content
Improved clarity in the README.
2026-03-06 03:52:47 -05:00
S Jannette
a50ca5c171 Add MIT License to the project 2026-02-25 18:50:59 -05:00
S Jannette
e271702bca Fix typo in README.md regarding implementation planning
Corrected 'incoporated in' to 'incoporated into' for clarity.
2026-02-24 21:11:24 -05:00
S Jannette
d3d76b1d0d Refine README.md content for clarity and accuracy
Updated descriptions for clarity and corrected typos.
2026-02-24 21:10:51 -05:00
S Jannette
8cae0665db Merge pull request #16 from kjannette/refinements
Refinements
2026-02-24 21:04:56 -05:00
KS Jannette
21453aab7d UI development 2026-02-24 21:02:43 -05:00
KS Jannette
691186e02c upgraded scoring algorithm model 2026-02-24 20:55:19 -05:00
S Jannette
fd48aaab4d Merge pull request #15 from kjannette/test-data-xfer
further DB infra buildout
2026-02-24 19:41:31 -05:00
KS Jannette
0b56d0d6c4 further DB infra buildout 2026-02-24 19:38:35 -05:00
S Jannette
0d22998c2e Merge pull request #14 from kjannette/db-build
db created, added tests -- tentative
2026-02-24 19:26:55 -05:00
KS Jannette
ce7bc2cde9 db created, added tests -- tentative 2026-02-24 19:25:42 -05:00
S Jannette
0ee426e1f3 Update README.md 2026-02-24 14:12:30 -05:00
KS Jannette
90a19e4525 edit 2026-02-24 13:51:19 -05:00
S Jannette
8b555bf729 Fix typo in README.md
Corrected a typo in the README regarding 'simply'.
2026-02-24 13:48:33 -05:00
KS Jannette
c8ee194bc5 hot edit readme 2026-02-24 13:47:24 -05:00
S Jannette
f77362e684 Merge pull request #13 from kjannette/workflow-UI
Add UI/logical temporal ordering feature
2026-02-24 13:32:29 -05:00
KS Jannette
9ba13206c5 Add UI/logical temporal ordering feature 2026-02-24 13:32:06 -05:00
KS Jannette
0c7baeb9d8 hotfix 2026-02-24 13:12:47 -05:00
S Jannette
07b0309adb Merge pull request #12 from kjannette/query-validation
Query validation
2026-02-24 13:07:39 -05:00
KS Jannette
7ec074a043 update documentation 2026-02-24 13:06:56 -05:00
KS Jannette
b7757ca406 add second layer of result validation 2026-02-24 12:56:18 -05:00
KS Jannette
d55f58985b more 2026-02-14 08:11:36 -05:00
KS Jannette
61835ae3aa format 2026-02-13 16:43:00 -05:00
KS Jannette
091315c188 m 2026-02-13 16:35:26 -05:00
KS Jannette
089b9e7e9d more 2026-02-13 16:31:39 -05:00
KS Jannette
3e7978c918 cccccLean 2026-02-13 15:41:50 -05:00
KS Jannette
317468b1ed more 2026-02-13 15:34:29 -05:00
KS Jannette
a720673896 clean 2026-02-13 15:27:10 -05:00
KS Jannette
1a85c6e1e0 fin 2026-02-13 15:21:02 -05:00
S Jannette
7f4c322280 Merge pull request #11 from kjannette/frontFact
more
2026-02-13 15:15:08 -05:00
KS Jannette
5208327b5d more 2026-02-13 15:12:18 -05:00
KS Jannette
ac76871d60 hottie 2026-02-13 13:47:43 -05:00
S Jannette
66c86c2625 Merge pull request #10 from kjannette/refact5
clean
2026-02-13 13:45:24 -05:00
KS Jannette
56d0c21b79 clean 2026-02-13 13:45:00 -05:00
S Jannette
37e81e773e Merge pull request #9 from kjannette/refactor4
more
2026-02-13 13:42:52 -05:00
KS Jannette
534da11217 more 2026-02-13 13:42:31 -05:00
S Jannette
2212755772 Merge pull request #8 from kjannette/refact3
cleanup
2026-02-13 13:35:29 -05:00
75 changed files with 8011 additions and 1240 deletions

93
ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,93 @@
# Architecture: third-party integration infrastructure
Phase 1 output. Defines the system that lets external tools push artifacts into kongruity and read ranked clusters back out. Scope is the shared infrastructure only — no individual vendor integration is built here.
## Constraints inherited from the existing system
- Express 4 on Node, ESM, strict TypeScript (`backend/tsconfig.json`), compiled to `dist/`.
- PostgreSQL through a single `pg.Pool` in [backend/db/index.ts](backend/db/index.ts).
- No validation libraries permitted ([agents.md](agents.md) section 4) — runtime narrowing uses hand-written type predicates.
- Today the API is read-only and unauthenticated: [backend/app.ts](backend/app.ts) mounts one router exposing `GET /v1/notes` and `POST /v1/notes/cluster`.
- The `notes` table already carries a `source_meta JSONB` column, which becomes the provenance record for ingested artifacts.
## Component graph
```mermaid
flowchart TD
subgraph external [External systems]
Provider["Provider (Slack, Jira, ...)"]
Client["REST / Zapier client"]
end
subgraph edge [Edge: trust boundary]
RawBody["express.raw on /v1/webhooks"]
Signatures["lib/signatures"]
ApiKey["middleware/apiKey"]
end
subgraph core [Core]
WebhookRoutes["routes/webhooks.routes"]
IngestRoutes["routes/ingest.routes"]
Registry["config/providers"]
Queue["lib/queue"]
Normalize["services/normalize.service"]
OAuth["services/oauth.service"]
HttpClient["lib/httpClient"]
end
subgraph data [Data]
IngestEvents["ingest_events"]
Integrations["integrations"]
Notes["notes"]
end
Provider --> RawBody --> Signatures --> WebhookRoutes
Client --> ApiKey --> IngestRoutes
WebhookRoutes --> IngestEvents
WebhookRoutes --> Queue
Queue --> Normalize
Normalize --> Registry
Normalize --> Notes
IngestRoutes --> Notes
Signatures --> Registry
OAuth --> Integrations
ApiKey --> Integrations
OAuth --> HttpClient
Notes --> Clustering["services/clustering.service"]
```
## Inbound request flow
Two doors, one destination.
**Webhook path.** A provider POSTs to `/v1/webhooks/:provider`. The raw body parser runs first so the exact bytes survive for HMAC comparison. `lib/signatures` looks the provider up in the registry and verifies the signature plus a timestamp replay window. Challenge and handshake requests short-circuit with the provider's expected response. Everything else records an `ingest_events` row keyed on `(provider, external_id)`, returns 200 immediately, and enqueues the payload. The queue worker normalizes it into `NoteInput[]` and writes through the existing `createNotes` DAO.
**REST path.** A client POSTs to `/v1/notes` with a bearer key. `middleware/apiKey` hashes the presented key and compares it against `integrations.api_key_hash` in constant time, attaching the resolved integration to the request. The route validates the body with a type predicate and calls `createNotes` directly. No queue: the caller is synchronous and wants the result.
## Trust boundary
Everything left of the core in the graph above is untrusted input. Three rules follow.
1. **Body ordering is load-bearing.** `express.raw` must be mounted on `/v1/webhooks` *before* the global `express.json()` in `app.ts`. Express runs middleware in registration order, so the webhook router terminates the request before the JSON parser is ever reached. Reversing these two lines silently breaks every signature check, because the raw bytes are gone by the time verification runs.
2. **No unsigned write reaches the database.** A webhook without a valid signature, and a REST call without a valid key, are both rejected before any DAO call.
3. **Secrets are encrypted at rest.** Access and refresh tokens are stored as AES-256-GCM ciphertext keyed by `TOKEN_ENCRYPTION_KEY`. API keys are never stored in recoverable form — only a SHA-256 hash, compared with `crypto.timingSafeEqual`.
## Data model
Two new tables alongside the existing `notes`.
**`integrations`** — one row per installed connection. Holds both credentials for a connection: `api_key_hash` is the inbound credential a client presents to us, `access_token_ciphertext` and `refresh_token_ciphertext` are the outbound credentials we present to the provider. Unique on `(provider, external_workspace_id)`.
**`ingest_events`** — one row per delivery. Unique on `(provider, external_id)`, which is what makes at-least-once delivery safe. Also carries `status`, `attempts`, and `last_error`, so the queue's in-memory state has a durable shadow and a restart can find work that was interrupted mid-flight.
## Technical risks
- **Queue durability.** `lib/queue` is in-process. A crash between the 200 ack and the insert loses that delivery; the `ingest_events` row is the only evidence. Acceptable for a proof of concept, and the row makes recovery possible, but this is the first thing to replace with a real broker before production.
- **Single-tenant assumptions.** The existing `notes` table has no workspace or board column. Multiple installed integrations all write into one flat board. Adding tenancy later means a migration on `notes`, not just the new tables.
- **Key rotation.** `TOKEN_ENCRYPTION_KEY` has no versioning in this design. Rotating it invalidates every stored token and forces reinstalls.
- **Clock skew on replay windows.** Signature timestamp tolerance is a fixed window; a badly skewed provider clock will fail verification in a way that looks like an attack.
- **Cohesion score sensitivity.** Duplicate notes from redelivered webhooks do more than clutter the board — they distort the silhouette calculation in [backend/services/validation.service.ts](backend/services/validation.service.ts), because near-identical vectors compress intra-cluster distance. Deduplication is a correctness requirement, not a tidiness one.
## Outbound (export) direction
Not built in this phase, but the shape is fixed now so the inbound work does not foreclose it: `lib/httpClient` provides retry and 429 backoff, `services/oauth.service` provides a valid token, and each future export adapter maps ranked clusters to one provider's write API. The registry is the single place an adapter registers itself.

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Steven Jannette
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

201
LOW_LEVEL_DESIGN.md Normal file
View File

@@ -0,0 +1,201 @@
# Low-level design
Phase 3 output. Exact interfaces for every module in [ROADMAP.md](ROADMAP.md), written before implementation. Error handling follows [agents.md](agents.md) section 2: async route handlers wrap in `try/catch`; services throw typed errors and let routes translate them into status codes.
## Shared types — `types/integration.ts`
```ts
export type ProviderSlug = 'slack' | 'jira' | 'linear' | 'github' | 'rest';
export type IntegrationRow = {
id: number;
provider: ProviderSlug;
externalWorkspaceId: string | null;
displayName: string | null;
scopes: string[];
tokenExpiresAt: Date | null;
};
export type SignatureResult =
| { ok: true }
| { ok: false; reason: 'missing' | 'malformed' | 'mismatch' | 'stale' };
export type NormalizedDelivery = {
externalId: string;
notes: NoteInput[];
};
export type DeliveryStatus = 'pending' | 'processing' | 'done' | 'failed';
```
String-literal unions rather than enums, per agents.md section 4.
## `lib/crypto.ts`
```ts
export const encryptSecret = (plaintext: string): string;
export const decryptSecret = (ciphertext: string): string;
export const hashApiKey = (key: string): string;
export const safeEquals = (a: string, b: string): boolean;
```
AES-256-GCM, serialized `base64(iv):base64(tag):base64(payload)`. `encryptSecret` throws if `TOKEN_ENCRYPTION_KEY` is missing or not 32 bytes after base64 decode. `safeEquals` pads to equal length before `timingSafeEqual` so it cannot leak length through an early throw.
Tests: round-trip fidelity, distinct ciphertext for identical plaintext (random IV), tamper detection on the auth tag, rejection of a short key, `safeEquals` true and false paths.
## `middleware/apiKey.ts`
```ts
declare module 'express-serve-static-core' {
interface Request { integration?: IntegrationRow }
}
export const requireApiKey: RequestHandler;
```
Reads `Authorization`, expects `Bearer <key>`. On success attaches `req.integration` and calls `next()`. On any failure responds `401 { error: 'Unauthorized' }` — identical body for missing, malformed, and unknown keys.
Tests: valid key passes and attaches, missing header 401s, wrong scheme 401s, unknown key 401s, database error surfaces as 500 via `next(err)`.
## `routes/ingest.routes.ts`
```ts
type IngestBody = { notes: NoteInput[] };
const isIngestBody = (value: unknown): value is IngestBody;
```
`POST /` returns `201 { inserted: number; notes: Note[] }`. Rejects an empty array and a batch over 5000 with `400`. The predicate checks `id`, `text`, and `author` are non-empty strings and that optional `x`, `y`, `color` have the right primitive types when present.
Tests: happy path inserts and echoes, missing `notes` key 400s, non-array 400s, element missing `text` 400s, empty array 400s, oversized batch 400s, unauthenticated 401s, DAO rejection 500s.
## `db/ingest_events.dao.ts`
```ts
export const recordDelivery = (input: {
provider: ProviderSlug;
externalId: string;
integrationId?: number;
}): Promise<number | null>;
export const markProcessing = (id: number): Promise<void>;
export const markDone = (id: number): Promise<void>;
export const markFailed = (id: number, error: string): Promise<void>;
export const resetStaleProcessing = (olderThanMs: number): Promise<number>;
```
`recordDelivery` resolves to `null` when the unique constraint fires, which is the caller's signal that this is a redelivery. `resetStaleProcessing` exists so a restart can recover rows the in-process queue was holding.
Tests: first delivery returns an id, identical redelivery returns null, different providers with the same external id both insert, status transitions persist, stale reset only touches `processing` rows past the cutoff.
## `lib/signatures.ts`
```ts
export const verifySignature = (input: {
provider: ProviderSlug;
rawBody: Buffer;
headers: IncomingHttpHeaders;
secret: string;
toleranceSeconds?: number;
}): SignatureResult;
```
Dispatches on the registry's `signatureScheme`. Never throws on bad input — a malformed header is a `{ ok: false }` result, because a thrown exception here would be an unhandled rejection path reachable by any anonymous caller.
Tests, per scheme: correct signature passes, altered body fails with `mismatch`, absent header fails with `missing`, timestamp outside tolerance fails with `stale`, and a signature of the right length but wrong content fails without a timing difference.
## `config/providers.ts`
```ts
export type ProviderConfig = {
slug: ProviderSlug;
signatureScheme: 'slack-v0' | 'github-sha256' | 'linear-sha256' | 'none';
signatureHeader: string;
timestampHeader?: string;
challenge?: (body: unknown) => { status: number; body: unknown } | null;
normalize: (payload: unknown) => NormalizedDelivery;
oauth?: {
authorizeUrl: string;
tokenUrl: string;
scopes: string[];
};
};
export const providers: Record<ProviderSlug, ProviderConfig>;
export const isProviderSlug = (value: string): value is ProviderSlug;
```
The `rest` entry uses scheme `none` and no normalizer of consequence — it exists so the registry is the complete inventory of inbound sources.
Tests: every slug resolves, `isProviderSlug` rejects unknown input, each entry declares a header when its scheme is not `none`.
## `services/normalize.service.ts`
```ts
export const normalize = (
provider: ProviderSlug,
payload: unknown
): NormalizedDelivery;
export const hasNormalizer = (provider: ProviderSlug): boolean;
```
Delegates to the registry entry. Throws `NormalizationError` when the payload lacks the fields that provider guarantees, and also when no normalizer is registered at all. Provenance written to `source_meta` as `{ provider, externalId, permalink, authorHandle, receivedAt }`.
Two normalizers ship with the infrastructure. `rest` is the generic bulk push and is a validation step rather than a translation. `linear` maps a comment webhook and exists so the webhook path is provable end to end; it is deliberately minimal and is not the Linear integration, which additionally needs OAuth install and label filtering. `slack`, `github`, and `jira` declare transport only.
Note ids are composed as `<provider>_<externalId>` and truncated to 64 characters, because `notes.id` is `VARCHAR(64)`.
Tests: a representative payload per shipped provider yields the expected notes, an empty message body yields zero notes rather than a blank note, provenance fields are populated, a long external id still fits the column, and a provider without a normalizer throws.
## `lib/queue.ts`
```ts
export type Job = () => Promise<void>;
export const enqueue = (name: string, job: Job): void;
export const size = (): number;
export const drain = (): Promise<void>;
export const configureQueue = (opts: {
maxAttempts?: number;
baseDelayMs?: number;
}): void;
```
Concurrency 1, exponential backoff with jitter, terminal failure logged and surfaced through the caller's `markFailed`. `drain` resolves when the queue is empty and no job is in flight, which is what makes the integration tests deterministic instead of timer-dependent.
Tests: jobs run in order, a failing job retries to the cap then stops, backoff delays grow, `drain` waits for in-flight work, `size` reflects pending count.
## `lib/httpClient.ts`
```ts
export const requestJson = <T>(url: string, init?: RequestInit & {
timeoutMs?: number;
maxAttempts?: number;
}): Promise<T>;
```
Retries 429 and 5xx, honors `Retry-After` when present, jittered exponential backoff otherwise, `AbortSignal.timeout` for the deadline. A 4xx other than 429 fails immediately — retrying a 401 just burns rate limit.
Tests: 200 parses, 429 with `Retry-After` waits then succeeds, 500 retries to cap, 400 fails without retry, timeout aborts.
## `routes/webhooks.routes.ts`
```ts
router.post('/:provider', rawBody, handler);
```
Handler order is fixed and each step has a distinct exit:
1. Unknown slug — `404`.
2. Body is not JSON — `400`.
3. Challenge request — provider-defined status and body.
4. No signing secret configured server-side — `500`, since that is our misconfiguration and not the caller's fault.
5. Signature invalid — `401`.
6. Provider registered but no normalizer yet — `501`. The transport is infrastructure and works; the payload mapping arrives with that provider's integration.
7. Payload fails normalization — `400`.
8. `recordDelivery` returns null — `200 { duplicate: true }`, no work enqueued.
9. Otherwise enqueue the insert and return `200 { accepted: true, notes: n }`.
Normalization runs *before* the ack rather than inside the queue job. It is a pure function, so it costs nothing on the request path, and doing it here means a malformed payload gets a `400` the sender can act on instead of failing silently in a background job. Only the database write is deferred.
Tests: each numbered branch, plus the ordering regression — sign a body with irregular internal whitespace and assert verification still succeeds, which can only happen if the exact bytes survived `express.raw` ahead of `express.json`.

199
README.md
View File

@@ -1,40 +1,188 @@
# kongruity app
# kongruity: Signal from noise
kongruity employs Large Language Model ("LLM") semantic grouping functionality to cluster large volumes of "to dos" or issue tags in development (or other) settings, according to thematic or topical similarity.
“...All those moments will be lost in time, like tears in rain.”
(To learn more about this topic, see, e.g., [Kozlowski A., Boutyline A., Semantic Structure in Large Language Model Embeddings Aug. 2025, arXiv:2508.10003v1:04 Aug 2025](https://arxiv.org/html/2508.10003v1)).
kongruity pulls in unstructured artifacts of the creative-engineering process -- to-dos, action items, agile tickets, Jira thread comments, Slack thread comments, retrospective notes -- and synthesizes them into semantically coherent, prioritized clusters that can be incorporated into implementation planning.
In the world of kongruity, these "to dos" are called "sticky notes." kongruity's React/Vite UI views a board of seemingly chaotic "sticky notes". But with one click, they are transformed into manageable, actionable groups, each with a header that explains the group semantic interrelation.
In kongruity, the artifacts become "sticky notes." A board full of them looks chaotic.
The backend is an Express API that serves "sticky note" data and proxies semantic grouping requests to Antrhopic Claude.
With a click, they are semantically evaluated, grouped into thematic clusters with descriptive headers, rankable and exportable to project planning and execution tools.
Developers may feel free to install other LLM SDKs and alter the syntax at backend/services/clustering.service.js to experiment with any LLM model/platform they prefer.
## Voyage AI voyage-3.5
![Embedding model benchmarking.](Voyage.jpg)
## Clustering and evaluation: methodology
Two models run in parallel, and neither sees the other's work. Anthropic's `claude-sonnet-5` (`backend/services/clustering.service.js`) reads the raw text of every note and groups them into labeled thematic clusters.
At the same time, Voyage AI's voyage-3.5 model (`backend/services/embedding.service.js`) converts each note's text into a numeric representation of its semantic meaning aka vector.
Once the LLM returns, kongruity scores that grouping (`backend/services/validation.service.js`) using an established silhouette coefficient, with cosine distance rather than Euclidean as the proximity metric.
For each note, it weighs the average distance to the other notes in its own cluster against the average distance to the notes in the nearest neighboring cluster. Averaged across every note, this yields a single numeric cohesion score, displayed at the top of the results, along with plaintext: Strong, Moderate, Weak, Poor.
This yields an empirical groundedness evaluation. One model proposes the grouping; an independent model evaluates grouping accuracy.
Note that: before scoring, structural validation confirms that each note landed in exactly one cluster, that no cluster is empty, and that no hallucinated note IDs appear. A malformed response to the validation completely fails, rather than quietly returning a partial board.
## Reading the cohesion score
Average silhouette width is a widely-used measure of clustering quality. Higher values indicate:
1. The qualitative semantic cohesiveness of clusters, and:
2. How well-separated each cluster is from its nearest neighboring cluster.
Although the coefficient is mathematically bounded by [−1, 1], cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. Because silhouette divides the gap between them by the larger of the two, the practical range on embedding data is roughly [−0.05, 0.10] rather than the full interval.
The bands below are therefore calibrated against that observed range. On the seed board, the five ideal thematic clusters score 0.09; swapping a few notes between clusters drops it to 0.06; a scrambled assignment falls below zero.
The score appears above the results with a plain-language band:
- **0.07 and above** — Strong
- **0.04 to 0.06** — Moderate
- **0.01 to 0.03** — Weak
- **Below 0.01** — Poor
A score near 0.00 means the grouping is no better than chance. Bands are specific to `voyage-3` cosine distance and would need recalibration behind a different embedding model. See Hugo Sträng, Tai Dinh. An upper bound on the silhouette evaluation metric for clustering. Pattern Recognition, Volume 178, 2026, 113402, ISSN 0031-3203.
## Organizing clusters, exporting to workflow software
Teams can drag-and-rank related task clusters by implementation priority - turning noise into an actionable workflow.
(Integrations with third-party project management, planning and workflow applications are action-items for next major version, see Roadmap, below)
## How it works
1. **Ingest** — Sticky notes are loaded and displayed on a board.
2. **Cluster** — An LLM reads every note and groups them by semantic similarity (not keywords).
3. **Evaluate** — In parallel, a separate embedding model (Voyage AI) generates vector representations of each note. A silhouette-based cohesion score measures how well-separated and internally consistent clusters are. The score is displayed alongside the results.
4. **Validate** — Structural checks confirm every note is assigned to exactly one cluster, no clusters are empty, and labels are present.
5. **Prioritize** — Clusters appear ranked and are drag-reorderable. Teams set implementation priority by dragging clusters into position.
## Dev implementation notes
Developers may swap in other LLM SDKs/APIs and alter prompt syntax in `backend/services/clustering.service.ts` to experiment with LLMs and platforms of their choice.
## Integration API
The shared infrastructure that third-party integrations are built on. See [ARCHITECTURE.md](ARCHITECTURE.md) for the design, [ROADMAP.md](ROADMAP.md) for milestones, and [LOW_LEVEL_DESIGN.md](LOW_LEVEL_DESIGN.md) for module contracts.
| Endpoint | Auth | Purpose |
| --- | --- | --- |
| `POST /v1/notes` | `Authorization: Bearer <api key>` | Bulk push notes. Body `{ notes: [{ id, text, author, x?, y?, color? }] }`, max 5000 per request. |
| `POST /v1/webhooks/:provider` | Per-provider request signature | Inbound provider deliveries. Acks immediately, inserts in the background. |
Registered providers live in `backend/config/providers.ts`. `rest` and `linear` carry payload mappings today; `slack`, `github`, and `jira` declare transport and OAuth endpoints only, so adding one of those integrations means writing a single normalizer rather than new plumbing.
Deliveries are deduplicated on `(provider, external_id)` in the `ingest_events` table, which matters beyond tidiness: duplicate notes compress intra-cluster distance and depress the cohesion score.
### Adding a provider
1. Add a slug to `ProviderSlug` in `backend/types/integration.ts`.
2. Add an entry to `providers` in `backend/config/providers.ts` with its signature scheme, header names, and OAuth endpoints.
3. Write one normalizer in `backend/config/normalizers.ts` mapping that provider's payload to notes.
4. Set the provider's signing secret and OAuth credentials in `backend/.env`.
## Development Roadmap
### Ingestion — pulling tagged artifacts in
- [ ] **Slack** — where decisions actually get made; a `:sticky:` emoji reaction fires an Events API webhook that pulls the message in.
- [ ] **Microsoft Teams** — same capture gesture for enterprise shops; message extension plus Graph change notifications.
- [ ] **Jira** — label- or mention-triggered webhook scoped by JQL. (This is where comments typically carry half the backlog's context.)
- [ ] **Linear** — engineering-side tickets and threads; label-triggered GraphQL webhook.
- [ ] **GitHub** — issue, PR review, and discussion comments; label- or mention-triggered webhook.
- [ ] **Miro / FigJam** — REST API import
- [ ] **Confluence / Notion** — page and inline-comment fetch. (Where retro and planning notes are born).
- [ ] **Meeting transcripts (Granola, Otter, Zoom, Google Meet)** — where retros are now recorded, an option for action-item extraction from the transcript API.
- [ ] **Generic REST, email, and Zapier** — authenticated bulk `POST /v1/notes`.
### Export — pushing ranked clusters to workflow tools
- [ ] **Jira** — drag-rank written through the Agile API's board rank endpoint.
- [ ] **Asana** — drag-rank written as task order within the section.
- [ ] **Rally** — drag-rank written as portfolio rank. (Clusters become features and notes, which become stories).
- [ ] **Linear** — drag-rank written to issue `sortOrder`.
- [ ] **Azure DevOps / GitHub Projects v2** — drag-rank written as project field ordering. Clusters become work-item parents.
- [ ] **CSV, JSON, and Markdown** — direct download from the cluster view. (Should ship before any OAuth work.)
## Prerequisites
- Node.js (v18 or later recommended)
- An [Anthropic API key](https://console.anthropic.com/)
- PostgreSQL (v14 or later recommended)
- An [Anthropic API key](https://console.anthropic.com/) (or other LLM platform, for clustering)
- A [Voyage AI API key](https://dash.voyageai.com/) (for embedding-based evaluation)
## Setup
### 1. Unzip the project
### 1. Clone the repository
```bash
unzip kongruity.zip
git clone https://github.com/kjannette/kongruity_
cd kongruity
```
### 2. Create an environment file
The backend expects a `.env` file containing an Anthropic API key in the root `backend/` directory. This file is git-ignored and must be created manually:
The backend expects a `.env` file in the `backend/` directory. This file is git-ignored and must be created manually:
```bash
echo 'ANTHROPIC_API_KEY=<your Anthropic API key>' > backend/.env
cat > backend/.env << 'EOF'
ANTHROPIC_API_KEY=<your Anthropic API key> (or other LLM platform key)
VOYAGEAI_API_KEY=<your Voyage AI API key>
DATABASE_URL=postgresql://<user>:<password>@localhost:5432/kongruity
EOF
```
Replace `<your Anthropic API key>` with your actual key.
Replace placeholder values with your actual keys and database credentials.
### 3. Install dependencies
#### Integration variables
Only needed once you start using the ingestion endpoints. `TOKEN_ENCRYPTION_KEY` is required by anything that stores third-party credentials; generate one with `openssl rand -base64 32`.
```bash
cat >> backend/.env << 'EOF'
TOKEN_ENCRYPTION_KEY=<32 bytes, base64 encoded>
LINEAR_SIGNING_SECRET=<webhook signing secret from Linear>
SLACK_SIGNING_SECRET=<webhook signing secret from Slack>
GITHUB_WEBHOOK_SECRET=<webhook secret from your GitHub App>
LINEAR_CLIENT_ID=<OAuth client id>
LINEAR_CLIENT_SECRET=<OAuth client secret>
EOF
```
The key must decode to exactly 32 bytes; the backend refuses to encrypt otherwise rather than falling back to something weaker.
### 3. Set up/run the database
Start DB for local development (assumes local dev env MacOS and Homebrew installed)
```bash
brew services start postgresql@15
```
Create a PostgreSQL database for the project:
```bash
createdb kongruity
```
Run the migration to create tables:
```bash
cd backend
npm run db:migrate
```
Seed the database with the sample sticky notes:
```bash
npm run db:seed
```
### 4. Install dependencies
```bash
cd backend
@@ -48,25 +196,32 @@ npm install
## Run the app
### Start the backend - Production Mode
### Start the backend — Production mode
From the `backend/` directory:
From the `backend/` directory, compile the TypeScript sources and run the output:
```bash
npm run build
npm run start
```
The API server starts on **http://localhost:3001** (configurable via the `PORT` environment variable).
### Start the backend - Development mode
### Start the backend — Development mode
To start using Nodemon for "hot reloads," if developing your own features:
To run the TypeScript sources directly with hot reloads while developing:
```bash
npm run dev
```
### Build the frontend - Production mode
To type-check without emitting:
```bash
npm run type-check
```
### Build the frontend — Production mode
From the `frontend/` directory:
@@ -74,13 +229,14 @@ From the `frontend/` directory:
npm run build
```
### Start the frontend - Development mode
### Start the frontend — Development mode
From the `frontend/` directory:
```bash
npm run dev
```
The Vite dev server starts on **http://localhost:5173** by default. Open that URL in a browser.
## Running tests
@@ -92,7 +248,8 @@ From the `backend/` directory:
```bash
npm test
```
Backend tests use Supertest for HTTP assertions.
Backend tests use Vitest with Supertest for HTTP assertions.
### Frontend tests
@@ -102,7 +259,7 @@ From the `frontend/` directory:
npm test
```
This runs `vitest run` with jsdom. For watch mode, during development:
This runs Vitest with jsdom. For watch mode during development:
```bash
npm run test:watch

98
ROADMAP.md Normal file
View File

@@ -0,0 +1,98 @@
# Roadmap: integration infrastructure
Phase 2 output. Deconstructs [ARCHITECTURE.md](ARCHITECTURE.md) into ordered milestones with explicit contracts. Every path is relative to `backend/`.
## Milestone dependency order
```mermaid
flowchart LR
M0["M0 TypeScript migration (done)"] --> M1["M1 Write path"]
M1 --> M2["M2 Inbound"]
M1 --> M3["M3 Identity"]
M2 --> M4["M4 Transform and deliver"]
M3 --> M4
M4 --> M5["M5 Verification"]
```
M2 and M3 both depend only on M1 and can run in parallel.
## Data models
### `integrations`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | `SERIAL PRIMARY KEY` | |
| `provider` | `VARCHAR(32) NOT NULL` | registry slug, e.g. `slack`, `rest` |
| `external_workspace_id` | `VARCHAR(128)` | provider's own workspace identifier |
| `display_name` | `VARCHAR(255)` | for operator legibility |
| `api_key_hash` | `CHAR(64)` | SHA-256 of the inbound key; null unless the integration accepts pushes |
| `access_token_ciphertext` | `TEXT` | AES-256-GCM, `iv:tag:payload` base64 triple |
| `refresh_token_ciphertext` | `TEXT` | same encoding |
| `token_expires_at` | `TIMESTAMPTZ` | drives refresh-before-use |
| `scopes` | `TEXT[]` | granted at install |
| `signing_secret_ciphertext` | `TEXT` | per-install webhook secret where the provider issues one |
| `created_at` / `updated_at` | `TIMESTAMPTZ DEFAULT NOW()` | |
Unique on `(provider, external_workspace_id)`. Index on `api_key_hash`.
### `ingest_events`
| Column | Type | Notes |
| --- | --- | --- |
| `id` | `SERIAL PRIMARY KEY` | |
| `provider` | `VARCHAR(32) NOT NULL` | |
| `external_id` | `VARCHAR(255) NOT NULL` | provider's event or message identifier |
| `integration_id` | `INTEGER REFERENCES integrations(id)` | nullable for unmatched deliveries |
| `status` | `VARCHAR(16) NOT NULL DEFAULT 'pending'` | `pending`, `processing`, `done`, `failed` |
| `attempts` | `INTEGER NOT NULL DEFAULT 0` | |
| `last_error` | `TEXT` | |
| `received_at` / `updated_at` | `TIMESTAMPTZ DEFAULT NOW()` | |
Unique on `(provider, external_id)`. This constraint is the deduplication mechanism: a redelivery hits the conflict and is dropped.
## M1 — Write path
**Ticket 1.1 — `db/migrate.ts`.** Add both tables above to the existing `up` script, `CREATE TABLE IF NOT EXISTS` in keeping with current style. Verify with `npm run db:migrate` against a live database.
**Ticket 1.2 — `middleware/apiKey.ts`.** Express middleware reading `Authorization: Bearer <key>`, hashing with SHA-256, looking up `integrations.api_key_hash`, comparing with `crypto.timingSafeEqual`, attaching the row to `req`. Responds 401 with `{ error: string }` on any failure and never distinguishes "no such key" from "wrong key" in the response.
**Ticket 1.3 — `routes/ingest.routes.ts`.** `POST /v1/notes` accepting `{ notes: NoteInput[] }`. Body narrowed by a type predicate, not a schema library. Returns `201` with `{ inserted: number, notes: Note[] }`, `400` on malformed body, `401` from the middleware. Delegates to the existing `createNotes` in [backend/db/notes.dao.ts](backend/db/notes.dao.ts), which already batches under the Postgres bind-parameter ceiling.
**Ticket 1.4 — `app.ts` wiring.** Mount the ingest router. Ordering does not matter yet; it becomes critical in M2.
## M2 — Inbound
**Ticket 2.1 — `db/ingest_events.dao.ts`.** `recordDelivery` performing `INSERT ... ON CONFLICT (provider, external_id) DO NOTHING RETURNING id`, returning null when the row already existed so the caller can drop the redelivery. Plus `markProcessing`, `markDone`, `markFailed`.
**Ticket 2.2 — `lib/signatures.ts`.** `verifySignature(provider, rawBody, headers, secret)` returning a discriminated result rather than throwing. HMAC comparison via `crypto.timingSafeEqual` on equal-length buffers. Enforces a replay window, default 300 seconds.
**Ticket 2.3 — `routes/webhooks.routes.ts`.** `POST /v1/webhooks/:provider`. Resolves the provider from the registry (404 on unknown), answers challenge requests, verifies the signature (401 on failure), records the delivery (200 and stop on duplicate), enqueues, then returns 200. Must respond within the tightest provider budget, which is Slack's three seconds.
**Ticket 2.4 — `app.ts` ordering.** Mount `express.raw({ type: '*/*' })` with the webhook router *before* `app.use(express.json())`. This is the single most breakable line in the system and needs a regression test of its own.
## M3 — Identity
**Ticket 3.1 — `lib/crypto.ts`.** `encryptSecret` and `decryptSecret` over AES-256-GCM using `TOKEN_ENCRYPTION_KEY`, node:crypto only. Fails loudly at startup if the key is absent or not 32 bytes.
**Ticket 3.2 — `db/integrations.dao.ts`.** CRUD returning decrypted secrets only through explicit accessors, so a careless `SELECT *` cannot leak plaintext. `findByApiKeyHash`, `findByProviderWorkspace`, `upsertInstall`, `updateTokens`.
**Ticket 3.3 — `services/oauth.service.ts`.** `buildAuthorizeUrl`, `exchangeCode`, `getValidAccessToken` refreshing when `token_expires_at` is inside a 60-second margin. Provider-specific endpoints come from the registry, not from this file.
**Ticket 3.4 — `config/providers.ts`.** The registry. Each entry declares slug, signature scheme, header names, normalizer, OAuth endpoints, and scopes. Adding a provider must touch only this file plus one normalizer.
## M4 — Transform and deliver
**Ticket 4.1 — `services/normalize.service.ts`.** `normalize(provider, payload): NormalizedDelivery` returning `{ externalId: string, notes: NoteInput[] }`. Writes provenance into `source_meta`: provider, external id, permalink, author handle, received timestamp.
**Ticket 4.2 — `lib/queue.ts`.** In-process FIFO with concurrency 1, exponential backoff, and a cap on attempts. On terminal failure calls `markFailed`. Exposes `enqueue`, `size`, and a `drain` promise for deterministic testing.
**Ticket 4.3 — `lib/httpClient.ts`.** `requestJson` with timeout, retry on 429 and 5xx, `Retry-After` respected, jittered exponential backoff. Used by OAuth exchange today and export adapters later.
## M5 — Verification
End-to-end test driving a signed synthetic-provider delivery through the whole chain and asserting the note lands. Explicit cases: valid delivery inserts, replayed delivery is dropped, bad signature is rejected, unknown provider 404s, malformed REST body 400s, missing key 401s, and the raw-body ordering regression.
## Definition of done per milestone
`npm run type-check`, `npm run build`, and `npm test` all clean, with new unit tests for every module and no change in behavior to the four pre-existing suites.

BIN
Voyage.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 114 KiB

76
agentic-orchestration.md Normal file
View File

@@ -0,0 +1,76 @@
# This framework pertaining to long-horizion work utilizing orchestration.
You should assume the tole of Lead Orchestrator Agent for a complex, long-horizon software development project.
# The first rule. Hereinfter, the "King's Rule".
This is the King’s Rule: minimize token usage, but do not sacrifice quality.
Strive for strict compliance to the King’s Rule.
Here is a non-exhaustive list of suggested strategies pertaining to the King’s Rule.
### Sub-agents
Main Agent (YOU) coordinates high-level plan.
Context limit work-around. Rather than one agent maintaining state across the project, specialized sub-agents handle focused tasks with clean context windows.
Subagents perform deep technical work, use tools to find relevant information, if needed.
Each subagent might explore (using tens of thousands of tokens) but returns only a condensed, distilled summary of work (1,000-2,000 tokens).
Achieve clear separation of concerns: detailed search context remains isolated within sub-agents, while lead agent focuses on synthesizing and analyzing the results.
Strive for substantial improvement over single-agent systems on complex research tasks
# Your job
Break down the user's high-level goal, delegate work to specialized subagents, maintain a global state/context (delinated as described above), and execute the plan from architecture down to final, detailed code implementation.
### High-level goal
BUILD ALL INFRASTRUCTURE TO SUPPORT THIRD PARTY APP INTERGRATIONS AS FOUND IN THIS FILE:
/Users/kjannette/.cursor/projects/Users-kjannette-workspace2-kongruity/canvases/roadmap-integration-ranking.canvas.tsx
### Execution Phases & Subagent Responsibilities
#### Phase 1: High-Level Architecture & System Design
* **Role:** Lead Architect Subagent
* **Tasks:**
1. Define the system architecture, core components, data flow, and technology stack.
2. Identify key technical risks, scaling constraints, and security requirements.
3. Output a structured `ARCHITECTURE.md` and a high-level component dependency graph.
#### Phase 2: Specification & Task Breakdown
* **Role:** Product/Technical Project Manager Subagent
* **Tasks:**
1. Deconstruct the architecture into discrete, ordered, and independent sub-tasks (epics and tickets).
2. Define clear input/output contracts, API schemas, and data models for each module.
3. Output a sequential `ROADMAP.md` tracking dependencies and milestones.
#### Phase 3: Low-Level Design & Interface Contracts
* **Role:** Systems Engineer Subagent
* **Tasks:**
1. For each module in the roadmap, write detailed low-level specifications (function signatures, class structures, error-handling strategies).
2. Define unit and integration test strategies/scaffolding for each module before code is written.
#### Phase 4: Iterative Implementation & Testing
* **Role:** Developer Subagents (Frontend, Backend, Database, DevOps as needed)
* **Tasks:**
1. Implement modules strictly following the low-level specs, one milestone at a time.
2. Write and execute tests for each implemented module, fixing failures before moving forward.
3. Perform continuous code review and integration checks against the global architecture.
#### Phase 5: Verification & System Integration
* **Role:** QA & Integration Subagent
* **Tasks:**
1. Run end-to-end integration tests across all completed modules.
2. Verify performance, security benchmarks, and edge cases.
3. Generate the final deployment guide and documentation.
### Operational Rules for the Orchestrator
1. **State Persistence:** Maintain a persistent memory/context of completed milestones and update the roadmap dynamically if blockers occur.
2. **Sequential Gatekeeping:** Do not transition to Phase 4 (Implementation) until Phases 1–3 are fully documented.
3. **Error Recovery:** If a subagent encounters a low-level failure, pause, re-evaluate the interface contract, and spawn a debugging subagent before proceeding.

42
agents.md Normal file
View File

@@ -0,0 +1,42 @@
# AI Agent Instructions: Fullstack Vite 7 (React) + Express + TypeScript + npm
You are an expert AI fullstack software engineer specialized in Vite 7, React, Express, TypeScript, and modern web architectures. Follow these rules strictly when modifying this codebase.
## 1. Project Structure & Context
* **Frontend:** React SPA powered by Vite 7.x (Entry: `src/main.tsx` or client folder).
* **Backend:** Express Node.js application (Server entry: `server.ts` or server folder).
* **Package Manager:** npm (`package-lock.json` is the strict source of truth).
* **TypeScript Setup:** Strict Mode enabled independently across both environments.
## 2. Express Backend TypeScript Rules
* **Typed Request/Response:** Explicitly type Express route handlers using native Express types:
```typescript
import { Request, Response, NextFunction } from 'express';
// Example for typed request bodies/params:
interface CreateUserBody { username: string; }
app.post('/user', (req: Request<{}, {}, CreateUserBody>, res: Response) => { ... });
```
* **Async Error Catching:** Always wrap async middleware/route handlers in `try/catch` and pass errors to `next(err)`. Do not let unhandled promise rejections crash the Node process.
* **Shared Types:** If frontend and backend share types (e.g., API payloads, User models), place them in a shared directory or export them cleanly from the backend to prevent duplicating code.
## 3. Frontend React + Vite Rules
* **Component Typings:** Use standard type inference or explicit return types (`function Component(): React.JSX.Element`). Avoid the legacy `React.FC`.
* **Strict Prop Types:** Every component must have an explicitly typed `interface` or `type` for its props. No implicit `any`.
* **Event Handlers:** Use exact React synthetic event types (e.g., `React.ChangeEvent<HTMLInputElement>`) instead of generic native events.
* **File Extensions:** Use `.tsx` exclusively for files containing JSX. Use `.ts` strictly for pure logic, hooks, or type definitions.
## 4. Strict Code Quality & Native Guards
* **No `any`:** Never use `any`. Use `unknown` for unpredictable runtime data (like Express `req.body` or frontend `fetch` payloads).
* **No Validation Libraries:** Do not install Zod, TypeBox, or Yup. Write explicit, manual type predicate functions (`function isUser(obj: any): obj is User`) to safely validate runtime data incoming to both the server and client.
* **No Enums:** Avoid TypeScript `enum`. Use string-literal unions (`type Status = 'active' | 'pending'`) or `const StatusEnum = { ... } as const`.
## 5. Verification & Workflow Commands
Before declaring a task complete, you must verify both environments compile flawlessly via npm:
* **Install Dependencies:** `npm install`
* **Type-Check Project:** Run the designated workspace or folder type-checking scripts (e.g., `npm run type-check` or `npx tsc --noEmit` across both roots).
* **Build Verification:** Run production build scripts (e.g., `npm run build`) to ensure both Express asset compilation and Vite bundling pass without error.
## 6. How to Respond
* **Verify Types First:** Run type-checking commands automatically after modifying files to capture compilation breaks before presenting the solution.
* **Targeted Diffs:** Provide concise, targeted updates. Do not rewrite whole files if only a few lines change.
* **Self-Correct:** If a build command fails, read the compiler/Vite/Node logs, fix the root cause, and re-test before asking the user for help.

View File

@@ -1,16 +0,0 @@
import express from 'express';
import cors from 'cors';
import router from './routes/notes.routes.js';
const app = express();
app.use(cors());
app.use(express.json());
app.use('/v1/notes', router);
app.use((req, res) => {
res.status(404).json({ error: `Requested path is invalid or does not exist: ${req.method} ${req.originalUrl}` });
});
export default app;

30
backend/app.ts Normal file
View File

@@ -0,0 +1,30 @@
import express, { type NextFunction, type Request, type Response } from 'express';
import cors from 'cors';
import notesRouter from './routes/notes.routes.js';
import ingestRouter from './routes/ingest.routes.js';
import webhooksRouter from './routes/webhooks.routes.js';
const app = express();
app.use(cors());
// Order matters: the raw parser must claim webhook bodies before express.json
// consumes them, because signature verification needs the exact bytes sent.
// Moving this below express.json silently breaks every signature check.
app.use('/v1/webhooks', express.raw({ type: '*/*', limit: '2mb' }), webhooksRouter);
app.use(express.json({ limit: '2mb' }));
app.use('/v1/notes', ingestRouter);
app.use('/v1/notes', notesRouter);
app.use((req: Request, res: Response) => {
res.status(404).json({ error: `Requested path is invalid or does not exist: ${req.method} ${req.originalUrl}` });
});
app.use((err: Error, _req: Request, res: Response, _next: NextFunction) => {
console.error(`Unhandled error: ${err.stack ?? err.message}`);
if (res.headersSent) return;
res.status(500).json({ error: 'Internal server error' });
});
export default app;

View File

@@ -0,0 +1,113 @@
import type { NormalizedDelivery, NoteProvenance } from '../types/integration.js';
import type { NoteInput } from '../types/domain.js';
export class NormalizationError extends Error {
constructor(message: string) {
super(message);
this.name = 'NormalizationError';
}
}
const asRecord = (value: unknown): Record<string, unknown> => {
if (typeof value !== 'object' || value === null || Array.isArray(value)) {
throw new NormalizationError('Payload is not an object');
}
return value as Record<string, unknown>;
};
const asString = (value: unknown): string | undefined =>
typeof value === 'string' && value.length > 0 ? value : undefined;
/** notes.id is VARCHAR(64), so composite keys have to be trimmed to fit. */
const noteId = (provider: string, externalId: string): string =>
`${provider}_${externalId}`.slice(0, 64);
const provenance = (
meta: Omit<NoteProvenance, 'receivedAt'>
): Record<string, unknown> => ({
...meta,
receivedAt: new Date().toISOString(),
});
const isNoteInput = (value: unknown): value is NoteInput => {
if (typeof value !== 'object' || value === null) return false;
const note = value as Record<string, unknown>;
if (typeof note.id !== 'string' || note.id.length === 0) return false;
if (typeof note.text !== 'string' || note.text.length === 0) return false;
if (typeof note.author !== 'string' || note.author.length === 0) return false;
if (note.x !== undefined && typeof note.x !== 'number') return false;
if (note.y !== undefined && typeof note.y !== 'number') return false;
if (note.color !== undefined && typeof note.color !== 'string') return false;
return true;
};
export const isNoteInputArray = (value: unknown): value is NoteInput[] =>
Array.isArray(value) && value.every(isNoteInput);
/**
* Direct push: the caller already speaks our note shape, so normalization is
* a validation step rather than a translation.
*/
export const normalizeRest = (payload: unknown): NormalizedDelivery => {
const body = asRecord(payload);
if (!isNoteInputArray(body.notes)) {
throw new NormalizationError('Expected a "notes" array of note objects');
}
const externalId = asString(body.batchId) ?? randomBatchId();
return {
externalId,
notes: body.notes.map((note) => ({
...note,
sourceMeta: provenance({ provider: 'rest', externalId }),
})),
};
};
const randomBatchId = (): string =>
`batch_${Date.now()}_${Math.random().toString(36).slice(2, 10)}`;
/**
* Linear comment webhook. Kept minimal on purpose: this proves the webhook
* pipeline end to end. Label filtering and issue-thread context belong to the
* Linear integration proper, not to this infrastructure.
*/
export const normalizeLinear = (payload: unknown): NormalizedDelivery => {
const body = asRecord(payload);
const data = asRecord(body.data);
const externalId = asString(data.id);
if (!externalId) {
throw new NormalizationError('Linear payload is missing data.id');
}
const text = asString(data.body);
if (!text) {
return { externalId, notes: [] };
}
const user = typeof data.user === 'object' && data.user !== null
? (data.user as Record<string, unknown>)
: {};
return {
externalId,
notes: [
{
id: noteId('linear', externalId),
text,
author: asString(user.name) ?? 'unknown',
sourceMeta: provenance({
provider: 'linear',
externalId,
permalink: asString(data.url),
authorHandle: asString(user.name),
}),
},
],
};
};

View File

@@ -0,0 +1,95 @@
import type {
NormalizedDelivery,
ProviderSlug,
SignatureScheme,
} from '../types/integration.js';
import { normalizeLinear, normalizeRest } from './normalizers.js';
export type ChallengeResponse = { status: number; body: unknown };
export type ProviderConfig = {
slug: ProviderSlug;
signatureScheme: SignatureScheme;
/** Header carrying the signature. Empty only when the scheme is 'none'. */
signatureHeader: string;
timestampHeader?: string;
/** Environment variable holding the shared secret when the install has none. */
secretEnvVar?: string;
/** Returns a response when the request is a handshake rather than an event. */
challenge?: (body: unknown) => ChallengeResponse | null;
/**
* Absent until that provider's integration is built. The transport above is
* infrastructure; the payload mapping belongs to the integration itself.
*/
normalize?: (payload: unknown) => NormalizedDelivery;
oauth?: {
authorizeUrl: string;
tokenUrl: string;
scopes: string[];
};
};
const slackChallenge = (body: unknown): ChallengeResponse | null => {
if (typeof body !== 'object' || body === null) return null;
const candidate = body as { type?: unknown; challenge?: unknown };
if (candidate.type !== 'url_verification') return null;
if (typeof candidate.challenge !== 'string') return null;
return { status: 200, body: { challenge: candidate.challenge } };
};
export const providers: Record<ProviderSlug, ProviderConfig> = {
rest: {
slug: 'rest',
signatureScheme: 'none',
signatureHeader: '',
normalize: normalizeRest,
},
linear: {
slug: 'linear',
signatureScheme: 'linear-sha256',
signatureHeader: 'linear-signature',
secretEnvVar: 'LINEAR_SIGNING_SECRET',
normalize: normalizeLinear,
oauth: {
authorizeUrl: 'https://linear.app/oauth/authorize',
tokenUrl: 'https://api.linear.app/oauth/token',
scopes: ['read'],
},
},
slack: {
slug: 'slack',
signatureScheme: 'slack-v0',
signatureHeader: 'x-slack-signature',
timestampHeader: 'x-slack-request-timestamp',
secretEnvVar: 'SLACK_SIGNING_SECRET',
challenge: slackChallenge,
oauth: {
authorizeUrl: 'https://slack.com/oauth/v2/authorize',
tokenUrl: 'https://slack.com/api/oauth.v2.access',
scopes: ['channels:history', 'reactions:read', 'users:read'],
},
},
github: {
slug: 'github',
signatureScheme: 'github-sha256',
signatureHeader: 'x-hub-signature-256',
secretEnvVar: 'GITHUB_WEBHOOK_SECRET',
oauth: {
authorizeUrl: 'https://github.com/login/oauth/authorize',
tokenUrl: 'https://github.com/login/oauth/access_token',
scopes: ['repo', 'read:discussion'],
},
},
jira: {
slug: 'jira',
signatureScheme: 'none',
signatureHeader: '',
oauth: {
authorizeUrl: 'https://auth.atlassian.com/authorize',
tokenUrl: 'https://auth.atlassian.com/oauth/token',
scopes: ['read:jira-work', 'offline_access'],
},
},
};
export const getProvider = (slug: ProviderSlug): ProviderConfig => providers[slug];

View File

@@ -1,402 +0,0 @@
[
{
"id": "note_038",
"text": "Live follow mode frequently breaks",
"x": 710,
"y": 579,
"author": "user_9",
"color": "yellow"
},
{
"id": "note_005",
"text": "SSO login loops back to the sign-in page",
"x": 184,
"y": 124,
"author": "user_7",
"color": "yellow"
},
{
"id": "note_023",
"text": "Undo/redo sometimes lags and applies late",
"x": 236,
"y": 687,
"author": "user_9",
"color": "purple"
},
{
"id": "note_017",
"text": "Exported file names are inconsistent and hard to track",
"x": 676,
"y": 201,
"author": "user_4",
"color": "blue"
},
{
"id": "note_049",
"text": "Need better controls for aligning and distributing notes",
"x": 462,
"y": 487,
"author": "user_7",
"color": "green"
},
{
"id": "note_010",
"text": "OAuth consent screen appears every time I sign in",
"x": 228,
"y": 124,
"author": "user_9",
"color": "yellow"
},
{
"id": "note_036",
"text": "Cursor presence is distracting and overlaps content",
"x": 788,
"y": 605,
"author": "user_5",
"color": "yellow"
},
{
"id": "note_011",
"text": "Need better export options (PDF quality is too low)",
"x": 748,
"y": 212,
"author": "user_9",
"color": "green"
},
{
"id": "note_012",
"text": "Exported PDF cuts off content near the edges",
"x": 733,
"y": 159,
"author": "user_6",
"color": "blue"
},
{
"id": "note_019",
"text": "Embedded exports don’t update when the board changes",
"x": 662,
"y": 132,
"author": "user_1",
"color": "blue"
},
{
"id": "note_020",
"text": "Can’t export comments and reactions with the content",
"x": 739,
"y": 139,
"author": "user_7",
"color": "green"
},
{
"id": "note_039",
"text": "Guest collaborators can’t see updates without refreshing",
"x": 759,
"y": 711,
"author": "user_9",
"color": "yellow"
},
{
"id": "note_024",
"text": "App crashes when opening a large board",
"x": 239,
"y": 597,
"author": "user_10",
"color": "purple"
},
{
"id": "note_041",
"text": "Hard to find the right template quickly",
"x": 536,
"y": 394,
"author": "user_4",
"color": "orange"
},
{
"id": "note_040",
"text": "Activity feed lacks context about what changed",
"x": 710,
"y": 771,
"author": "user_7",
"color": "yellow"
},
{
"id": "note_014",
"text": "Export takes too long and sometimes never finishes",
"x": 798,
"y": 211,
"author": "user_2",
"color": "green"
},
{
"id": "note_031",
"text": "Hard to tell who is editing what in real time",
"x": 761,
"y": 704,
"author": "user_8",
"color": "yellow"
},
{
"id": "note_032",
"text": "Comments get lost — no clear thread view",
"x": 818,
"y": 641,
"author": "user_5",
"color": "yellow"
},
{
"id": "note_050",
"text": "Can’t lock sections to prevent accidental edits during workshops",
"x": 521,
"y": 471,
"author": "user_6",
"color": "orange"
},
{
"id": "note_001",
"text": "Login flow feels confusing",
"x": 193,
"y": 191,
"author": "user_5",
"color": "yellow"
},
{
"id": "note_025",
"text": "Saving indicator spins but changes aren’t saved",
"x": 129,
"y": 781,
"author": "user_3",
"color": "purple"
},
{
"id": "note_021",
"text": "Board feels slow when there are many sticky notes",
"x": 341,
"y": 695,
"author": "user_10",
"color": "purple"
},
{
"id": "note_002",
"text": "Login flow is broken on mobile",
"x": 214,
"y": 281,
"author": "user_9",
"color": "yellow"
},
{
"id": "note_046",
"text": "Tags are missing — I need better categorization",
"x": 472,
"y": 435,
"author": "user_6",
"color": "green"
},
{
"id": "note_018",
"text": "No way to schedule recurring exports for stakeholders",
"x": 661,
"y": 229,
"author": "user_9",
"color": "blue"
},
{
"id": "note_034",
"text": "Too many notification emails for minor edits",
"x": 769,
"y": 739,
"author": "user_9",
"color": "yellow"
},
{
"id": "note_003",
"text": "When I enter my username and password I get an unknown error",
"x": 189,
"y": 193,
"author": "user_2",
"color": "yellow"
},
{
"id": "note_008",
"text": "Cannot change my password — save button does nothing",
"x": 244,
"y": 188,
"author": "user_2",
"color": "orange"
},
{
"id": "note_015",
"text": "Sharing link permissions are confusing",
"x": 688,
"y": 290,
"author": "user_6",
"color": "blue"
},
{
"id": "note_033",
"text": "Mentions (@) don’t notify the right people",
"x": 696,
"y": 669,
"author": "user_5",
"color": "yellow"
},
{
"id": "note_006",
"text": "Two-factor code is rejected even when correct",
"x": 162,
"y": 213,
"author": "user_1",
"color": "yellow"
},
{
"id": "note_028",
"text": "High CPU usage even when idle on a board",
"x": 190,
"y": 695,
"author": "user_8",
"color": "purple"
},
{
"id": "note_009",
"text": "Account gets locked too quickly after one failed attempt",
"x": 280,
"y": 255,
"author": "user_10",
"color": "orange"
},
{
"id": "note_029",
"text": "Offline mode loses edits when reconnecting",
"x": 338,
"y": 632,
"author": "user_1",
"color": "pink"
},
{
"id": "note_035",
"text": "No notification when someone resolves my comment",
"x": 673,
"y": 636,
"author": "user_2",
"color": "blue"
},
{
"id": "note_037",
"text": "Can’t easily hand off facilitation to another user",
"x": 816,
"y": 707,
"author": "user_2",
"color": "yellow"
},
{
"id": "note_013",
"text": "Cannot export selected area — only full board exports",
"x": 696,
"y": 205,
"author": "user_4",
"color": "green"
},
{
"id": "note_016",
"text": "Downloaded image is blurry compared to the canvas",
"x": 677,
"y": 245,
"author": "user_5",
"color": "blue"
},
{
"id": "note_043",
"text": "Can’t organize boards into folders the way I need",
"x": 504,
"y": 398,
"author": "user_4",
"color": "green"
},
{
"id": "note_022",
"text": "Canvas freezes for a few seconds when zooming",
"x": 254,
"y": 707,
"author": "user_8",
"color": "pink"
},
{
"id": "note_026",
"text": "Search is slow on boards with lots of content",
"x": 253,
"y": 658,
"author": "user_2",
"color": "pink"
},
{
"id": "note_048",
"text": "Duplicating a board loses some formatting",
"x": 382,
"y": 410,
"author": "user_10",
"color": "orange"
},
{
"id": "note_007",
"text": "I’m logged out unexpectedly after a few minutes",
"x": 185,
"y": 157,
"author": "user_3",
"color": "yellow"
},
{
"id": "note_045",
"text": "No bulk rename for multiple sticky notes",
"x": 455,
"y": 427,
"author": "user_1",
"color": "green"
},
{
"id": "note_042",
"text": "Template search results feel irrelevant",
"x": 400,
"y": 474,
"author": "user_6",
"color": "orange"
},
{
"id": "note_044",
"text": "Naming conventions aren’t enforced and things get messy",
"x": 469,
"y": 434,
"author": "user_9",
"color": "green"
},
{
"id": "note_004",
"text": "Password reset email never arrives",
"x": 207,
"y": 267,
"author": "user_9",
"color": "yellow"
},
{
"id": "note_047",
"text": "Hard to keep consistent styles across boards",
"x": 420,
"y": 426,
"author": "user_8",
"color": "green"
},
{
"id": "note_030",
"text": "I see random 'something went wrong' banners with no details",
"x": 214,
"y": 594,
"author": "user_5",
"color": "purple"
},
{
"id": "note_027",
"text": "Scrolling stutters on older laptops",
"x": 178,
"y": 586,
"author": "user_7",
"color": "pink"
}
]

131
backend/db/README.md Normal file
View File

@@ -0,0 +1,131 @@
# Database — kongruity backend
PostgreSQL database layer for the kongruity backend. All database modules live in this directory.
## Prerequisites
- PostgreSQL v14 or later
- The `DATABASE_URL` environment variable set in `backend/.env`
## Configuration
### Environment variable
The connection pool reads a single env var:
```
DATABASE_URL=postgresql://<user>:<password>@localhost:5432/kongruity
```
This is loaded via `dotenv` from `backend/.env` (git-ignored). The pool is created once in `index.js` and shared across the application.
### Connection pool (`index.js`)
| Export | Description |
|-------------|--------------------------------------------------|
| `query` | Execute a parameterized SQL statement |
| `getPool` | Return the underlying `pg.Pool` instance |
| `close` | Gracefully shut down the pool (`pool.end()`) |
| `default` | The pool itself (default export) |
## Schema
### `notes` table
Created by the migration in `migrate.js`. The DDL is idempotent (`CREATE TABLE IF NOT EXISTS`).
| Column | Type | Constraints / Defaults |
|---------------|----------------|------------------------------|
| `id` | `VARCHAR(64)` | `PRIMARY KEY` |
| `text` | `TEXT` | `NOT NULL` |
| `x` | `INTEGER` | `DEFAULT 0` |
| `y` | `INTEGER` | `DEFAULT 0` |
| `author` | `VARCHAR(128)` | |
| `color` | `VARCHAR(32)` | `DEFAULT 'yellow'` |
| `source_meta` | `JSONB` | `DEFAULT '{}'` |
| `created_at` | `TIMESTAMPTZ` | `DEFAULT NOW()` |
| `updated_at` | `TIMESTAMPTZ` | `DEFAULT NOW()` |
## npm scripts
Run these from the `backend/` directory.
### Migrate
Creates the `notes` table (safe to re-run):
```bash
npm run db:migrate
```
### Seed
Inserts 50 sample sticky notes. Uses `ON CONFLICT (id) DO NOTHING`, so re-running is safe and will not duplicate data:
```bash
npm run db:seed
```
## Data access layer (`notes.dao.js`)
| Function | Description |
|---------------------------|--------------------------------------------------------------------|
| `getAllNotes()` | Returns all notes ordered by `id` |
| `getNoteById(id)` | Returns a single note by primary key, or `null` if not found |
| `createNote(note)` | Inserts one note and returns the created row |
| `createNotes(notes)` | Bulk-inserts an array of notes in a single query and returns rows |
All functions return plain objects with columns: `id`, `text`, `x`, `y`, `author`, `color`.
## File overview
```
db/
├── index.js # Connection pool and query helper
├── migrate.js # Table creation (run via npm run db:migrate)
├── notes.dao.js # Data access functions for the notes table
├── seed.js # Sample data seeder (run via npm run db:seed)
└── README.md # This file
```
## Useful psql commands
Connect to the database:
```bash
psql kongruity
```
Quick checks:
```sql
-- Row count
SELECT count(*) FROM notes;
-- Preview data
SELECT id, text, color FROM notes LIMIT 10;
-- Full schema info
\d notes
-- Drop and re-seed (destructive)
TRUNCATE notes;
```
Then re-seed:
```bash
npm run db:seed
```
## Resetting the database
To start completely fresh:
```bash
dropdb kongruity
createdb kongruity
cd backend
npm run db:migrate
npm run db:seed
```

View File

@@ -0,0 +1,50 @@
{"id":"note_001","text":"Login flow feels confusing","x":193,"y":191,"author":"user_5","color":"yellow"}
{"id":"note_002","text":"Login flow is broken on mobile","x":214,"y":281,"author":"user_9","color":"yellow"}
{"id":"note_003","text":"When I enter my username and password I get an unknown error","x":189,"y":193,"author":"user_2","color":"yellow"}
{"id":"note_004","text":"Password reset email never arrives","x":207,"y":267,"author":"user_9","color":"yellow"}
{"id":"note_005","text":"SSO login loops back to the sign-in page","x":184,"y":124,"author":"user_7","color":"yellow"}
{"id":"note_006","text":"Two-factor code is rejected even when correct","x":162,"y":213,"author":"user_1","color":"yellow"}
{"id":"note_007","text":"I'm logged out unexpectedly after a few minutes","x":185,"y":157,"author":"user_3","color":"yellow"}
{"id":"note_008","text":"Cannot change my password — save button does nothing","x":244,"y":188,"author":"user_2","color":"orange"}
{"id":"note_009","text":"Account gets locked too quickly after one failed attempt","x":280,"y":255,"author":"user_10","color":"orange"}
{"id":"note_010","text":"OAuth consent screen appears every time I sign in","x":228,"y":124,"author":"user_9","color":"yellow"}
{"id":"note_011","text":"Need better export options (PDF quality is too low)","x":748,"y":212,"author":"user_9","color":"green"}
{"id":"note_012","text":"Exported PDF cuts off content near the edges","x":733,"y":159,"author":"user_6","color":"blue"}
{"id":"note_013","text":"Cannot export selected area — only full board exports","x":696,"y":205,"author":"user_4","color":"green"}
{"id":"note_014","text":"Export takes too long and sometimes never finishes","x":798,"y":211,"author":"user_2","color":"green"}
{"id":"note_015","text":"Sharing link permissions are confusing","x":688,"y":290,"author":"user_6","color":"blue"}
{"id":"note_016","text":"Downloaded image is blurry compared to the canvas","x":677,"y":245,"author":"user_5","color":"blue"}
{"id":"note_017","text":"Exported file names are inconsistent and hard to track","x":676,"y":201,"author":"user_4","color":"blue"}
{"id":"note_018","text":"No way to schedule recurring exports for stakeholders","x":661,"y":229,"author":"user_9","color":"blue"}
{"id":"note_019","text":"Embedded exports don't update when the board changes","x":662,"y":132,"author":"user_1","color":"blue"}
{"id":"note_020","text":"Can't export comments and reactions with the content","x":739,"y":139,"author":"user_7","color":"green"}
{"id":"note_021","text":"Board feels slow when there are many sticky notes","x":341,"y":695,"author":"user_10","color":"purple"}
{"id":"note_022","text":"Canvas freezes for a few seconds when zooming","x":254,"y":707,"author":"user_8","color":"pink"}
{"id":"note_023","text":"Undo/redo sometimes lags and applies late","x":236,"y":687,"author":"user_9","color":"purple"}
{"id":"note_024","text":"App crashes when opening a large board","x":239,"y":597,"author":"user_10","color":"purple"}
{"id":"note_025","text":"Saving indicator spins but changes aren't saved","x":129,"y":781,"author":"user_3","color":"purple"}
{"id":"note_026","text":"Search is slow on boards with lots of content","x":253,"y":658,"author":"user_2","color":"pink"}
{"id":"note_027","text":"Scrolling stutters on older laptops","x":178,"y":586,"author":"user_7","color":"pink"}
{"id":"note_028","text":"High CPU usage even when idle on a board","x":190,"y":695,"author":"user_8","color":"purple"}
{"id":"note_029","text":"Offline mode loses edits when reconnecting","x":338,"y":632,"author":"user_1","color":"pink"}
{"id":"note_030","text":"I see random 'something went wrong' banners with no details","x":214,"y":594,"author":"user_5","color":"purple"}
{"id":"note_031","text":"Hard to tell who is editing what in real time","x":761,"y":704,"author":"user_8","color":"yellow"}
{"id":"note_032","text":"Comments get lost — no clear thread view","x":818,"y":641,"author":"user_5","color":"yellow"}
{"id":"note_033","text":"Mentions (@) don't notify the right people","x":696,"y":669,"author":"user_5","color":"yellow"}
{"id":"note_034","text":"Too many notification emails for minor edits","x":769,"y":739,"author":"user_9","color":"yellow"}
{"id":"note_035","text":"No notification when someone resolves my comment","x":673,"y":636,"author":"user_2","color":"blue"}
{"id":"note_036","text":"Cursor presence is distracting and overlaps content","x":788,"y":605,"author":"user_5","color":"yellow"}
{"id":"note_037","text":"Can't easily hand off facilitation to another user","x":816,"y":707,"author":"user_2","color":"yellow"}
{"id":"note_038","text":"Live follow mode frequently breaks","x":710,"y":579,"author":"user_9","color":"yellow"}
{"id":"note_039","text":"Guest collaborators can't see updates without refreshing","x":759,"y":711,"author":"user_9","color":"yellow"}
{"id":"note_040","text":"Activity feed lacks context about what changed","x":710,"y":771,"author":"user_7","color":"yellow"}
{"id":"note_041","text":"Hard to find the right template quickly","x":536,"y":394,"author":"user_4","color":"orange"}
{"id":"note_042","text":"Template search results feel irrelevant","x":400,"y":474,"author":"user_6","color":"orange"}
{"id":"note_043","text":"Can't organize boards into folders the way I need","x":504,"y":398,"author":"user_4","color":"green"}
{"id":"note_044","text":"Naming conventions aren't enforced and things get messy","x":469,"y":434,"author":"user_9","color":"green"}
{"id":"note_045","text":"No bulk rename for multiple sticky notes","x":455,"y":427,"author":"user_1","color":"green"}
{"id":"note_046","text":"Tags are missing — I need better categorization","x":472,"y":435,"author":"user_6","color":"green"}
{"id":"note_047","text":"Hard to keep consistent styles across boards","x":420,"y":426,"author":"user_8","color":"green"}
{"id":"note_048","text":"Duplicating a board loses some formatting","x":382,"y":410,"author":"user_10","color":"orange"}
{"id":"note_049","text":"Need better controls for aligning and distributing notes","x":462,"y":487,"author":"user_7","color":"green"}
{"id":"note_050","text":"Can't lock sections to prevent accidental edits during workshops","x":521,"y":471,"author":"user_6","color":"orange"}

17
backend/db/index.ts Normal file
View File

@@ -0,0 +1,17 @@
import pg from 'pg';
import type { QueryResult, QueryResultRow } from 'pg';
const pool = new pg.Pool({
connectionString: process.env.DATABASE_URL,
});
export const query = <R extends QueryResultRow = QueryResultRow>(
text: string,
params?: unknown[]
): Promise<QueryResult<R>> => pool.query<R>(text, params);
export const getPool = (): pg.Pool => pool;
export const close = (): Promise<void> => pool.end();
export default pool;

View File

@@ -0,0 +1,64 @@
import { query } from './index.js';
import type { ProviderSlug } from '../types/integration.js';
/**
* Returns the new row id, or null when this delivery was already recorded.
* Providers deliver at least once, so the unique constraint firing is the
* expected path for a redelivery rather than an error.
*/
export const recordDelivery = async (input: {
provider: ProviderSlug;
externalId: string;
integrationId?: number;
}): Promise<number | null> => {
const { rows } = await query<{ id: number }>(
`INSERT INTO ingest_events (provider, external_id, integration_id)
VALUES ($1, $2, $3)
ON CONFLICT (provider, external_id) DO NOTHING
RETURNING id`,
[input.provider, input.externalId, input.integrationId ?? null]
);
return rows[0]?.id ?? null;
};
export const markProcessing = async (id: number): Promise<void> => {
await query(
`UPDATE ingest_events
SET status = 'processing', attempts = attempts + 1, updated_at = NOW()
WHERE id = $1`,
[id]
);
};
export const markDone = async (id: number): Promise<void> => {
await query(
`UPDATE ingest_events
SET status = 'done', last_error = NULL, updated_at = NOW()
WHERE id = $1`,
[id]
);
};
export const markFailed = async (id: number, error: string): Promise<void> => {
await query(
`UPDATE ingest_events
SET status = 'failed', last_error = $2, updated_at = NOW()
WHERE id = $1`,
[id, error.slice(0, 2000)]
);
};
/**
* The queue lives in process memory, so a restart strands anything that was
* mid-flight. Returns rows to 'pending' so they can be picked up again.
*/
export const resetStaleProcessing = async (olderThanMs: number): Promise<number> => {
const { rowCount } = await query(
`UPDATE ingest_events
SET status = 'pending', updated_at = NOW()
WHERE status = 'processing'
AND updated_at < NOW() - ($1 || ' milliseconds')::interval`,
[String(olderThanMs)]
);
return rowCount ?? 0;
};

View File

@@ -0,0 +1,132 @@
import { query } from './index.js';
import { decryptSecret, encryptSecret } from '../lib/crypto.js';
import type { IntegrationRow, ProviderSlug } from '../types/integration.js';
type IntegrationRecord = {
id: number;
provider: ProviderSlug;
external_workspace_id: string | null;
display_name: string | null;
api_key_hash: string | null;
access_token_ciphertext: string | null;
refresh_token_ciphertext: string | null;
signing_secret_ciphertext: string | null;
token_expires_at: Date | null;
scopes: string[] | null;
};
const PUBLIC_COLUMNS = `
id, provider, external_workspace_id, display_name, token_expires_at, scopes
`;
/**
* Ciphertext columns are dropped here rather than in the query so that every
* caller path converges on a value that structurally cannot carry a secret.
*/
const toPublicRow = (record: IntegrationRecord): IntegrationRow => ({
id: record.id,
provider: record.provider,
externalWorkspaceId: record.external_workspace_id,
displayName: record.display_name,
scopes: record.scopes ?? [],
tokenExpiresAt: record.token_expires_at,
});
export const findByApiKeyHash = async (
hash: string
): Promise<IntegrationRow | null> => {
const { rows } = await query<IntegrationRecord>(
`SELECT ${PUBLIC_COLUMNS} FROM integrations WHERE api_key_hash = $1`,
[hash]
);
return rows[0] ? toPublicRow(rows[0]) : null;
};
export const findByProviderWorkspace = async (
provider: ProviderSlug,
externalWorkspaceId: string
): Promise<IntegrationRow | null> => {
const { rows } = await query<IntegrationRecord>(
`SELECT ${PUBLIC_COLUMNS} FROM integrations
WHERE provider = $1 AND external_workspace_id = $2`,
[provider, externalWorkspaceId]
);
return rows[0] ? toPublicRow(rows[0]) : null;
};
export const upsertInstall = async (input: {
provider: ProviderSlug;
externalWorkspaceId: string;
displayName?: string;
apiKeyHash?: string;
signingSecret?: string;
scopes?: string[];
}): Promise<IntegrationRow> => {
const { rows } = await query<IntegrationRecord>(
`INSERT INTO integrations
(provider, external_workspace_id, display_name, api_key_hash,
signing_secret_ciphertext, scopes)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (provider, external_workspace_id) DO UPDATE SET
display_name = EXCLUDED.display_name,
api_key_hash = COALESCE(EXCLUDED.api_key_hash, integrations.api_key_hash),
signing_secret_ciphertext = COALESCE(
EXCLUDED.signing_secret_ciphertext, integrations.signing_secret_ciphertext
),
scopes = EXCLUDED.scopes,
updated_at = NOW()
RETURNING ${PUBLIC_COLUMNS}`,
[
input.provider,
input.externalWorkspaceId,
input.displayName ?? null,
input.apiKeyHash ?? null,
input.signingSecret ? encryptSecret(input.signingSecret) : null,
input.scopes ?? [],
]
);
return toPublicRow(rows[0]);
};
export const updateTokens = async (input: {
id: number;
accessToken: string;
refreshToken?: string;
expiresAt?: Date;
}): Promise<void> => {
await query(
`UPDATE integrations SET
access_token_ciphertext = $2,
refresh_token_ciphertext = COALESCE($3, refresh_token_ciphertext),
token_expires_at = $4,
updated_at = NOW()
WHERE id = $1`,
[
input.id,
encryptSecret(input.accessToken),
input.refreshToken ? encryptSecret(input.refreshToken) : null,
input.expiresAt ?? null,
]
);
};
const readSecret = async (
id: number,
column: 'access_token_ciphertext' | 'refresh_token_ciphertext' | 'signing_secret_ciphertext'
): Promise<string | null> => {
const { rows } = await query<Record<string, string | null>>(
`SELECT ${column} AS value FROM integrations WHERE id = $1`,
[id]
);
const value = rows[0]?.value;
return value ? decryptSecret(value) : null;
};
export const getAccessToken = (id: number): Promise<string | null> =>
readSecret(id, 'access_token_ciphertext');
export const getRefreshToken = (id: number): Promise<string | null> =>
readSecret(id, 'refresh_token_ciphertext');
export const getSigningSecret = (id: number): Promise<string | null> =>
readSecret(id, 'signing_secret_ciphertext');

67
backend/db/migrate.ts Normal file
View File

@@ -0,0 +1,67 @@
import 'dotenv/config';
import { query, close } from './index.js';
const up = `
CREATE TABLE IF NOT EXISTS notes (
id VARCHAR(64) PRIMARY KEY,
text TEXT NOT NULL,
x INTEGER DEFAULT 0,
y INTEGER DEFAULT 0,
author VARCHAR(128),
color VARCHAR(32) DEFAULT 'yellow',
source_meta JSONB DEFAULT '{}',
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS integrations (
id SERIAL PRIMARY KEY,
provider VARCHAR(32) NOT NULL,
external_workspace_id VARCHAR(128),
display_name VARCHAR(255),
api_key_hash CHAR(64),
access_token_ciphertext TEXT,
refresh_token_ciphertext TEXT,
signing_secret_ciphertext TEXT,
token_expires_at TIMESTAMPTZ,
scopes TEXT[] DEFAULT '{}',
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE (provider, external_workspace_id)
);
CREATE INDEX IF NOT EXISTS integrations_api_key_hash_idx
ON integrations (api_key_hash);
-- The unique constraint is the deduplication mechanism: providers deliver
-- at least once, so a redelivery must collide rather than insert.
CREATE TABLE IF NOT EXISTS ingest_events (
id SERIAL PRIMARY KEY,
provider VARCHAR(32) NOT NULL,
external_id VARCHAR(255) NOT NULL,
integration_id INTEGER REFERENCES integrations(id) ON DELETE SET NULL,
status VARCHAR(16) NOT NULL DEFAULT 'pending',
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
received_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE (provider, external_id)
);
CREATE INDEX IF NOT EXISTS ingest_events_status_idx
ON ingest_events (status, updated_at);
`;
const run = async (): Promise<void> => {
try {
await query(up);
console.log('Migration complete — notes, integrations, ingest_events ready.');
} catch (err) {
console.error('Migration failed:', err instanceof Error ? err.message : err);
process.exit(1);
} finally {
await close();
}
};
run();

125
backend/db/notes.dao.ts Normal file
View File

@@ -0,0 +1,125 @@
import QueryStream from 'pg-query-stream';
import { pipeline } from 'node:stream/promises';
import { Readable } from 'node:stream';
import { query, getPool } from './index.js';
import { batch } from '../lib/streams.js';
import type { Note, NoteInput } from '../types/domain.js';
const SELECT_NOTES = 'SELECT id, text, x, y, author, color FROM notes ORDER BY id';
// Postgres caps a statement at 65535 bind parameters; seven columns per note
// leaves 9362 as the hard ceiling.
const INSERT_BATCH_SIZE = 1000;
const INSERT_COLUMNS = 7;
export const getAllNotes = async (): Promise<Note[]> => {
const { rows } = await query<Note>(SELECT_NOTES);
return rows;
};
/**
* Streams every note as an object-mode Readable. The pooled client is released on end, error, or
* destruction by consumer.
*/
export const streamAllNotes = async (): Promise<Readable> => {
const client = await getPool().connect();
let released = false;
const release = () => {
if (released) return;
released = true;
client.release();
};
try {
const rows = client.query(new QueryStream(SELECT_NOTES)) as unknown as Readable;
rows.once('end', release);
rows.once('error', release);
rows.once('close', release);
return rows;
} catch (err) {
release();
throw err;
}
};
export const getNoteById = async (id: string): Promise<Note | null> => {
const { rows } = await query<Note>(
'SELECT id, text, x, y, author, color FROM notes WHERE id = $1',
[id]
);
return rows[0] || null;
};
export const createNote = async (note: NoteInput): Promise<Note> => {
const { rows } = await query<Note>(
`INSERT INTO notes (id, text, x, y, author, color, source_meta)
VALUES ($1, $2, $3, $4, $5, $6, $7)
RETURNING id, text, x, y, author, color`,
[
note.id,
note.text,
note.x ?? 0,
note.y ?? 0,
note.author,
note.color ?? 'yellow',
JSON.stringify(note.sourceMeta ?? {}),
]
);
return rows[0];
};
const insertNoteBatch = async (notes: NoteInput[]): Promise<Note[]> => {
const values: unknown[] = [];
const placeholders: string[] = [];
notes.forEach((note, i) => {
const offset = i * INSERT_COLUMNS;
const slots = Array.from(
{ length: INSERT_COLUMNS },
(_, col) => `$${offset + col + 1}`
);
placeholders.push(`(${slots.join(', ')})`);
values.push(
note.id,
note.text,
note.x ?? 0,
note.y ?? 0,
note.author,
note.color ?? 'yellow',
JSON.stringify(note.sourceMeta ?? {})
);
});
// Redelivered webhooks can carry a note that already landed; skipping the
// conflict keeps ingestion idempotent at the row level as well.
const { rows } = await query<Note>(
`INSERT INTO notes (id, text, x, y, author, color, source_meta)
VALUES ${placeholders.join(', ')}
ON CONFLICT (id) DO NOTHING
RETURNING id, text, x, y, author, color`,
values
);
return rows;
};
export const createNotes = async (notes: NoteInput[]): Promise<Note[]> => {
if (!notes || notes.length === 0) {
return [];
}
const inserted: Note[] = [];
await pipeline(
Readable.from(notes, { objectMode: true }),
batch<NoteInput>(INSERT_BATCH_SIZE),
async (batches: AsyncIterable<NoteInput[]>) => {
for await (const chunk of batches) {
inserted.push(...await insertNoteBatch(chunk));
}
}
);
return inserted;
};

82
backend/db/seed.ts Normal file
View File

@@ -0,0 +1,82 @@
import 'dotenv/config';
import split2 from 'split2';
import { from as copyFrom } from 'pg-copy-streams';
import { createReadStream } from 'node:fs';
import { Transform, type TransformCallback } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { fileURLToPath } from 'node:url';
import { getPool, close } from './index.js';
const FIXTURE = fileURLToPath(new URL('./fixtures/notes.jsonl', import.meta.url));
const COLUMNS = ['id', 'text', 'x', 'y', 'author', 'color'] as const;
type Column = typeof COLUMNS[number];
const DEFAULTS: Partial<Record<Column, unknown>> = { x: 0, y: 0, color: 'yellow' };
const csvField = (value: unknown): string => {
if (value === null || value === undefined) return '';
return `"${String(value).replaceAll('"', '""')}"`;
};
const toCsvRows = (): Transform => new Transform({
writableObjectMode: true,
transform(line: string, _encoding: BufferEncoding, callback: TransformCallback) {
if (line.trim().length === 0) {
callback();
return;
}
let note: Record<string, unknown>;
try {
note = JSON.parse(line) as Record<string, unknown>;
} catch {
callback(new Error(`Fixture contains a malformed JSON line: ${line.slice(0, 80)}`));
return;
}
const row = COLUMNS.map((col) => csvField(note[col] ?? DEFAULTS[col]));
callback(null, `${row.join(',')}\n`);
},
});
const run = async (): Promise<void> => {
const client = await getPool().connect();
try {
await client.query('BEGIN');
// COPY has no ON CONFLICT, so land the fixture in a temp table first and
// let a single INSERT ... SELECT apply the existing idempotency.
await client.query(
'CREATE TEMP TABLE notes_import (LIKE notes INCLUDING DEFAULTS) ON COMMIT DROP'
);
const copy = client.query(
copyFrom(`COPY notes_import (${COLUMNS.join(', ')}) FROM STDIN WITH (FORMAT csv)`)
);
await pipeline(createReadStream(FIXTURE), split2(), toCsvRows(), copy);
const { rowCount: staged } = await client.query('SELECT 1 FROM notes_import');
const { rowCount: inserted } = await client.query(
`INSERT INTO notes (${COLUMNS.join(', ')})
SELECT ${COLUMNS.join(', ')} FROM notes_import
ON CONFLICT (id) DO NOTHING`
);
await client.query('COMMIT');
console.log(`Seed complete — ${inserted} notes inserted (${(staged ?? 0) - (inserted ?? 0)} already existed).`);
} catch (err) {
await client.query('ROLLBACK').catch(() => {});
console.error('Seed failed:', err instanceof Error ? err.message : err);
process.exitCode = 1;
} finally {
client.release();
await close();
}
};
run();

77
backend/lib/crypto.ts Normal file
View File

@@ -0,0 +1,77 @@
import {
createCipheriv,
createDecipheriv,
createHash,
randomBytes,
timingSafeEqual,
} from 'node:crypto';
const ALGORITHM = 'aes-256-gcm';
const IV_BYTES = 12;
const KEY_BYTES = 32;
const loadKey = (): Buffer => {
const raw = process.env.TOKEN_ENCRYPTION_KEY;
if (!raw) {
throw new Error('TOKEN_ENCRYPTION_KEY is not set');
}
const key = Buffer.from(raw, 'base64');
if (key.length !== KEY_BYTES) {
throw new Error(
`TOKEN_ENCRYPTION_KEY must decode to ${KEY_BYTES} bytes, got ${key.length}`
);
}
return key;
};
/** Serialized as base64(iv):base64(authTag):base64(ciphertext). */
export const encryptSecret = (plaintext: string): string => {
const iv = randomBytes(IV_BYTES);
const cipher = createCipheriv(ALGORITHM, loadKey(), iv);
const payload = Buffer.concat([
cipher.update(plaintext, 'utf8'),
cipher.final(),
]);
return [
iv.toString('base64'),
cipher.getAuthTag().toString('base64'),
payload.toString('base64'),
].join(':');
};
export const decryptSecret = (ciphertext: string): string => {
const parts = ciphertext.split(':');
if (parts.length !== 3) {
throw new Error('Ciphertext is not in the expected iv:tag:payload form');
}
const [iv, tag, payload] = parts;
const decipher = createDecipheriv(
ALGORITHM,
loadKey(),
Buffer.from(iv, 'base64')
);
decipher.setAuthTag(Buffer.from(tag, 'base64'));
return Buffer.concat([
decipher.update(Buffer.from(payload, 'base64')),
decipher.final(),
]).toString('utf8');
};
export const hashApiKey = (key: string): string =>
createHash('sha256').update(key, 'utf8').digest('hex');
/**
* Constant-time string comparison. Both sides are hashed first so that
* unequal lengths cannot short-circuit the comparison or throw.
*/
export const safeEquals = (a: string, b: string): boolean => {
const digestA = createHash('sha256').update(a, 'utf8').digest();
const digestB = createHash('sha256').update(b, 'utf8').digest();
return timingSafeEqual(digestA, digestB);
};

234
backend/lib/httpClient.ts Normal file
View File

@@ -0,0 +1,234 @@
const DEFAULT_TIMEOUT_MS = 10_000;
const DEFAULT_MAX_ATTEMPTS = 3;
const DEFAULT_BASE_DELAY_MS = 250;
const DEFAULT_RETRY_AFTER_CAP_MS = 60_000;
export type HttpFailureKind = 'status' | 'timeout' | 'network' | 'aborted' | 'invalid-body';
export type RequestJsonInit = RequestInit & {
timeoutMs?: number;
maxAttempts?: number;
/** First-retry backoff, doubled per attempt. Exposed so tests need not wait on real backoff. */
baseDelayMs?: number;
/** Upper bound on a honored `Retry-After`, so a hostile header cannot park the process. */
retryAfterCapMs?: number;
};
export class HttpRequestError extends Error {
readonly url: string;
readonly status: number | undefined;
readonly attempts: number;
readonly kind: HttpFailureKind;
constructor(
url: string,
kind: HttpFailureKind,
attempts: number,
status?: number,
detail?: string
) {
const statusPart = status === undefined ? '' : ` status ${status}`;
const detailPart = detail === undefined ? '' : `: ${detail}`;
super(
`HTTP request to ${url} failed after ${attempts} attempt(s) (${kind}${statusPart})${detailPart}`
);
this.name = 'HttpRequestError';
this.url = url;
this.status = status;
this.attempts = attempts;
this.kind = kind;
}
}
export const isHttpRequestError = (err: unknown): err is HttpRequestError =>
err instanceof HttpRequestError;
/**
* Returns the delay a `Retry-After` header asks for, clamped to `capMs`, or null when the
* header is absent or unintelligible. Supports both delay-seconds and HTTP-date forms.
*/
export const parseRetryAfter = (
headerValue: string | null,
capMs: number = DEFAULT_RETRY_AFTER_CAP_MS
): number | null => {
if (headerValue === null) {
return null;
}
const raw = headerValue.trim();
if (raw === '') {
return null;
}
if (/^\d+$/.test(raw)) {
return Math.min(Number(raw) * 1000, capMs);
}
const deadline = Date.parse(raw);
if (Number.isNaN(deadline)) {
return null;
}
return Math.min(Math.max(deadline - Date.now(), 0), capMs);
};
const backoffDelay = (attempt: number, baseDelayMs: number, capMs: number): number => {
const ceiling = Math.min(baseDelayMs * 2 ** (attempt - 1), capMs);
// Half fixed, half jittered, so concurrent callers do not resynchronize on the same tick.
return Math.round(ceiling / 2 + Math.random() * (ceiling / 2));
};
/** Settles early when `signal` fires; the caller re-checks the signal before retrying. */
const sleep = (ms: number, signal: AbortSignal | undefined): Promise<void> =>
new Promise<void>((resolve) => {
if (signal?.aborted === true) {
resolve();
return;
}
let timer: ReturnType<typeof setTimeout>;
const onAbort = (): void => {
clearTimeout(timer);
resolve();
};
timer = setTimeout(() => {
signal?.removeEventListener('abort', onAbort);
resolve();
}, ms);
signal?.addEventListener('abort', onAbort, { once: true });
});
const isRetryableStatus = (status: number): boolean => status === 429 || status >= 500;
/** Frees the socket for reuse; a body we never read would otherwise stay pending. */
const discardBody = async (response: Response): Promise<void> => {
try {
await response.body?.cancel();
} catch {
// A already-consumed or errored body is irrelevant to the retry decision.
}
};
const readJson = async <T>(response: Response, url: string, attempts: number): Promise<T> => {
// 204/205 are defined as bodiless, so absence of JSON is the correct outcome, not a failure.
if (response.status === 204 || response.status === 205) {
return undefined as T;
}
let text: string;
try {
text = await response.text();
} catch (err) {
throw new HttpRequestError(
url,
'invalid-body',
attempts,
response.status,
err instanceof Error ? err.message : 'could not read response body'
);
}
try {
return JSON.parse(text) as T;
} catch {
throw new HttpRequestError(
url,
'invalid-body',
attempts,
response.status,
'response body was not valid JSON'
);
}
};
/**
* Performs a JSON request, retrying only failures a retry can plausibly fix: 429, 5xx, timeouts
* and transport errors. Any other non-2xx fails on the first attempt, since re-sending a 400 or
* 401 only spends rate limit against an answer that will not change.
*
* Resolves `undefined` for bodiless 204/205 responses; any other 2xx whose body is not valid
* JSON rejects rather than resolving `undefined` silently.
*/
export const requestJson = async <T>(url: string, init?: RequestJsonInit): Promise<T> => {
const {
timeoutMs = DEFAULT_TIMEOUT_MS,
maxAttempts = DEFAULT_MAX_ATTEMPTS,
baseDelayMs = DEFAULT_BASE_DELAY_MS,
retryAfterCapMs = DEFAULT_RETRY_AFTER_CAP_MS,
signal,
...requestInit
} = init ?? {};
if (!Number.isInteger(maxAttempts) || maxAttempts < 1) {
throw new TypeError('requestJson(init.maxAttempts) requires a positive integer');
}
const callerSignal = signal ?? undefined;
// Read through a call so narrowing never freezes this at its first observed value.
const callerAborted = (): boolean => callerSignal !== undefined && callerSignal.aborted;
let lastFailure: HttpRequestError | undefined;
for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
if (callerAborted()) {
throw new HttpRequestError(url, 'aborted', attempt - 1, undefined, 'caller aborted');
}
const timeoutSignal = AbortSignal.timeout(timeoutMs);
const attemptSignal =
callerSignal === undefined
? timeoutSignal
: AbortSignal.any([callerSignal, timeoutSignal]);
let response: Response;
try {
response = await fetch(url, { ...requestInit, signal: attemptSignal });
} catch (err) {
if (callerAborted()) {
throw new HttpRequestError(url, 'aborted', attempt, undefined, 'caller aborted');
}
const kind: HttpFailureKind = timeoutSignal.aborted ? 'timeout' : 'network';
const detail = timeoutSignal.aborted
? `attempt exceeded ${timeoutMs}ms`
: err instanceof Error
? err.message
: 'transport error';
lastFailure = new HttpRequestError(url, kind, attempt, undefined, detail);
if (attempt === maxAttempts) {
throw lastFailure;
}
await sleep(backoffDelay(attempt, baseDelayMs, retryAfterCapMs), callerSignal);
continue;
}
if (response.ok) {
return await readJson<T>(response, url, attempt);
}
await discardBody(response);
if (!isRetryableStatus(response.status)) {
throw new HttpRequestError(url, 'status', attempt, response.status);
}
lastFailure = new HttpRequestError(url, 'status', attempt, response.status);
if (attempt === maxAttempts) {
throw lastFailure;
}
const retryAfter = parseRetryAfter(response.headers.get('retry-after'), retryAfterCapMs);
await sleep(
retryAfter ?? backoffDelay(attempt, baseDelayMs, retryAfterCapMs),
callerSignal
);
}
// Unreachable while maxAttempts >= 1; the loop either returns or throws.
throw lastFailure ?? new HttpRequestError(url, 'network', maxAttempts);
};

91
backend/lib/queue.ts Normal file
View File

@@ -0,0 +1,91 @@
export type Job = () => Promise<void>;
type QueueEntry = { name: string; job: Job };
type QueueSettings = { maxAttempts: number; baseDelayMs: number };
const settings: QueueSettings = { maxAttempts: 3, baseDelayMs: 100 };
const pending: QueueEntry[] = [];
const idleWaiters: Array<() => void> = [];
let active = false;
const sleep = (ms: number): Promise<void> =>
new Promise((resolve) => {
setTimeout(resolve, ms);
});
// Delay for attempt n is drawn from [base * 2^(n-1), base * 2^n), so successive
// waits always grow while jitter keeps retries from synchronizing across jobs.
const backoffDelay = (attempt: number): number => {
const window = settings.baseDelayMs * 2 ** (attempt - 1);
return window + Math.random() * window;
};
const runEntry = async (entry: QueueEntry): Promise<void> => {
for (let attempt = 1; ; attempt += 1) {
try {
await entry.job();
return;
} catch (err) {
if (attempt >= settings.maxAttempts) {
console.error(
`[queue] job "${entry.name}" abandoned after ${attempt} attempt(s)`,
err
);
return;
}
await sleep(backoffDelay(attempt));
}
}
};
const runLoop = async (): Promise<void> => {
try {
for (;;) {
const entry = pending.shift();
if (!entry) return;
await runEntry(entry);
}
} finally {
active = false;
for (const resolve of idleWaiters.splice(0)) resolve();
}
};
export const enqueue = (name: string, job: Job): void => {
pending.push({ name, job });
if (active) return;
active = true;
// Deferred to a microtask so enqueue() returns to its caller — typically a
// request handler that has already responded — before any job body runs.
void Promise.resolve()
.then(runLoop)
.catch((err: unknown) => {
active = false;
console.error('[queue] queue loop stopped unexpectedly', err);
});
};
export const size = (): number => pending.length;
export const drain = (): Promise<void> => {
if (!active && pending.length === 0) return Promise.resolve();
return new Promise<void>((resolve) => {
idleWaiters.push(resolve);
});
};
export const configureQueue = (opts: {
maxAttempts?: number;
baseDelayMs?: number;
}): void => {
if (opts.maxAttempts !== undefined) {
settings.maxAttempts = Math.max(1, Math.floor(opts.maxAttempts));
}
if (opts.baseDelayMs !== undefined) {
settings.baseDelayMs = Math.max(0, opts.baseDelayMs);
}
};

81
backend/lib/signatures.ts Normal file
View File

@@ -0,0 +1,81 @@
import { createHmac, timingSafeEqual } from 'node:crypto';
import type { IncomingHttpHeaders } from 'node:http';
import { getProvider } from '../config/providers.js';
import type { ProviderSlug, SignatureResult } from '../types/integration.js';
const DEFAULT_TOLERANCE_SECONDS = 300;
const OK: SignatureResult = { ok: true };
const fail = (reason: Exclude<SignatureResult, { ok: true }>['reason']): SignatureResult => ({
ok: false,
reason,
});
const header = (headers: IncomingHttpHeaders, name: string): string | undefined => {
const value = headers[name.toLowerCase()];
if (Array.isArray(value)) return value[0];
return value;
};
const hmacHex = (secret: string, payload: string | Buffer): string =>
createHmac('sha256', secret).update(payload).digest('hex');
const constantTimeEquals = (a: string, b: string): boolean => {
const bufA = Buffer.from(a, 'utf8');
const bufB = Buffer.from(b, 'utf8');
if (bufA.length !== bufB.length) return false;
return timingSafeEqual(bufA, bufB);
};
const withinTolerance = (timestamp: string, toleranceSeconds: number): boolean => {
const sent = Number(timestamp);
if (!Number.isFinite(sent)) return false;
const nowSeconds = Math.floor(Date.now() / 1000);
return Math.abs(nowSeconds - sent) <= toleranceSeconds;
};
/**
* Never throws. A malformed header from an anonymous caller must be an
* ordinary negative result, not an exception reachable from the edge.
*/
export const verifySignature = (input: {
provider: ProviderSlug;
rawBody: Buffer;
headers: IncomingHttpHeaders;
secret: string;
toleranceSeconds?: number;
}): SignatureResult => {
const config = getProvider(input.provider);
const tolerance = input.toleranceSeconds ?? DEFAULT_TOLERANCE_SECONDS;
if (config.signatureScheme === 'none') return OK;
const presented = header(input.headers, config.signatureHeader);
if (!presented) return fail('missing');
if (config.signatureScheme === 'slack-v0') {
const timestamp = config.timestampHeader
? header(input.headers, config.timestampHeader)
: undefined;
if (!timestamp) return fail('missing');
if (!withinTolerance(timestamp, tolerance)) return fail('stale');
if (!presented.startsWith('v0=')) return fail('malformed');
const base = `v0:${timestamp}:${input.rawBody.toString('utf8')}`;
const expected = `v0=${hmacHex(input.secret, base)}`;
return constantTimeEquals(presented, expected) ? OK : fail('mismatch');
}
if (config.signatureScheme === 'github-sha256') {
if (!presented.startsWith('sha256=')) return fail('malformed');
const expected = `sha256=${hmacHex(input.secret, input.rawBody)}`;
return constantTimeEquals(presented, expected) ? OK : fail('mismatch');
}
// linear-sha256: bare lowercase hex digest of the raw body.
if (!/^[0-9a-f]{64}$/.test(presented)) return fail('malformed');
const expected = hmacHex(input.secret, input.rawBody);
return constantTimeEquals(presented, expected) ? OK : fail('mismatch');
};

64
backend/lib/streams.ts Normal file
View File

@@ -0,0 +1,64 @@
import { Transform, type TransformCallback } from 'node:stream';
/**
* Backpressure on readable side is limits how many batches are in flight.
*
* @param size - maximum items per emitted batch
*/
export const batch = <T>(size: number): Transform => {
if (!Number.isInteger(size) || size < 1) {
throw new TypeError('batch(size) requires a positive integer size');
}
let pending: T[] = [];
return new Transform({
objectMode: true,
transform(item: T, _encoding: BufferEncoding, callback: TransformCallback) {
pending.push(item);
if (pending.length < size) {
callback();
return;
}
const full = pending;
pending = [];
callback(null, full);
},
flush(callback: TransformCallback) {
if (pending.length === 0) {
callback();
return;
}
const remainder = pending;
pending = [];
callback(null, remainder);
},
});
};
export const jsonArray = (): Transform => {
let wroteFirst = false;
return new Transform({
writableObjectMode: true,
transform(item: unknown, _encoding: BufferEncoding, callback: TransformCallback) {
let serialized: string;
try {
serialized = JSON.stringify(item);
} catch (err) {
callback(err as Error);
return;
}
const prefix = wroteFirst ? ',' : '[';
wroteFirst = true;
callback(null, prefix + serialized);
},
flush(callback: TransformCallback) {
callback(null, wroteFirst ? ']' : '[]');
},
});
};

View File

@@ -0,0 +1,41 @@
import type { NextFunction, Request, RequestHandler, Response } from 'express';
import { hashApiKey } from '../lib/crypto.js';
import { findByApiKeyHash } from '../db/integrations.dao.js';
import type { IntegrationRow } from '../types/integration.js';
declare module 'express-serve-static-core' {
interface Request {
integration?: IntegrationRow;
}
}
const BEARER = /^Bearer (.+)$/;
/**
* Every rejection returns the same body. Distinguishing "no such key" from
* "wrong key" would let a caller enumerate valid keys.
*/
export const requireApiKey: RequestHandler = async (
req: Request,
res: Response,
next: NextFunction
) => {
try {
const match = BEARER.exec(req.get('authorization') ?? '');
if (!match) {
res.status(401).json({ error: 'Unauthorized' });
return;
}
const integration = await findByApiKeyHash(hashApiKey(match[1]));
if (!integration) {
res.status(401).json({ error: 'Unauthorized' });
return;
}
req.integration = integration;
next();
} catch (err) {
next(err);
}
};

1796
backend/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -3,21 +3,38 @@
"version": "0.1.0",
"description": "kongruity AI stick note clustering feature - backend for ProjectPilot",
"type": "module",
"main": "server.js",
"main": "dist/server.js",
"scripts": {
"start": "node server.js",
"dev": "nodemon server.js",
"test": "vitest run"
"build": "tsc -p tsconfig.build.json",
"start": "node dist/server.js",
"dev": "tsx watch server.ts",
"type-check": "tsc --noEmit",
"test": "vitest run",
"db:migrate": "tsx db/migrate.ts",
"db:seed": "tsx db/seed.ts"
},
"dependencies": {
"@anthropic-ai/sdk": "^0.74.0",
"cors": "^2.8.5",
"dotenv": "^16.4.7",
"express": "^4.21.2"
"express": "^4.21.2",
"pg": "^8.18.0",
"pg-copy-streams": "^7.0.0",
"pg-query-stream": "^4.16.0",
"split2": "^4.2.0",
"voyageai": "^0.1.0"
},
"devDependencies": {
"nodemon": "^3.1.9",
"@types/cors": "^2.8.19",
"@types/express": "^4.17.25",
"@types/node": "^26.1.2",
"@types/pg": "^8.20.3",
"@types/pg-copy-streams": "^1.2.5",
"@types/split2": "^4.2.3",
"@types/supertest": "^7.2.1",
"supertest": "^7.2.2",
"tsx": "^4.23.1",
"typescript": "^7.0.2",
"vitest": "^4.0.18"
}
}

View File

@@ -0,0 +1,48 @@
import { Router, type Request, type Response } from 'express';
import { requireApiKey } from '../middleware/apiKey.js';
import { createNotes } from '../db/notes.dao.js';
import { isNoteInputArray } from '../config/normalizers.js';
import type { NoteInput } from '../types/domain.js';
const router = Router();
const MAX_BATCH = 5000;
type IngestBody = { notes: NoteInput[] };
const isIngestBody = (value: unknown): value is IngestBody => {
if (typeof value !== 'object' || value === null) return false;
const body = value as Record<string, unknown>;
return isNoteInputArray(body.notes);
};
router.post('/', requireApiKey, async (req: Request, res: Response) => {
try {
if (!isIngestBody(req.body)) {
res.status(400).json({
error: 'Body must be { notes: [{ id, text, author, ... }] }',
});
return;
}
const { notes } = req.body;
if (notes.length === 0) {
res.status(400).json({ error: 'notes must not be empty' });
return;
}
if (notes.length > MAX_BATCH) {
res.status(400).json({ error: `notes exceeds the ${MAX_BATCH} per-request limit` });
return;
}
const inserted = await createNotes(notes);
res.status(201).json({ inserted: inserted.length, notes: inserted });
} catch (err) {
console.error(`Ingest failed: ${err}`);
res.status(500).json({ error: 'Ingest failed' });
}
});
export default router;

View File

@@ -1,34 +0,0 @@
import { Router } from 'express';
import { readFile } from 'fs/promises';
import { clusterNotes } from '../services/clustering.service.js';
const router = Router();
const DATA_PATH = '../data/notes.json'
const loadNotes = async () => {
const raw = await readFile(DATA_PATH, 'utf-8');
return JSON.parse(raw);
};
router.get('/', async (req, res) => {
try {
const notes = await loadNotes();
res.json(notes);
} catch (err) {
console.error(`Error loading notes: ${err}`)
res.status(500).json({ error: 'Failed to load notes' });
}
});
router.post('/cluster', async (req, res) => {
try {
const notes = await loadNotes();
const clusters = await clusterNotes(notes);
res.json(clusters);
} catch (err) {
console.error(`Clustering failed: ${err}` )
res.status(500).json({ error: `Clustering failed: ${err}` });
}
});
export default router;

View File

@@ -0,0 +1,41 @@
import { Router, type Request, type Response } from 'express';
import { pipeline } from 'node:stream/promises';
import { getAllNotes, streamAllNotes } from '../db/notes.dao.js';
import { clusterNotes } from '../services/clustering.service.js';
import { jsonArray } from '../lib/streams.js';
const router = Router();
router.get('/', async (_req: Request, res: Response) => {
try {
const rows = await streamAllNotes();
res.type('application/json');
await pipeline(rows, jsonArray(), res);
} catch (err) {
console.error(`Error loading notes: ${err}`);
if (res.headersSent) {
res.destroy(err as Error);
return;
}
res.status(500).json({ error: 'Failed to load notes' });
}
});
router.post('/cluster', async (_req: Request, res: Response) => {
const controller = new AbortController();
res.on('close', () => {
if (!res.writableEnded) controller.abort();
});
try {
const notes = await getAllNotes();
const result = await clusterNotes(notes, { signal: controller.signal });
res.json(result);
} catch (err) {
if (controller.signal.aborted) return;
console.error(`Clustering failed: ${err}`);
res.status(500).json({ error: 'Clustering failed' });
}
});
export default router;

View File

@@ -0,0 +1,113 @@
import { Router, type NextFunction, type Request, type Response } from 'express';
import { getProvider } from '../config/providers.js';
import { verifySignature } from '../lib/signatures.js';
import { enqueue } from '../lib/queue.js';
import { normalize, NormalizationError } from '../services/normalize.service.js';
import {
markDone,
markFailed,
markProcessing,
recordDelivery,
} from '../db/ingest_events.dao.js';
import { createNotes } from '../db/notes.dao.js';
import { isProviderSlug, type NormalizedDelivery } from '../types/integration.js';
const router = Router();
const parseJson = (raw: Buffer): unknown => {
if (raw.length === 0) return {};
return JSON.parse(raw.toString('utf8'));
};
router.post('/:provider', async (req: Request, res: Response, next: NextFunction) => {
try {
const slug = req.params.provider;
if (!isProviderSlug(slug)) {
res.status(404).json({ error: `Unknown provider "${slug}"` });
return;
}
const config = getProvider(slug);
const rawBody = Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0);
let payload: unknown;
try {
payload = parseJson(rawBody);
} catch {
res.status(400).json({ error: 'Body is not valid JSON' });
return;
}
const challenge = config.challenge?.(payload);
if (challenge) {
res.status(challenge.status).json(challenge.body);
return;
}
if (config.signatureScheme !== 'none') {
const secret = config.secretEnvVar ? process.env[config.secretEnvVar] : undefined;
if (!secret) {
console.error(`No signing secret configured for provider "${slug}"`);
res.status(500).json({ error: 'Provider is not configured' });
return;
}
const result = verifySignature({
provider: slug,
rawBody,
headers: req.headers,
secret,
});
if (!result.ok) {
res.status(401).json({ error: 'Invalid signature' });
return;
}
}
if (!config.normalize) {
res.status(501).json({ error: `No ingestion mapping for provider "${slug}"` });
return;
}
// Normalizing before the ack costs nothing (it is pure) and lets a malformed
// payload fail loudly here rather than silently in a background job.
let delivery: NormalizedDelivery;
try {
delivery = normalize(slug, payload);
} catch (err) {
if (!(err instanceof NormalizationError)) throw err;
res.status(400).json({ error: err.message });
return;
}
const eventId = await recordDelivery({
provider: slug,
externalId: delivery.externalId,
});
if (eventId === null) {
res.status(200).json({ duplicate: true });
return;
}
const { notes } = delivery;
enqueue(`${slug}:${delivery.externalId}`, async () => {
await markProcessing(eventId);
try {
await createNotes(notes);
await markDone(eventId);
} catch (err) {
await markFailed(eventId, err instanceof Error ? err.message : String(err));
throw err;
}
});
res.status(200).json({ accepted: true, notes: notes.length });
} catch (err) {
next(err);
}
});
export default router;

View File

@@ -1,45 +0,0 @@
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY,
});
const buildPrompt = (notes) => {
const notesJson = JSON.stringify(notes, null, 2);
return `You are an expert at analyzing text for semantic similarity and thematic patterns.
Below is a JSON array of sticky notes. Each note has an "id" and a "text" field. Analyze the "text" field of every note and group them into meaningful thematic clusters.
For each cluster, return ONLY a valid JSON array with the below exact structure — no markdown, no explanation, no extra text - where the value for the "label" key is a name you create to describe the cluster's theme and the value for the "noteIds" key is an array containing the Ids of the notes that fit into that cluster theme.
[
{
"label": "Short descriptive theme name for cluster",
"noteIds": ["note_001", "note_002"]
}
]
Rules:
- Every note must appear in exactly one cluster
- Each cluster must have a concise, descriptive label
- Group by semantic meaning, not by keywords
- Aim for the most natural number of groups given the data
Here are the notes:
${notesJson}`;
};
export const clusterNotes = async (notes) => {
const response = await client.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 4096,
messages: [
{ role: "user", content: buildPrompt(notes) },
],
});
const raw = response.content[0].text;
return JSON.parse(raw);
};

View File

@@ -0,0 +1,145 @@
import Anthropic from "@anthropic-ai/sdk";
import { pipeline } from "node:stream/promises";
import { Transform, Writable, type TransformCallback } from "node:stream";
import { embedNotes } from "./embedding.service.js";
import { validateStructure, computeCohesionScore } from "./validation.service.js";
import { isClusterArray, type Cluster, type ClusterResponse } from "../types/domain.js";
const client = new Anthropic({
apiKey: process.env.ANTHROPIC_API_KEY,
});
type ClusterableNote = { id: string; text: string };
type ClusterOptions = { signal?: AbortSignal };
type JsonSink = { parts: string[]; sawOpeningBracket: boolean };
const buildPrompt = (notes: ClusterableNote[]): string => {
const notesJson = JSON.stringify(notes, null, 2);
return `You are an expert at analyzing text for semantic similarity and thematic patterns.
Below is a JSON array of sticky notes. Each note has an "id" and a "text" field. Analyze the "text" field of every note and group them into meaningful thematic clusters.
For each cluster, return ONLY a valid JSON array with the below exact structure — no markdown, no explanation, no extra text - where the value for the "label" key is a name you create to describe the cluster's theme and the value for the "noteIds" key is an array containing the Ids of the notes that fit into that cluster theme.
[
{
"label": "Short descriptive theme name for cluster",
"noteIds": ["note_001", "note_002"]
}
]
Rules:
- Every note must appear in exactly one cluster
- Each cluster must have a concise, descriptive label
- Group by semantic meaning, not by keywords
- Aim for the most natural number of groups given the data
Here are the notes:
${notesJson}`;
};
const isTextDelta = (event: unknown): event is { delta: { text: string } } => {
if (typeof event !== 'object' || event === null) return false;
const candidate = event as { type?: unknown; delta?: { type?: unknown; text?: unknown } };
return (
candidate.type === 'content_block_delta' &&
candidate.delta?.type === 'text_delta' &&
typeof candidate.delta.text === 'string'
);
};
const textDeltas = (): Transform => new Transform({
objectMode: true,
transform(event: unknown, _encoding: BufferEncoding, callback: TransformCallback) {
if (isTextDelta(event)) {
callback(null, event.delta.text);
return;
}
callback();
},
});
// Rejects as soon as the first non-whitespace character proves the response
// is not the JSON array we asked for, rather than after the full generation.
const collectClusterJson = (sink: JsonSink): Writable => new Writable({
objectMode: true,
write(text: string, _encoding: BufferEncoding, callback: (error?: Error | null) => void) {
if (!sink.sawOpeningBracket) {
const leading = (sink.parts.join('') + text).trimStart();
if (leading.length > 0) {
if (!leading.startsWith('[')) {
callback(new Error('LLM API returned non-JSON response'));
return;
}
sink.sawOpeningBracket = true;
}
}
sink.parts.push(text);
callback();
},
});
const requestClusters = async (
notes: ClusterableNote[],
signal?: AbortSignal
): Promise<Cluster[]> => {
const sink: JsonSink = { parts: [], sawOpeningBracket: false };
const options = signal ? [{ signal }] : [];
const events = client.messages.stream(
{
model: "claude-sonnet-5",
max_tokens: 4096,
messages: [
{ role: "user", content: buildPrompt(notes) },
],
},
...options
);
await pipeline(events, textDeltas(), collectClusterJson(sink), ...options);
const text = sink.parts.join('');
if (text.trim().length === 0) {
throw new Error('Unexpected response from LLM API: no text content returned');
}
let parsed: unknown;
try {
parsed = JSON.parse(text);
} catch {
throw new Error('LLM API returned non-JSON response');
}
if (!isClusterArray(parsed)) {
throw new Error('LLM API returned clusters in an unexpected shape');
}
return parsed;
};
export const clusterNotes = async (
notes: ClusterableNote[],
{ signal }: ClusterOptions = {}
): Promise<ClusterResponse> => {
const [clusters, embeddingMap] = await Promise.all([
requestClusters(notes, signal),
embedNotes(notes),
]);
const noteIds = notes.map((n) => n.id);
const { valid, reasons } = validateStructure(clusters, noteIds);
if (!valid) {
throw new Error(`Cluster validation failed: ${reasons.join('; ')}`);
}
const score = computeCohesionScore(clusters, embeddingMap);
return { clusters, score: Math.round(score * 100) / 100 };
};

View File

@@ -0,0 +1,45 @@
import { VoyageAIClient } from "voyageai";
import { pipeline } from "node:stream/promises";
import { Readable } from "node:stream";
import { batch } from "../lib/streams.js";
import type { EmbeddingMap } from "../types/domain.js";
const client = new VoyageAIClient({
apiKey: process.env.VOYAGEAI_API_KEY,
});
const EMBED_BATCH_SIZE = 128;
type EmbeddableNote = { id: string; text: string };
/**
* @returns noteId to embedding vector
*/
export const embedNotes = async (notes: EmbeddableNote[]): Promise<EmbeddingMap> => {
const embeddingMap: EmbeddingMap = new Map();
if (!notes || notes.length === 0) {
return embeddingMap;
}
await pipeline(
Readable.from(notes, { objectMode: true }),
batch<EmbeddableNote>(EMBED_BATCH_SIZE),
async (batches: AsyncIterable<EmbeddableNote[]>) => {
for await (const chunk of batches) {
const response = await client.embed({
input: chunk.map((n) => n.text),
model: "voyage-3.5",
});
response.data?.forEach((item, i) => {
if (item.embedding) {
embeddingMap.set(chunk[i].id, item.embedding);
}
});
}
}
);
return embeddingMap;
};

View File

@@ -0,0 +1,23 @@
import { getProvider } from '../config/providers.js';
import { NormalizationError } from '../config/normalizers.js';
import type { NormalizedDelivery, ProviderSlug } from '../types/integration.js';
export { NormalizationError };
export const normalize = (
provider: ProviderSlug,
payload: unknown
): NormalizedDelivery => {
const normalizer = getProvider(provider).normalize;
if (!normalizer) {
throw new NormalizationError(
`No normalizer registered for provider "${provider}"`
);
}
return normalizer(payload);
};
export const hasNormalizer = (provider: ProviderSlug): boolean =>
getProvider(provider).normalize !== undefined;

View File

@@ -0,0 +1,182 @@
import { requestJson } from '../lib/httpClient.js';
import { getProvider } from '../config/providers.js';
import {
getAccessToken,
getRefreshToken,
findByProviderWorkspace,
updateTokens,
} from '../db/integrations.dao.js';
import type { IntegrationRow, ProviderSlug } from '../types/integration.js';
/** Refresh this far ahead of expiry so an in-flight call cannot straddle it. */
const REFRESH_MARGIN_MS = 60_000;
export type TokenSet = {
accessToken: string;
refreshToken?: string;
expiresAt?: Date;
scopes: string[];
};
type TokenResponse = {
access_token: string;
refresh_token?: string;
expires_in?: number;
scope?: string;
};
export class OAuthError extends Error {
constructor(message: string) {
super(message);
this.name = 'OAuthError';
}
}
const isTokenResponse = (value: unknown): value is TokenResponse => {
if (typeof value !== 'object' || value === null) return false;
const body = value as Record<string, unknown>;
if (typeof body.access_token !== 'string' || body.access_token.length === 0) return false;
if (body.refresh_token !== undefined && typeof body.refresh_token !== 'string') return false;
if (body.expires_in !== undefined && typeof body.expires_in !== 'number') return false;
if (body.scope !== undefined && typeof body.scope !== 'string') return false;
return true;
};
const credentials = (provider: ProviderSlug): { id: string; secret: string } => {
const prefix = provider.toUpperCase();
const id = process.env[`${prefix}_CLIENT_ID`];
const secret = process.env[`${prefix}_CLIENT_SECRET`];
if (!id || !secret) {
throw new OAuthError(
`Missing ${prefix}_CLIENT_ID or ${prefix}_CLIENT_SECRET`
);
}
return { id, secret };
};
const oauthConfig = (provider: ProviderSlug) => {
const config = getProvider(provider).oauth;
if (!config) {
throw new OAuthError(`Provider "${provider}" does not support OAuth`);
}
return config;
};
const toTokenSet = (body: TokenResponse): TokenSet => ({
accessToken: body.access_token,
refreshToken: body.refresh_token,
expiresAt: body.expires_in
? new Date(Date.now() + body.expires_in * 1000)
: undefined,
scopes: body.scope ? body.scope.split(/[\s,]+/).filter(Boolean) : [],
});
export const buildAuthorizeUrl = (
provider: ProviderSlug,
input: { redirectUri: string; state: string }
): string => {
const config = oauthConfig(provider);
const url = new URL(config.authorizeUrl);
url.searchParams.set('client_id', credentials(provider).id);
url.searchParams.set('redirect_uri', input.redirectUri);
url.searchParams.set('response_type', 'code');
url.searchParams.set('state', input.state);
url.searchParams.set('scope', config.scopes.join(' '));
return url.toString();
};
const postForm = async (
url: string,
form: Record<string, string>
): Promise<TokenSet> => {
const body = await requestJson<unknown>(url, {
method: 'POST',
headers: {
'content-type': 'application/x-www-form-urlencoded',
accept: 'application/json',
},
body: new URLSearchParams(form).toString(),
});
if (!isTokenResponse(body)) {
throw new OAuthError('Token endpoint returned an unexpected payload');
}
return toTokenSet(body);
};
export const exchangeCode = async (
provider: ProviderSlug,
input: { code: string; redirectUri: string }
): Promise<TokenSet> => {
const { id, secret } = credentials(provider);
return postForm(oauthConfig(provider).tokenUrl, {
grant_type: 'authorization_code',
code: input.code,
redirect_uri: input.redirectUri,
client_id: id,
client_secret: secret,
});
};
export const refreshAccessToken = async (
provider: ProviderSlug,
refreshToken: string
): Promise<TokenSet> => {
const { id, secret } = credentials(provider);
return postForm(oauthConfig(provider).tokenUrl, {
grant_type: 'refresh_token',
refresh_token: refreshToken,
client_id: id,
client_secret: secret,
});
};
const needsRefresh = (integration: IntegrationRow): boolean => {
if (!integration.tokenExpiresAt) return false;
return integration.tokenExpiresAt.getTime() - Date.now() <= REFRESH_MARGIN_MS;
};
/**
* Resolves a usable access token, refreshing first when the stored one is at
* or near expiry. Throws rather than returning null so a caller cannot make an
* unauthenticated request by forgetting a null check.
*/
export const getValidAccessToken = async (
provider: ProviderSlug,
externalWorkspaceId: string
): Promise<string> => {
const integration = await findByProviderWorkspace(provider, externalWorkspaceId);
if (!integration) {
throw new OAuthError(`No ${provider} integration for workspace ${externalWorkspaceId}`);
}
if (needsRefresh(integration)) {
const refreshToken = await getRefreshToken(integration.id);
if (!refreshToken) {
throw new OAuthError(`${provider} token expired and no refresh token is stored`);
}
const tokens = await refreshAccessToken(provider, refreshToken);
await updateTokens({
id: integration.id,
accessToken: tokens.accessToken,
refreshToken: tokens.refreshToken,
expiresAt: tokens.expiresAt,
});
return tokens.accessToken;
}
const accessToken = await getAccessToken(integration.id);
if (!accessToken) {
throw new OAuthError(`No access token stored for ${provider}`);
}
return accessToken;
};

View File

@@ -0,0 +1,135 @@
import type { Cluster, EmbeddingMap, ValidationResult } from '../types/domain.js';
/**
* Structural validation for LLM output
*
* @param clusters
* @param inputNoteIds - the original note IDs that were sent to the LLM
*/
export const validateStructure = (
clusters: Cluster[],
inputNoteIds: string[]
): ValidationResult => {
const reasons: string[] = [];
if (!Array.isArray(clusters) || clusters.length === 0) {
return { valid: false, reasons: ['Response is not a non-empty array'] };
}
const assignedIds: string[] = [];
for (const cluster of clusters) {
if (!cluster.label || typeof cluster.label !== 'string') {
reasons.push(`Cluster missing a valid label`);
}
if (!Array.isArray(cluster.noteIds) || cluster.noteIds.length === 0) {
reasons.push(`Cluster "${cluster.label ?? '(unlabeled)'}" has no noteIds`);
}
assignedIds.push(...(cluster.noteIds ?? []));
}
const inputSet = new Set(inputNoteIds);
const assignedSet = new Set(assignedIds);
if (assignedIds.length !== assignedSet.size) {
reasons.push('One or more notes appear in multiple clusters');
}
const missing = inputNoteIds.filter((id) => !assignedSet.has(id));
if (missing.length > 0) {
reasons.push(`Notes missing from clusters: ${missing.join(', ')}`);
}
const extra = assignedIds.filter((id) => !inputSet.has(id));
if (extra.length > 0) {
reasons.push(`Unknown noteIds in clusters: ${[...new Set(extra)].join(', ')}`);
}
if (clusters.length > inputNoteIds.length) {
reasons.push(`More clusters (${clusters.length}) than notes (${inputNoteIds.length})`);
}
return { valid: reasons.length === 0, reasons };
};
const cosineSimilarity = (a: number[], b: number[]): number => {
let dot = 0;
let magA = 0;
let magB = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
magA += a[i] * a[i];
magB += b[i] * b[i];
}
const denom = Math.sqrt(magA) * Math.sqrt(magB);
return denom === 0 ? 0 : dot / denom;
};
/**
* Computes silhouette-style cohesion score for the clustering.
*
* For each note, measures how much more similar it is to its own cluster
* versus the nearest neighboring cluster. Returns a score in [-1, 1]
* where higher is better.
*
* @param clusters
* @param embeddingMap - noteId to vector
* @returns average silhouette score
*/
export const computeCohesionScore = (
clusters: Cluster[],
embeddingMap: EmbeddingMap
): number => {
if (clusters.length <= 1) return 1.0;
const scores: number[] = [];
for (let ci = 0; ci < clusters.length; ci++) {
const clusterIds = clusters[ci].noteIds;
if (clusterIds.length <= 1) {
scores.push(0);
continue;
}
for (const noteId of clusterIds) {
const vec = embeddingMap.get(noteId);
if (!vec) continue;
// a(i): avg distance to other notes in same cluster
let intraSum = 0;
let intraCount = 0;
for (const otherId of clusterIds) {
if (otherId === noteId) continue;
const otherVec = embeddingMap.get(otherId);
if (!otherVec) continue;
intraSum += 1 - cosineSimilarity(vec, otherVec);
intraCount++;
}
const a = intraCount > 0 ? intraSum / intraCount : 0;
// b(i): min avg distance to notes in any other cluster
let b = Infinity;
for (let oi = 0; oi < clusters.length; oi++) {
if (oi === ci) continue;
const otherClusterIds = clusters[oi].noteIds;
let interSum = 0;
let interCount = 0;
for (const otherId of otherClusterIds) {
const otherVec = embeddingMap.get(otherId);
if (!otherVec) continue;
interSum += 1 - cosineSimilarity(vec, otherVec);
interCount++;
}
if (interCount > 0) {
b = Math.min(b, interSum / interCount);
}
}
if (b === Infinity) b = 0;
const max = Math.max(a, b);
scores.push(max === 0 ? 0 : (b - a) / max);
}
}
if (scores.length === 0) return 0;
return scores.reduce((sum, s) => sum + s, 0) / scores.length;
};

View File

@@ -1,86 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
const { createMock } = vi.hoisted(() => {
return { createMock: vi.fn() };
});
vi.mock('@anthropic-ai/sdk', () => {
return {
default: class MockAnthropic {
constructor() {
this.messages = { create: createMock };
}
},
};
});
import { clusterNotes } from '../services/clustering.service.js';
const MOCK_NOTES = [
{ id: 'note_001', text: 'Login is broken' },
{ id: 'note_002', text: 'Export fails' },
];
const MOCK_CLUSTERS = [
{ label: 'Auth Issues', noteIds: ['note_001'] },
{ label: 'Export Issues', noteIds: ['note_002'] },
];
describe('clusterNotes service', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('should call Anthropic messages.create with the correct model', async () => {
createMock.mockResolvedValue({
content: [{ text: JSON.stringify(MOCK_CLUSTERS) }],
});
await clusterNotes(MOCK_NOTES);
expect(createMock).toHaveBeenCalledOnce();
const callArgs = createMock.mock.calls[0][0];
expect(callArgs.model).toBe('claude-sonnet-4-20250514');
expect(callArgs.max_tokens).toBe(4096);
});
it('should include all note texts in prompt sent to the LLM API', async () => {
createMock.mockResolvedValue({
content: [{ text: JSON.stringify(MOCK_CLUSTERS) }],
});
await clusterNotes(MOCK_NOTES);
const prompt = createMock.mock.calls[0][0].messages[0].content;
expect(prompt).toContain('note_001');
expect(prompt).toContain('Login is broken');
expect(prompt).toContain('note_002');
expect(prompt).toContain('Export fails');
});
it('should parse and return the clustered JSON from the API response', async () => {
createMock.mockResolvedValue({
content: [{ text: JSON.stringify(MOCK_CLUSTERS) }],
});
const result = await clusterNotes(MOCK_NOTES);
expect(result).toEqual(MOCK_CLUSTERS);
});
it('should throw error when the API returns non-JSON', async () => {
createMock.mockResolvedValue({
content: [{ text: 'An unknown error occured when generting structured response.' }],
});
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow();
});
it('should throw error when Anthropic API authentication fails', async () => {
createMock.mockRejectedValue(new Error('401 Unauthorized'));
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('401 Unauthorized');
});
});

View File

@@ -0,0 +1,172 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import type { Cluster, EmbeddingMap } from '../types/domain.js';
type StreamEvent = {
type: string;
delta?: { type: string; text: string };
};
const { streamMock, mockEmbeddings } = vi.hoisted(() => {
const embeddings = new Map<string, number[]>([
['note_001', [1.0, 0.0, 0.0]],
['note_002', [0.0, 1.0, 0.0]],
]);
return { streamMock: vi.fn(), mockEmbeddings: embeddings as EmbeddingMap };
});
vi.mock('@anthropic-ai/sdk', () => {
return {
default: class MockAnthropic {
messages = { stream: streamMock };
},
};
});
vi.mock('../services/embedding.service.js', () => ({
embedNotes: vi.fn().mockResolvedValue(mockEmbeddings),
}));
import { clusterNotes } from '../services/clustering.service.js';
const MOCK_NOTES = [
{ id: 'note_001', text: 'Login is broken' },
{ id: 'note_002', text: 'Export fails' },
];
const MOCK_CLUSTERS: Cluster[] = [
{ label: 'Auth Issues', noteIds: ['note_001'] },
{ label: 'Export Issues', noteIds: ['note_002'] },
];
// Splits text into several text_delta events so the service is exercised
// against a genuinely incremental stream rather than one whole payload.
const textEvents = (text: string, pieces = 4): StreamEvent[] => {
const size = Math.max(1, Math.ceil(text.length / pieces));
const events: StreamEvent[] = [];
for (let i = 0; i < text.length; i += size) {
events.push({
type: 'content_block_delta',
delta: { type: 'text_delta', text: text.slice(i, i + size) },
});
}
return events;
};
const mockStreamOf = (text: string): void => {
const events: StreamEvent[] = [
{ type: 'message_start' },
...textEvents(text),
{ type: 'message_stop' },
];
streamMock.mockImplementation(() => ({
async *[Symbol.asyncIterator]() {
for (const event of events) yield event;
},
}));
};
const mockStreamThrowing = (err: Error): void => {
streamMock.mockImplementation(() => ({
async *[Symbol.asyncIterator]() {
throw err;
},
}));
};
describe('clusterNotes service', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('should call Anthropic messages.stream with the correct model', async () => {
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
await clusterNotes(MOCK_NOTES);
expect(streamMock).toHaveBeenCalledOnce();
const callArgs = streamMock.mock.calls[0][0];
expect(callArgs.model).toBe('claude-sonnet-5');
expect(callArgs.max_tokens).toBe(4096);
});
it('should forward an abort signal to the LLM request', async () => {
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
const controller = new AbortController();
await clusterNotes(MOCK_NOTES, { signal: controller.signal });
expect(streamMock.mock.calls[0][1]).toEqual({ signal: controller.signal });
});
it('should include all note texts in prompt sent to the LLM API', async () => {
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
await clusterNotes(MOCK_NOTES);
const prompt = streamMock.mock.calls[0][0].messages[0].content;
expect(prompt).toContain('note_001');
expect(prompt).toContain('Login is broken');
expect(prompt).toContain('note_002');
expect(prompt).toContain('Export fails');
});
it('should return clusters and a cohesion score', async () => {
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
const result = await clusterNotes(MOCK_NOTES);
expect(result.clusters).toEqual(MOCK_CLUSTERS);
expect(typeof result.score).toBe('number');
expect(result.score).toBeGreaterThanOrEqual(-1);
expect(result.score).toBeLessThanOrEqual(1);
});
it('should reassemble clusters split across many stream deltas', async () => {
const json = JSON.stringify(MOCK_CLUSTERS);
streamMock.mockImplementation(() => ({
async *[Symbol.asyncIterator]() {
for (const char of json) {
yield { type: 'content_block_delta', delta: { type: 'text_delta', text: char } };
}
},
}));
const result = await clusterNotes(MOCK_NOTES);
expect(result.clusters).toEqual(MOCK_CLUSTERS);
});
it('should throw error when the API returns non-JSON', async () => {
mockStreamOf('An unknown error occured when generting structured response.');
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('non-JSON response');
});
it('should throw error when the API response has no text content', async () => {
mockStreamOf('');
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('no text content returned');
});
it('should throw error when Anthropic API authentication fails', async () => {
mockStreamThrowing(new Error('401 Unauthorized'));
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('401 Unauthorized');
});
it('should throw a validation error when a note is missing from clusters', async () => {
const incompleteClusters: Cluster[] = [
{ label: 'Auth Issues', noteIds: ['note_001'] },
];
mockStreamOf(JSON.stringify(incompleteClusters));
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('Cluster validation failed');
});
it('should throw when the API returns a JSON array of the wrong shape', async () => {
mockStreamOf(JSON.stringify([{ name: 'Auth Issues', ids: ['note_001'] }]));
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('unexpected shape');
});
});

View File

@@ -1,135 +0,0 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import request from 'supertest';
import app from '../app.js';
vi.mock('../services/clustering.service.js', () => ({
clusterNotes: vi.fn(),
}));
vi.mock('fs/promises', () => ({
readFile: vi.fn(),
}));
import { clusterNotes } from '../services/clustering.service.js';
import { readFile } from 'fs/promises';
const MOCK_NOTES = [
{ id: 'note_001', text: 'Login flow feels confusing', x: 193, y: 191, author: 'user_5', color: 'yellow' },
{ id: 'note_002', text: 'Login flow is broken on mobile', x: 214, y: 281, author: 'user_9', color: 'yellow' },
{ id: 'note_003', text: 'Export takes too long', x: 798, y: 211, author: 'user_2', color: 'green' },
];
const MOCK_CLUSTERS = [
{ label: 'Login Issues', noteIds: ['note_001', 'note_002'] },
{ label: 'Export Problems', noteIds: ['note_003'] },
];
describe('GET /v1/notes', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('should return 200 and an array of notes', async () => {
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
const res = await request(app).get('/v1/notes');
expect(res.status).toBe(200);
expect(res.body).toEqual(MOCK_NOTES);
expect(Array.isArray(res.body)).toBe(true);
});
it('should return notes with expected properties', async () => {
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
const res = await request(app).get('/v1/notes');
const note = res.body[0];
expect(note).toHaveProperty('id');
expect(note).toHaveProperty('text');
expect(note).toHaveProperty('x');
expect(note).toHaveProperty('y');
expect(note).toHaveProperty('author');
expect(note).toHaveProperty('color');
});
it('should return 500 when the data file cannot be read', async () => {
readFile.mockRejectedValue(new Error('ENOENT: file not found'));
const res = await request(app).get('/v1/notes');
expect(res.status).toBe(500);
expect(res.body).toHaveProperty('error');
expect(res.body.error).toBe('Failed to load notes');
});
it('should return 500 when data file is invalid JSON', async () => {
readFile.mockResolvedValue('{ this is not valid json }');
const res = await request(app).get('/v1/notes');
expect(res.status).toBe(500);
expect(res.body).toHaveProperty('error');
});
});
describe('POST /v1/notes/cluster', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('should return 200 and clustered results', async () => {
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
clusterNotes.mockResolvedValue(MOCK_CLUSTERS);
const res = await request(app).post('/v1/notes/cluster');
expect(res.status).toBe(200);
expect(res.body).toEqual(MOCK_CLUSTERS);
});
it('should return clusters with the expected shape (label, noteIds)', async () => {
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
clusterNotes.mockResolvedValue(MOCK_CLUSTERS);
const res = await request(app).post('/v1/notes/cluster');
const cluster = res.body[0];
expect(cluster).toHaveProperty('label');
expect(cluster).toHaveProperty('noteIds');
expect(typeof cluster.label).toBe('string');
expect(Array.isArray(cluster.noteIds)).toBe(true);
});
it('should pass the loaded notes to clusterNotes', async () => {
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
clusterNotes.mockResolvedValue(MOCK_CLUSTERS);
await request(app).post('/v1/notes/cluster');
expect(clusterNotes).toHaveBeenCalledOnce();
expect(clusterNotes).toHaveBeenCalledWith(MOCK_NOTES);
});
it('should return 500 when clusterNotes (API call) fails', async () => {
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
clusterNotes.mockRejectedValue(new Error('Anthropic API error'));
const res = await request(app).post('/v1/notes/cluster');
expect(res.status).toBe(500);
expect(res.body).toHaveProperty('error');
expect(res.body.error).toMatch(/^Clustering failed/);
});
it('should return 500 when the data file cannot be read', async () => {
readFile.mockRejectedValue(new Error('ENOENT: file not found'));
const res = await request(app).post('/v1/notes/cluster');
expect(res.status).toBe(500);
expect(res.body).toHaveProperty('error');
});
});

183
backend/tests/Notes.test.ts Normal file
View File

@@ -0,0 +1,183 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import request from 'supertest';
import { Readable } from 'node:stream';
import app from '../app.js';
import type { Cluster, ClusterResponse, Note } from '../types/domain.js';
vi.mock('../db/notes.dao.js', () => ({
getAllNotes: vi.fn(),
streamAllNotes: vi.fn(),
}));
vi.mock('../services/clustering.service.js', () => ({
clusterNotes: vi.fn(),
}));
import { getAllNotes, streamAllNotes } from '../db/notes.dao.js';
import { clusterNotes } from '../services/clustering.service.js';
const mockGetAllNotes = vi.mocked(getAllNotes);
const mockStreamAllNotes = vi.mocked(streamAllNotes);
const mockClusterNotes = vi.mocked(clusterNotes);
const MOCK_NOTES: Note[] = [
{ id: 'note_001', text: 'Login flow feels confusing', x: 193, y: 191, author: 'user_5', color: 'yellow' },
{ id: 'note_002', text: 'Login flow is broken on mobile', x: 214, y: 281, author: 'user_9', color: 'yellow' },
{ id: 'note_003', text: 'Export takes too long', x: 798, y: 211, author: 'user_2', color: 'green' },
];
const MOCK_CLUSTERS: Cluster[] = [
{ label: 'Login Issues', noteIds: ['note_001', 'note_002'] },
{ label: 'Export Problems', noteIds: ['note_003'] },
];
const MOCK_RESULT: ClusterResponse = { clusters: MOCK_CLUSTERS, score: 0.09 };
const rowStream = (rows: Note[]): Readable => Readable.from(rows, { objectMode: true });
describe('GET /v1/notes', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('should return 200 and an array of notes', async () => {
mockStreamAllNotes.mockResolvedValue(rowStream(MOCK_NOTES));
const res = await request(app).get('/v1/notes');
expect(res.status).toBe(200);
expect(res.body).toEqual(MOCK_NOTES);
expect(Array.isArray(res.body)).toBe(true);
});
it('should send JSON incrementally rather than buffering the row set', async () => {
mockStreamAllNotes.mockResolvedValue(rowStream(MOCK_NOTES));
const res = await request(app).get('/v1/notes');
expect(res.headers['content-type']).toMatch(/application\/json/);
expect(res.headers['content-length']).toBeUndefined();
});
it('should return an empty array when there are no notes', async () => {
mockStreamAllNotes.mockResolvedValue(rowStream([]));
const res = await request(app).get('/v1/notes');
expect(res.status).toBe(200);
expect(res.body).toEqual([]);
});
it('should return notes with expected properties', async () => {
mockStreamAllNotes.mockResolvedValue(rowStream(MOCK_NOTES));
const res = await request(app).get('/v1/notes');
const note = res.body[0];
expect(note).toHaveProperty('id');
expect(note).toHaveProperty('text');
expect(note).toHaveProperty('x');
expect(note).toHaveProperty('y');
expect(note).toHaveProperty('author');
expect(note).toHaveProperty('color');
});
it('should return 500 when the database query fails', async () => {
mockStreamAllNotes.mockRejectedValue(new Error('connection refused'));
const res = await request(app).get('/v1/notes');
expect(res.status).toBe(500);
expect(res.body).toHaveProperty('error');
expect(res.body.error).toBe('Failed to load notes');
});
it('should abort the response when the row stream fails mid-flight', async () => {
const failing = new Readable({
objectMode: true,
read() {
this.push(MOCK_NOTES[0]);
this.destroy(new Error('connection lost'));
},
});
mockStreamAllNotes.mockResolvedValue(failing);
await expect(request(app).get('/v1/notes')).rejects.toThrow();
});
});
describe('POST /v1/notes/cluster', () => {
beforeEach(() => {
vi.clearAllMocks();
});
it('should return 200 and clustered results', async () => {
mockGetAllNotes.mockResolvedValue(MOCK_NOTES);
mockClusterNotes.mockResolvedValue(MOCK_RESULT);
const res = await request(app).post('/v1/notes/cluster');
expect(res.status).toBe(200);
expect(res.body).toEqual(MOCK_RESULT);
});
it('should return clusters with the expected shape (label, noteIds)', async () => {
mockGetAllNotes.mockResolvedValue(MOCK_NOTES);
mockClusterNotes.mockResolvedValue(MOCK_RESULT);
const res = await request(app).post('/v1/notes/cluster');
const cluster = res.body.clusters[0];
expect(cluster).toHaveProperty('label');
expect(cluster).toHaveProperty('noteIds');
expect(typeof cluster.label).toBe('string');
expect(Array.isArray(cluster.noteIds)).toBe(true);
});
it('should pass the loaded notes to clusterNotes', async () => {
mockGetAllNotes.mockResolvedValue(MOCK_NOTES);
mockClusterNotes.mockResolvedValue(MOCK_RESULT);
await request(app).post('/v1/notes/cluster');
expect(mockClusterNotes).toHaveBeenCalledOnce();
expect(mockClusterNotes).toHaveBeenCalledWith(
MOCK_NOTES,
expect.objectContaining({ signal: expect.any(AbortSignal) })
);
});
it('should pass a signal that is not aborted while the request is open', async () => {
mockGetAllNotes.mockResolvedValue(MOCK_NOTES);
mockClusterNotes.mockImplementation(async (_notes, options = {}) => {
expect(options.signal?.aborted).toBe(false);
return MOCK_RESULT;
});
const res = await request(app).post('/v1/notes/cluster');
expect(res.status).toBe(200);
});
it('should return 500 when clusterNotes (API call) fails', async () => {
mockGetAllNotes.mockResolvedValue(MOCK_NOTES);
mockClusterNotes.mockRejectedValue(new Error('LLM API error'));
const res = await request(app).post('/v1/notes/cluster');
expect(res.status).toBe(500);
expect(res.body).toHaveProperty('error');
expect(res.body.error).toMatch(/^Clustering failed/);
});
it('should return 500 when the database query fails', async () => {
mockGetAllNotes.mockRejectedValue(new Error('connection refused'));
const res = await request(app).post('/v1/notes/cluster');
expect(res.status).toBe(500);
expect(res.body).toHaveProperty('error');
});
});

View File

@@ -0,0 +1,170 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { randomBytes } from 'node:crypto';
import {
encryptSecret,
decryptSecret,
hashApiKey,
safeEquals,
} from '../lib/crypto.js';
const base64Key = (bytes: number): string => randomBytes(bytes).toString('base64');
const VALID_KEY = base64Key(32);
/** Rewrites one `iv:tag:payload` segment, flipping every bit of its first byte. */
const corruptSegment = (ciphertext: string, index: number): string => {
const parts = ciphertext.split(':');
const bytes = Buffer.from(parts[index], 'base64');
bytes[0] = bytes[0] ^ 0xff;
parts[index] = bytes.toString('base64');
return parts.join(':');
};
describe('encryptSecret / decryptSecret', () => {
const originalKey = process.env.TOKEN_ENCRYPTION_KEY;
beforeEach(() => {
process.env.TOKEN_ENCRYPTION_KEY = VALID_KEY;
});
afterEach(() => {
if (originalKey === undefined) {
delete process.env.TOKEN_ENCRYPTION_KEY;
} else {
process.env.TOKEN_ENCRYPTION_KEY = originalKey;
}
});
it('should round-trip a plaintext secret', () => {
const plaintext = 'lin_oauth_abc123';
expect(decryptSecret(encryptSecret(plaintext))).toBe(plaintext);
});
it('should round-trip an empty string and multi-byte characters', () => {
expect(decryptSecret(encryptSecret(''))).toBe('');
expect(decryptSecret(encryptSecret('clé—😀'))).toBe('clé—😀');
});
it('should emit three base64 segments', () => {
const parts = encryptSecret('token').split(':');
expect(parts).toHaveLength(3);
expect(Buffer.from(parts[0], 'base64')).toHaveLength(12);
expect(Buffer.from(parts[1], 'base64')).toHaveLength(16);
});
it('should produce different ciphertext for the same plaintext each time', () => {
const first = encryptSecret('same-secret');
const second = encryptSecret('same-secret');
expect(first).not.toBe(second);
expect(first.split(':')[0]).not.toBe(second.split(':')[0]);
expect(decryptSecret(first)).toBe('same-secret');
expect(decryptSecret(second)).toBe('same-secret');
});
it('should throw when the ciphertext payload is tampered with', () => {
const tampered = corruptSegment(encryptSecret('payload-under-attack'), 2);
expect(() => decryptSecret(tampered)).toThrow();
});
it('should throw when the auth tag is tampered with', () => {
const tampered = corruptSegment(encryptSecret('tag-under-attack'), 1);
expect(() => decryptSecret(tampered)).toThrow();
});
it('should throw when the iv is tampered with', () => {
const tampered = corruptSegment(encryptSecret('iv-under-attack'), 0);
expect(() => decryptSecret(tampered)).toThrow();
});
it('should throw for a ciphertext that is not in iv:tag:payload form', () => {
expect(() => decryptSecret('not-a-ciphertext')).toThrow(
/iv:tag:payload/
);
expect(() => decryptSecret('only:two')).toThrow(/iv:tag:payload/);
expect(() => decryptSecret('a:b:c:d')).toThrow(/iv:tag:payload/);
expect(() => decryptSecret('')).toThrow(/iv:tag:payload/);
});
it('should throw when decrypting under a different key', () => {
const ciphertext = encryptSecret('bound-to-one-key');
process.env.TOKEN_ENCRYPTION_KEY = base64Key(32);
expect(() => decryptSecret(ciphertext)).toThrow();
});
it('should throw when TOKEN_ENCRYPTION_KEY is missing', () => {
delete process.env.TOKEN_ENCRYPTION_KEY;
expect(() => encryptSecret('anything')).toThrow(/not set/);
expect(() => decryptSecret('a:b:c')).toThrow(/not set/);
});
it('should throw when TOKEN_ENCRYPTION_KEY is empty', () => {
process.env.TOKEN_ENCRYPTION_KEY = '';
expect(() => encryptSecret('anything')).toThrow(/not set/);
});
it('should throw when the key decodes to the wrong byte length', () => {
process.env.TOKEN_ENCRYPTION_KEY = base64Key(16);
expect(() => encryptSecret('anything')).toThrow(/32 bytes, got 16/);
process.env.TOKEN_ENCRYPTION_KEY = base64Key(48);
expect(() => encryptSecret('anything')).toThrow(/32 bytes, got 48/);
});
it('should read the key at call time rather than at import time', () => {
process.env.TOKEN_ENCRYPTION_KEY = base64Key(31);
expect(() => encryptSecret('anything')).toThrow(/got 31/);
process.env.TOKEN_ENCRYPTION_KEY = VALID_KEY;
expect(decryptSecret(encryptSecret('recovered'))).toBe('recovered');
});
});
describe('hashApiKey', () => {
it('should be stable for the same input', () => {
expect(hashApiKey('kg_live_abc')).toBe(hashApiKey('kg_live_abc'));
});
it('should differ for different inputs', () => {
expect(hashApiKey('kg_live_abc')).not.toBe(hashApiKey('kg_live_abd'));
expect(hashApiKey('')).not.toBe(hashApiKey(' '));
});
it('should return a 64-character lowercase hex digest', () => {
expect(hashApiKey('kg_live_abc')).toMatch(/^[0-9a-f]{64}$/);
});
it('should match the plain sha256 digest of the input', () => {
expect(hashApiKey('')).toBe(
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
);
});
});
describe('safeEquals', () => {
it('should return true for identical strings', () => {
expect(safeEquals('kg_live_abc', 'kg_live_abc')).toBe(true);
expect(safeEquals('', '')).toBe(true);
});
it('should return false for different strings of equal length', () => {
expect(safeEquals('kg_live_abc', 'kg_live_abd')).toBe(false);
});
it('should return false for strings of differing lengths without throwing', () => {
expect(safeEquals('short', 'a-much-longer-value')).toBe(false);
expect(safeEquals('', 'x')).toBe(false);
});
it('should be case sensitive', () => {
expect(safeEquals('Secret', 'secret')).toBe(false);
});
});

View File

@@ -0,0 +1,241 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { requestJson, isHttpRequestError, parseRetryAfter } from '../lib/httpClient.js';
const URL_UNDER_TEST = 'https://api.example.com/v1/token';
const jsonResponse = (
body: unknown,
status = 200,
headers: Record<string, string> = {}
): Response =>
new Response(JSON.stringify(body), {
status,
headers: { 'content-type': 'application/json', ...headers },
});
/** Never settles on its own; only the per-attempt timeout or the caller's signal ends it. */
const hangUntilAborted = (init?: RequestInit): Promise<Response> =>
new Promise<Response>((_resolve, reject) => {
init?.signal?.addEventListener('abort', () => {
reject(init.signal?.reason ?? new Error('aborted'));
});
});
// Backoff is kept at a single millisecond so retries are asserted by call count, never by clock.
const fast = { maxAttempts: 3, baseDelayMs: 1 } as const;
describe('requestJson', () => {
let fetchMock: ReturnType<typeof vi.fn>;
beforeEach(() => {
fetchMock = vi.fn();
vi.stubGlobal('fetch', fetchMock);
});
afterEach(() => {
vi.unstubAllGlobals();
});
it('should resolve and parse a 200 JSON body', async () => {
fetchMock.mockResolvedValueOnce(jsonResponse({ access_token: 'abc', expires_in: 3600 }));
const result = await requestJson<{ access_token: string; expires_in: number }>(
URL_UNDER_TEST,
fast
);
expect(result).toEqual({ access_token: 'abc', expires_in: 3600 });
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('should retry a 429 with a small Retry-After and then succeed', async () => {
fetchMock
.mockResolvedValueOnce(jsonResponse({ error: 'slow down' }, 429, { 'retry-after': '0' }))
.mockResolvedValueOnce(jsonResponse({ ok: true }));
const result = await requestJson<{ ok: boolean }>(URL_UNDER_TEST, fast);
expect(result).toEqual({ ok: true });
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('should honor an HTTP-date Retry-After on a 429', async () => {
const httpDate = new Date(Date.now() + 500).toUTCString();
fetchMock
.mockResolvedValueOnce(jsonResponse({}, 429, { 'retry-after': httpDate }))
.mockResolvedValueOnce(jsonResponse({ ok: true }));
const result = await requestJson<{ ok: boolean }>(URL_UNDER_TEST, fast);
expect(result).toEqual({ ok: true });
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('should retry a 500 up to maxAttempts and then throw', async () => {
fetchMock.mockImplementation(() => Promise.resolve(jsonResponse({ error: 'boom' }, 500)));
await expect(requestJson(URL_UNDER_TEST, fast)).rejects.toThrow(/status 500/);
expect(fetchMock).toHaveBeenCalledTimes(3);
});
it('should throw immediately on a 400 without retrying', async () => {
fetchMock.mockResolvedValueOnce(jsonResponse({ error: 'invalid_grant' }, 400));
await expect(requestJson(URL_UNDER_TEST, fast)).rejects.toThrow();
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('should throw immediately on a 401 without retrying', async () => {
fetchMock.mockResolvedValueOnce(jsonResponse({ error: 'unauthorized' }, 401));
await expect(requestJson(URL_UNDER_TEST, fast)).rejects.toThrow();
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('should treat a per-attempt timeout as retryable', async () => {
fetchMock
.mockImplementationOnce((_url: string, init?: RequestInit) => hangUntilAborted(init))
.mockResolvedValueOnce(jsonResponse({ ok: true }));
const result = await requestJson<{ ok: boolean }>(URL_UNDER_TEST, {
...fast,
timeoutMs: 1,
});
expect(result).toEqual({ ok: true });
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('should exhaust attempts when every attempt times out', async () => {
fetchMock.mockImplementation((_url: string, init?: RequestInit) => hangUntilAborted(init));
const error = await requestJson(URL_UNDER_TEST, { ...fast, maxAttempts: 2, timeoutMs: 1 })
.then(() => null)
.catch((err: unknown) => err);
expect(isHttpRequestError(error)).toBe(true);
expect(isHttpRequestError(error) && error.kind).toBe('timeout');
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('should identify the url, status and attempt count in the thrown message', async () => {
fetchMock.mockResolvedValueOnce(jsonResponse({ error: 'invalid_grant' }, 400));
const error = await requestJson(URL_UNDER_TEST, fast)
.then(() => null)
.catch((err: unknown) => err);
expect(isHttpRequestError(error)).toBe(true);
const message = error instanceof Error ? error.message : '';
expect(message).toContain(URL_UNDER_TEST);
expect(message).toContain('status 400');
expect(message).toContain('1 attempt(s)');
expect(isHttpRequestError(error) && error.status).toBe(400);
});
it('should distinguish an HTTP failure from a programming error', async () => {
fetchMock.mockResolvedValueOnce(jsonResponse({}, 400));
const httpError = await requestJson(URL_UNDER_TEST, fast)
.then(() => null)
.catch((err: unknown) => err);
expect(isHttpRequestError(httpError)).toBe(true);
expect(isHttpRequestError(new TypeError('bad call'))).toBe(false);
});
it('should reject a 2xx whose body is not valid JSON', async () => {
fetchMock.mockResolvedValueOnce(new Response('<html>maintenance</html>', { status: 200 }));
const error = await requestJson(URL_UNDER_TEST, fast)
.then(() => null)
.catch((err: unknown) => err);
expect(isHttpRequestError(error) && error.kind).toBe('invalid-body');
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('should resolve undefined for a bodiless 204', async () => {
fetchMock.mockResolvedValueOnce(new Response(null, { status: 204 }));
await expect(requestJson(URL_UNDER_TEST, fast)).resolves.toBeUndefined();
});
it('should retry a transport error and then succeed', async () => {
fetchMock
.mockRejectedValueOnce(new TypeError('fetch failed'))
.mockResolvedValueOnce(jsonResponse({ ok: true }));
const result = await requestJson<{ ok: boolean }>(URL_UNDER_TEST, fast);
expect(result).toEqual({ ok: true });
expect(fetchMock).toHaveBeenCalledTimes(2);
});
it('should not call fetch when the caller signal is already aborted', async () => {
fetchMock.mockResolvedValue(jsonResponse({ ok: true }));
const error = await requestJson(URL_UNDER_TEST, { ...fast, signal: AbortSignal.abort() })
.then(() => null)
.catch((err: unknown) => err);
expect(isHttpRequestError(error) && error.kind).toBe('aborted');
expect(fetchMock).not.toHaveBeenCalled();
});
it('should abort without retrying when the caller signal fires mid-flight', async () => {
const controller = new AbortController();
fetchMock.mockImplementation((_url: string, init?: RequestInit) => {
queueMicrotask(() => controller.abort());
return hangUntilAborted(init);
});
const error = await requestJson(URL_UNDER_TEST, { ...fast, signal: controller.signal })
.then(() => null)
.catch((err: unknown) => err);
expect(isHttpRequestError(error) && error.kind).toBe('aborted');
expect(fetchMock).toHaveBeenCalledTimes(1);
});
it('should reject a non-positive maxAttempts', async () => {
await expect(requestJson(URL_UNDER_TEST, { maxAttempts: 0 })).rejects.toThrow(TypeError);
expect(fetchMock).not.toHaveBeenCalled();
});
});
describe('parseRetryAfter', () => {
it('should read the delay-seconds form', () => {
expect(parseRetryAfter('30')).toBe(30_000);
expect(parseRetryAfter('0')).toBe(0);
});
it('should read the HTTP-date form as a delay from now', () => {
const delay = parseRetryAfter(new Date(Date.now() + 30_000).toUTCString());
expect(delay).not.toBeNull();
expect(delay).toBeGreaterThan(25_000);
expect(delay).toBeLessThanOrEqual(30_000);
});
it('should clamp a past HTTP-date to zero', () => {
expect(parseRetryAfter(new Date(Date.now() - 60_000).toUTCString())).toBe(0);
});
it('should cap a hostile delay-seconds value', () => {
expect(parseRetryAfter('86400')).toBe(60_000);
});
it('should cap a hostile HTTP-date value', () => {
expect(parseRetryAfter(new Date(Date.now() + 86_400_000).toUTCString())).toBe(60_000);
});
it('should return null for a missing or unintelligible header', () => {
expect(parseRetryAfter(null)).toBeNull();
expect(parseRetryAfter(' ')).toBeNull();
expect(parseRetryAfter('soon')).toBeNull();
});
});

View File

@@ -0,0 +1,158 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import request from 'supertest';
import app from '../app.js';
import type { IntegrationRow } from '../types/integration.js';
import type { Note, NoteInput } from '../types/domain.js';
vi.mock('../db/integrations.dao.js', () => ({
findByApiKeyHash: vi.fn(),
}));
vi.mock('../db/notes.dao.js', () => ({
createNotes: vi.fn(),
getAllNotes: vi.fn(),
streamAllNotes: vi.fn(),
}));
import { findByApiKeyHash } from '../db/integrations.dao.js';
import { createNotes } from '../db/notes.dao.js';
const mockFindByApiKeyHash = vi.mocked(findByApiKeyHash);
const mockCreateNotes = vi.mocked(createNotes);
const INTEGRATION: IntegrationRow = {
id: 7,
provider: 'rest',
externalWorkspaceId: 'acme',
displayName: 'Acme',
scopes: [],
tokenExpiresAt: null,
};
const INPUT: NoteInput[] = [
{ id: 'note_100', text: 'Retro: deploys are scary', author: 'user_1' },
];
const STORED: Note[] = [
{ id: 'note_100', text: 'Retro: deploys are scary', x: 0, y: 0, author: 'user_1', color: 'yellow' },
];
const post = (body: object, key = 'secret-key') =>
request(app).post('/v1/notes').set('Authorization', `Bearer ${key}`).send(body);
describe('POST /v1/notes', () => {
beforeEach(() => {
vi.clearAllMocks();
mockFindByApiKeyHash.mockResolvedValue(INTEGRATION);
mockCreateNotes.mockResolvedValue(STORED);
});
it('should insert notes and return 201 with the stored rows', async () => {
const res = await post({ notes: INPUT });
expect(res.status).toBe(201);
expect(res.body).toEqual({ inserted: 1, notes: STORED });
expect(mockCreateNotes).toHaveBeenCalledWith(INPUT);
});
it('should report the inserted count from the database, not the request', async () => {
mockCreateNotes.mockResolvedValue([]);
const res = await post({ notes: INPUT });
expect(res.body.inserted).toBe(0);
});
it('should reject a body with no notes key', async () => {
const res = await post({});
expect(res.status).toBe(400);
expect(mockCreateNotes).not.toHaveBeenCalled();
});
it('should reject notes that is not an array', async () => {
const res = await post({ notes: 'nope' });
expect(res.status).toBe(400);
});
it('should reject an empty notes array', async () => {
const res = await post({ notes: [] });
expect(res.status).toBe(400);
expect(mockCreateNotes).not.toHaveBeenCalled();
});
it('should reject a note missing text', async () => {
const res = await post({ notes: [{ id: 'a', author: 'user_1' }] });
expect(res.status).toBe(400);
});
it('should reject a note whose x is not a number', async () => {
const res = await post({ notes: [{ ...INPUT[0], x: 'left' }] });
expect(res.status).toBe(400);
});
it('should reject a batch over the per-request limit', async () => {
const many = Array.from({ length: 5001 }, (_, i) => ({
id: `note_${i}`,
text: 'text',
author: 'user_1',
}));
const res = await post({ notes: many });
expect(res.status).toBe(400);
expect(mockCreateNotes).not.toHaveBeenCalled();
});
it('should return 401 when no Authorization header is sent', async () => {
const res = await request(app).post('/v1/notes').send({ notes: INPUT });
expect(res.status).toBe(401);
expect(mockCreateNotes).not.toHaveBeenCalled();
});
it('should return 401 for a non-bearer scheme', async () => {
const res = await request(app)
.post('/v1/notes')
.set('Authorization', 'Basic abc123')
.send({ notes: INPUT });
expect(res.status).toBe(401);
});
it('should return 401 for an unknown key without revealing why', async () => {
mockFindByApiKeyHash.mockResolvedValue(null);
const res = await post({ notes: INPUT });
expect(res.status).toBe(401);
expect(res.body).toEqual({ error: 'Unauthorized' });
});
it('should not send the raw key to the database', async () => {
await post({ notes: INPUT }, 'plaintext-key');
const [hash] = mockFindByApiKeyHash.mock.calls[0];
expect(hash).not.toContain('plaintext-key');
expect(hash).toMatch(/^[0-9a-f]{64}$/);
});
it('should return 500 when the insert fails', async () => {
mockCreateNotes.mockRejectedValue(new Error('connection refused'));
const res = await post({ notes: INPUT });
expect(res.status).toBe(500);
});
it('should leave the existing GET /v1/notes route reachable', async () => {
const res = await request(app).get('/v1/notes');
expect(res.status).not.toBe(404);
});
});

View File

@@ -0,0 +1,142 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
const { mockQuery, mockConnect } = vi.hoisted(() => ({
mockQuery: vi.fn(),
mockConnect: vi.fn(),
}));
vi.mock('../db/index.js', () => ({
query: mockQuery,
getPool: () => ({ connect: mockConnect }),
}));
import {
recordDelivery,
markProcessing,
markDone,
markFailed,
resetStaleProcessing,
} from '../db/ingest_events.dao.js';
describe('ingest_events.dao', () => {
beforeEach(() => {
vi.clearAllMocks();
});
describe('recordDelivery', () => {
it('should return the new row id for a delivery that has not been seen', async () => {
mockQuery.mockResolvedValue({ rows: [{ id: 42 }] });
const result = await recordDelivery({ provider: 'slack', externalId: 'evt_1' });
expect(result).toBe(42);
});
it('should return null when the delivery conflicts with an existing row', async () => {
mockQuery.mockResolvedValue({ rows: [] });
const result = await recordDelivery({ provider: 'slack', externalId: 'evt_1' });
expect(result).toBeNull();
});
it('should insert with ON CONFLICT DO NOTHING on the provider and external id', async () => {
mockQuery.mockResolvedValue({ rows: [{ id: 1 }] });
await recordDelivery({ provider: 'linear', externalId: 'evt_2' });
const [sql] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('INSERT INTO ingest_events');
expect(sql).toContain('ON CONFLICT (provider, external_id) DO NOTHING');
expect(sql).toContain('RETURNING id');
});
it('should bind the provider, external id and integration id', async () => {
mockQuery.mockResolvedValue({ rows: [{ id: 7 }] });
await recordDelivery({ provider: 'github', externalId: 'evt_3', integrationId: 12 });
const [, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(params).toEqual(['github', 'evt_3', 12]);
});
it('should bind null when no integration id is supplied', async () => {
mockQuery.mockResolvedValue({ rows: [{ id: 8 }] });
await recordDelivery({ provider: 'rest', externalId: 'evt_4' });
const [, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(params[2]).toBeNull();
});
});
describe('markProcessing', () => {
it('should set the status to processing and increment attempts', async () => {
mockQuery.mockResolvedValue({ rows: [] });
await markProcessing(5);
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain("status = 'processing'");
expect(sql).toContain('attempts = attempts + 1');
expect(params).toEqual([5]);
});
});
describe('markDone', () => {
it('should set the status to done and clear the last error', async () => {
mockQuery.mockResolvedValue({ rows: [] });
await markDone(9);
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain("status = 'done'");
expect(sql).toContain('last_error = NULL');
expect(params).toEqual([9]);
});
});
describe('markFailed', () => {
it('should store the failure message against the row', async () => {
mockQuery.mockResolvedValue({ rows: [] });
await markFailed(3, 'normalizer threw');
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain("status = 'failed'");
expect(sql).toContain('last_error = $2');
expect(params).toEqual([3, 'normalizer threw']);
});
it('should truncate an over-long error to 2000 characters', async () => {
mockQuery.mockResolvedValue({ rows: [] });
await markFailed(3, 'x'.repeat(5000));
const [, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(params[1]).toBe('x'.repeat(2000));
});
});
describe('resetStaleProcessing', () => {
it('should return the number of rows returned to pending', async () => {
mockQuery.mockResolvedValue({ rows: [], rowCount: 4 });
const result = await resetStaleProcessing(30_000);
expect(result).toBe(4);
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain("SET status = 'pending'");
expect(sql).toContain("status = 'processing'");
expect(params).toEqual(['30000']);
});
it('should return 0 when the driver reports a null row count', async () => {
mockQuery.mockResolvedValue({ rows: [], rowCount: null });
const result = await resetStaleProcessing(30_000);
expect(result).toBe(0);
});
});
});

View File

@@ -0,0 +1,271 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
const { mockQuery, mockConnect } = vi.hoisted(() => ({
mockQuery: vi.fn(),
mockConnect: vi.fn(),
}));
vi.mock('../db/index.js', () => ({
query: mockQuery,
getPool: () => ({ connect: mockConnect }),
}));
import {
findByApiKeyHash,
findByProviderWorkspace,
upsertInstall,
updateTokens,
getAccessToken,
getRefreshToken,
getSigningSecret,
} from '../db/integrations.dao.js';
import { encryptSecret } from '../lib/crypto.js';
const TEST_KEY = Buffer.alloc(32, 0x2b).toString('base64');
const RAW_ROW = {
id: 3,
provider: 'slack',
external_workspace_id: 'T123',
display_name: 'Acme',
api_key_hash: 'abc123',
access_token_ciphertext: 'iv:tag:payload',
refresh_token_ciphertext: 'iv:tag:payload',
signing_secret_ciphertext: 'iv:tag:payload',
token_expires_at: new Date('2030-01-01T00:00:00.000Z'),
scopes: ['channels:history'],
};
describe('integrations.dao', () => {
let originalKey: string | undefined;
beforeEach(() => {
vi.clearAllMocks();
originalKey = process.env.TOKEN_ENCRYPTION_KEY;
process.env.TOKEN_ENCRYPTION_KEY = TEST_KEY;
});
afterEach(() => {
if (originalKey === undefined) {
delete process.env.TOKEN_ENCRYPTION_KEY;
} else {
process.env.TOKEN_ENCRYPTION_KEY = originalKey;
}
});
describe('findByApiKeyHash', () => {
it('should return the integration mapped to camelCase fields', async () => {
mockQuery.mockResolvedValue({ rows: [RAW_ROW] });
const result = await findByApiKeyHash('abc123');
expect(result).toEqual({
id: 3,
provider: 'slack',
externalWorkspaceId: 'T123',
displayName: 'Acme',
scopes: ['channels:history'],
tokenExpiresAt: new Date('2030-01-01T00:00:00.000Z'),
});
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('WHERE api_key_hash = $1');
expect(params).toEqual(['abc123']);
});
it('should not expose any ciphertext or token field on the returned row', async () => {
mockQuery.mockResolvedValue({ rows: [RAW_ROW] });
const result = await findByApiKeyHash('abc123');
expect(result).not.toBeNull();
expect(result).not.toHaveProperty('access_token_ciphertext');
expect(result).not.toHaveProperty('refresh_token_ciphertext');
expect(result).not.toHaveProperty('signing_secret_ciphertext');
expect(result).not.toHaveProperty('accessToken');
expect(result).not.toHaveProperty('api_key_hash');
expect(Object.keys(result ?? {}).sort()).toEqual([
'displayName',
'externalWorkspaceId',
'id',
'provider',
'scopes',
'tokenExpiresAt',
]);
});
it('should default scopes to an empty array when the column is null', async () => {
mockQuery.mockResolvedValue({ rows: [{ ...RAW_ROW, scopes: null }] });
const result = await findByApiKeyHash('abc123');
expect(result?.scopes).toEqual([]);
});
it('should return null when no integration matches the hash', async () => {
mockQuery.mockResolvedValue({ rows: [] });
const result = await findByApiKeyHash('nope');
expect(result).toBeNull();
});
});
describe('findByProviderWorkspace', () => {
it('should bind the provider and external workspace id', async () => {
mockQuery.mockResolvedValue({ rows: [RAW_ROW] });
const result = await findByProviderWorkspace('slack', 'T123');
expect(result?.id).toBe(3);
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('WHERE provider = $1 AND external_workspace_id = $2');
expect(params).toEqual(['slack', 'T123']);
});
it('should return null when the workspace has no integration', async () => {
mockQuery.mockResolvedValue({ rows: [] });
const result = await findByProviderWorkspace('slack', 'T999');
expect(result).toBeNull();
});
});
describe('upsertInstall', () => {
it('should encrypt the signing secret before binding it', async () => {
mockQuery.mockResolvedValue({ rows: [RAW_ROW] });
await upsertInstall({
provider: 'slack',
externalWorkspaceId: 'T123',
displayName: 'Acme',
signingSecret: 'super-secret',
});
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('INSERT INTO integrations');
expect(sql).toContain('ON CONFLICT (provider, external_workspace_id) DO UPDATE SET');
const stored = params[4];
expect(typeof stored).toBe('string');
expect(stored).not.toBe('super-secret');
expect(String(stored).split(':')).toHaveLength(3);
});
it('should bind null when no signing secret is supplied', async () => {
mockQuery.mockResolvedValue({ rows: [RAW_ROW] });
await upsertInstall({ provider: 'slack', externalWorkspaceId: 'T123' });
const [, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(params[4]).toBeNull();
expect(params[2]).toBeNull();
expect(params[3]).toBeNull();
expect(params[5]).toEqual([]);
});
it('should return the public row for the upserted integration', async () => {
mockQuery.mockResolvedValue({ rows: [RAW_ROW] });
const result = await upsertInstall({ provider: 'slack', externalWorkspaceId: 'T123' });
expect(result.externalWorkspaceId).toBe('T123');
expect(result).not.toHaveProperty('signing_secret_ciphertext');
});
});
describe('updateTokens', () => {
it('should encrypt the access token rather than storing it in plaintext', async () => {
mockQuery.mockResolvedValue({ rows: [] });
await updateTokens({ id: 3, accessToken: 'at-plain', refreshToken: 'rt-plain' });
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('access_token_ciphertext = $2');
expect(params[0]).toBe(3);
expect(params[1]).not.toBe('at-plain');
expect(String(params[1]).split(':')).toHaveLength(3);
expect(params[2]).not.toBe('rt-plain');
});
it('should bind null for an absent refresh token and expiry', async () => {
mockQuery.mockResolvedValue({ rows: [] });
await updateTokens({ id: 3, accessToken: 'at-plain' });
const [, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(params[2]).toBeNull();
expect(params[3]).toBeNull();
});
it('should bind the supplied expiry date', async () => {
mockQuery.mockResolvedValue({ rows: [] });
const expiresAt = new Date('2031-05-05T10:00:00.000Z');
await updateTokens({ id: 3, accessToken: 'at-plain', expiresAt });
const [, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(params[3]).toEqual(expiresAt);
});
});
describe('getAccessToken', () => {
it('should decrypt the stored ciphertext back to the original token', async () => {
mockQuery.mockResolvedValue({ rows: [{ value: encryptSecret('at-plain') }] });
const result = await getAccessToken(3);
expect(result).toBe('at-plain');
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('SELECT access_token_ciphertext AS value');
expect(params).toEqual([3]);
});
it('should return null when no access token is stored', async () => {
mockQuery.mockResolvedValue({ rows: [{ value: null }] });
const result = await getAccessToken(3);
expect(result).toBeNull();
});
it('should return null when the integration does not exist', async () => {
mockQuery.mockResolvedValue({ rows: [] });
const result = await getAccessToken(404);
expect(result).toBeNull();
});
});
describe('getRefreshToken', () => {
it('should decrypt the refresh token column', async () => {
mockQuery.mockResolvedValue({ rows: [{ value: encryptSecret('rt-plain') }] });
const result = await getRefreshToken(3);
expect(result).toBe('rt-plain');
const [sql] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('SELECT refresh_token_ciphertext AS value');
});
});
describe('getSigningSecret', () => {
it('should decrypt the signing secret column', async () => {
mockQuery.mockResolvedValue({ rows: [{ value: encryptSecret('shh') }] });
const result = await getSigningSecret(3);
expect(result).toBe('shh');
const [sql] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('SELECT signing_secret_ciphertext AS value');
});
it('should return null when no signing secret is stored', async () => {
mockQuery.mockResolvedValue({ rows: [{ value: null }] });
const result = await getSigningSecret(3);
expect(result).toBeNull();
});
});
});

View File

@@ -0,0 +1,340 @@
import { describe, it, expect } from 'vitest';
import {
NormalizationError,
isNoteInputArray,
normalizeLinear,
normalizeRest,
} from '../config/normalizers.js';
import { normalize, hasNormalizer } from '../services/normalize.service.js';
import type { NoteInput } from '../types/domain.js';
const note = (overrides: Record<string, unknown> = {}): Record<string, unknown> => ({
id: 'n1',
text: 'Deploys are scary',
author: 'kim',
...overrides,
});
const sourceMeta = (input: NoteInput): Record<string, unknown> => {
expect(input.sourceMeta).toBeDefined();
return input.sourceMeta as Record<string, unknown>;
};
const linearPayload = (
data: Record<string, unknown> = {}
): Record<string, unknown> => ({
action: 'create',
type: 'Comment',
data: {
id: 'cmt_9f2',
body: 'Retros keep surfacing the same deploy pain',
url: 'https://linear.app/acme/issue/ENG-42#comment-cmt_9f2',
user: { name: 'dana' },
...data,
},
});
describe('isNoteInputArray', () => {
it('should accept an empty array', () => {
expect(isNoteInputArray([])).toBe(true);
});
it('should accept an array of well-formed notes', () => {
expect(isNoteInputArray([note(), note({ id: 'n2', x: 1, y: 2, color: '#fff' })])).toBe(true);
});
it('should reject a non-array', () => {
expect(isNoteInputArray(undefined)).toBe(false);
expect(isNoteInputArray(null)).toBe(false);
expect(isNoteInputArray('notes')).toBe(false);
expect(isNoteInputArray({ 0: note(), length: 1 })).toBe(false);
});
it('should reject an element that is not an object', () => {
expect(isNoteInputArray([note(), 'nope'])).toBe(false);
expect(isNoteInputArray([null])).toBe(false);
});
it('should reject an element missing a required field', () => {
expect(isNoteInputArray([note({ id: undefined })])).toBe(false);
expect(isNoteInputArray([note({ text: undefined })])).toBe(false);
expect(isNoteInputArray([note({ author: undefined })])).toBe(false);
});
it('should reject an element whose required field has the wrong type', () => {
expect(isNoteInputArray([note({ id: 7 })])).toBe(false);
expect(isNoteInputArray([note({ text: { body: 'hi' } })])).toBe(false);
expect(isNoteInputArray([note({ author: ['kim'] })])).toBe(false);
});
it('should reject an element with empty strings', () => {
expect(isNoteInputArray([note({ id: '' })])).toBe(false);
expect(isNoteInputArray([note({ text: '' })])).toBe(false);
expect(isNoteInputArray([note({ author: '' })])).toBe(false);
});
it('should reject an element whose optional field has the wrong type', () => {
expect(isNoteInputArray([note({ x: '10' })])).toBe(false);
expect(isNoteInputArray([note({ y: null })])).toBe(false);
expect(isNoteInputArray([note({ color: 0xffffff })])).toBe(false);
});
it('should accept an element whose optional fields are undefined', () => {
expect(isNoteInputArray([note({ x: undefined, y: undefined, color: undefined })])).toBe(true);
});
});
describe('normalizeRest', () => {
it('should return the supplied notes with provenance attached', () => {
const result = normalizeRest({
batchId: 'batch_7',
notes: [note(), note({ id: 'n2', text: 'Standups run long' })],
});
expect(result.externalId).toBe('batch_7');
expect(result.notes).toHaveLength(2);
expect(result.notes[0].id).toBe('n1');
expect(result.notes[0].text).toBe('Deploys are scary');
expect(result.notes[0].author).toBe('kim');
expect(sourceMeta(result.notes[0])).toMatchObject({
provider: 'rest',
externalId: 'batch_7',
});
expect(sourceMeta(result.notes[1]).externalId).toBe('batch_7');
});
it('should record an ISO receivedAt timestamp in provenance', () => {
const result = normalizeRest({ notes: [note()] });
const receivedAt = String(sourceMeta(result.notes[0]).receivedAt);
expect(new Date(receivedAt).toISOString()).toBe(receivedAt);
});
it('should preserve optional positional and color fields', () => {
const result = normalizeRest({
notes: [note({ x: 12, y: -3, color: '#ffcc00' })],
});
expect(result.notes[0]).toMatchObject({ x: 12, y: -3, color: '#ffcc00' });
});
it('should generate a non-empty externalId when batchId is absent', () => {
const result = normalizeRest({ notes: [note()] });
expect(result.externalId.length).toBeGreaterThan(0);
expect(sourceMeta(result.notes[0]).externalId).toBe(result.externalId);
});
it('should generate a distinct externalId per call', () => {
const first = normalizeRest({ notes: [note()] });
const second = normalizeRest({ notes: [note()] });
expect(first.externalId).not.toBe(second.externalId);
});
it('should ignore an empty batchId in favour of a generated one', () => {
const result = normalizeRest({ batchId: '', notes: [note()] });
expect(result.externalId.length).toBeGreaterThan(0);
});
it('should ignore a non-string batchId in favour of a generated one', () => {
const result = normalizeRest({ batchId: 42, notes: [note()] });
expect(result.externalId).not.toBe('42');
expect(result.externalId.length).toBeGreaterThan(0);
});
it('should accept an empty notes array', () => {
const result = normalizeRest({ batchId: 'batch_empty', notes: [] });
expect(result).toEqual({ externalId: 'batch_empty', notes: [] });
});
it('should throw when notes is missing', () => {
expect(() => normalizeRest({ batchId: 'batch_7' })).toThrow(NormalizationError);
expect(() => normalizeRest({})).toThrow(/"notes" array/);
});
it('should throw when notes is not an array', () => {
expect(() => normalizeRest({ notes: 'one note' })).toThrow(NormalizationError);
expect(() => normalizeRest({ notes: { id: 'n1' } })).toThrow(NormalizationError);
});
it('should throw when a note is missing text', () => {
expect(() => normalizeRest({ notes: [note({ text: undefined })] })).toThrow(
NormalizationError
);
});
it('should throw when a note id is not a string', () => {
expect(() => normalizeRest({ notes: [note({ id: 99 })] })).toThrow(NormalizationError);
});
it('should throw when a note carries empty strings', () => {
expect(() => normalizeRest({ notes: [note({ text: '' })] })).toThrow(NormalizationError);
expect(() => normalizeRest({ notes: [note({ id: '' })] })).toThrow(NormalizationError);
expect(() => normalizeRest({ notes: [note({ author: '' })] })).toThrow(NormalizationError);
});
it('should throw when one note in an otherwise valid batch is invalid', () => {
expect(() => normalizeRest({ notes: [note(), note({ id: 'n2', author: '' })] })).toThrow(
NormalizationError
);
});
it('should throw for a non-object payload', () => {
expect(() => normalizeRest(null)).toThrow(/not an object/);
expect(() => normalizeRest(undefined)).toThrow(NormalizationError);
expect(() => normalizeRest('notes')).toThrow(NormalizationError);
expect(() => normalizeRest([note()])).toThrow(/not an object/);
});
it('should overwrite any caller-supplied sourceMeta with real provenance', () => {
const result = normalizeRest({
batchId: 'batch_7',
notes: [note({ sourceMeta: { provider: 'slack', externalId: 'spoofed' } })],
});
expect(sourceMeta(result.notes[0])).toMatchObject({
provider: 'rest',
externalId: 'batch_7',
});
});
});
describe('normalizeLinear', () => {
it('should map a comment webhook to a single note', () => {
const result = normalizeLinear(linearPayload());
expect(result.externalId).toBe('cmt_9f2');
expect(result.notes).toHaveLength(1);
expect(result.notes[0].id).toBe('linear_cmt_9f2');
expect(result.notes[0].text).toBe('Retros keep surfacing the same deploy pain');
expect(result.notes[0].author).toBe('dana');
});
it('should attach provenance including provider, externalId and permalink', () => {
const meta = sourceMeta(normalizeLinear(linearPayload()).notes[0]);
expect(meta).toMatchObject({
provider: 'linear',
externalId: 'cmt_9f2',
permalink: 'https://linear.app/acme/issue/ENG-42#comment-cmt_9f2',
authorHandle: 'dana',
});
expect(typeof meta.receivedAt).toBe('string');
});
it('should fall back to an unknown author when the user is missing or unnamed', () => {
expect(normalizeLinear(linearPayload({ user: undefined })).notes[0].author).toBe('unknown');
expect(normalizeLinear(linearPayload({ user: null })).notes[0].author).toBe('unknown');
expect(normalizeLinear(linearPayload({ user: {} })).notes[0].author).toBe('unknown');
expect(normalizeLinear(linearPayload({ user: { name: '' } })).notes[0].author).toBe('unknown');
expect(normalizeLinear(linearPayload({ user: 'dana' })).notes[0].author).toBe('unknown');
});
it('should omit the permalink when url is absent', () => {
const meta = sourceMeta(normalizeLinear(linearPayload({ url: undefined })).notes[0]);
expect(meta.permalink).toBeUndefined();
});
it('should throw when data.id is missing', () => {
expect(() => normalizeLinear(linearPayload({ id: undefined }))).toThrow(NormalizationError);
expect(() => normalizeLinear(linearPayload({ id: '' }))).toThrow(/missing data.id/);
expect(() => normalizeLinear(linearPayload({ id: 42 }))).toThrow(NormalizationError);
});
it('should throw when data itself is missing or not an object', () => {
expect(() => normalizeLinear({ action: 'create', type: 'Comment' })).toThrow(
/not an object/
);
expect(() => normalizeLinear({ data: 'cmt_9f2' })).toThrow(NormalizationError);
expect(() => normalizeLinear({ data: [] })).toThrow(NormalizationError);
});
it('should return zero notes but keep the externalId when the body is empty', () => {
expect(normalizeLinear(linearPayload({ body: '' }))).toEqual({
externalId: 'cmt_9f2',
notes: [],
});
});
it('should return zero notes but keep the externalId when the body is absent', () => {
expect(normalizeLinear(linearPayload({ body: undefined }))).toEqual({
externalId: 'cmt_9f2',
notes: [],
});
expect(normalizeLinear(linearPayload({ body: null }))).toEqual({
externalId: 'cmt_9f2',
notes: [],
});
});
it('should throw for a non-object payload', () => {
expect(() => normalizeLinear(null)).toThrow(/not an object/);
expect(() => normalizeLinear('cmt_9f2')).toThrow(NormalizationError);
expect(() => normalizeLinear(7)).toThrow(NormalizationError);
expect(() => normalizeLinear([linearPayload()])).toThrow(/not an object/);
});
it('should truncate a generated note id to 64 characters', () => {
const longId = 'x'.repeat(400);
const result = normalizeLinear(linearPayload({ id: longId }));
expect(result.externalId).toBe(longId);
expect(result.notes[0].id).toHaveLength(64);
expect(result.notes[0].id).toBe(`linear_${longId}`.slice(0, 64));
expect(sourceMeta(result.notes[0]).externalId).toBe(longId);
});
it('should produce notes accepted by the note input guard', () => {
expect(isNoteInputArray(normalizeLinear(linearPayload()).notes)).toBe(true);
});
});
describe('normalize', () => {
it('should delegate rest deliveries to the rest normalizer', () => {
const result = normalize('rest', { batchId: 'batch_7', notes: [note()] });
expect(result.externalId).toBe('batch_7');
expect(result.notes[0]).toMatchObject({ id: 'n1', author: 'kim' });
expect(sourceMeta(result.notes[0])).toMatchObject({
provider: 'rest',
externalId: 'batch_7',
});
});
it('should delegate linear deliveries to the linear normalizer', () => {
const result = normalize('linear', linearPayload());
expect(result.externalId).toBe('cmt_9f2');
expect(result.notes[0].id).toBe('linear_cmt_9f2');
});
it('should propagate normalizer failures unchanged', () => {
expect(() => normalize('rest', {})).toThrow(NormalizationError);
expect(() => normalize('linear', {})).toThrow(NormalizationError);
});
it('should throw for a provider with no registered normalizer', () => {
expect(() => normalize('github', {})).toThrow(NormalizationError);
expect(() => normalize('github', {})).toThrow(/No normalizer registered/);
expect(() => normalize('jira', {})).toThrow(/No normalizer registered/);
expect(() => normalize('slack', {})).toThrow(/No normalizer registered/);
});
});
describe('hasNormalizer', () => {
it('should be true for providers with a normalizer', () => {
expect(hasNormalizer('rest')).toBe(true);
expect(hasNormalizer('linear')).toBe(true);
});
it('should be false for providers awaiting an integration', () => {
expect(hasNormalizer('github')).toBe(false);
expect(hasNormalizer('jira')).toBe(false);
expect(hasNormalizer('slack')).toBe(false);
});
});

View File

@@ -0,0 +1,217 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { Readable } from 'node:stream';
import type { Note, NoteInput } from '../types/domain.js';
const { mockQuery, mockConnect } = vi.hoisted(() => ({
mockQuery: vi.fn(),
mockConnect: vi.fn(),
}));
vi.mock('../db/index.js', () => ({
query: mockQuery,
getPool: () => ({ connect: mockConnect }),
}));
import {
getAllNotes,
streamAllNotes,
getNoteById,
createNote,
createNotes,
} from '../db/notes.dao.js';
const MOCK_ROWS: Note[] = [
{ id: 'note_001', text: 'Login flow feels confusing', x: 193, y: 191, author: 'user_5', color: 'yellow' },
{ id: 'note_002', text: 'Login flow is broken on mobile', x: 214, y: 281, author: 'user_9', color: 'yellow' },
];
describe('notes.dao', () => {
beforeEach(() => {
vi.clearAllMocks();
});
describe('getAllNotes', () => {
it('should return all notes ordered by id', async () => {
mockQuery.mockResolvedValue({ rows: MOCK_ROWS });
const result = await getAllNotes();
expect(result).toEqual(MOCK_ROWS);
expect(mockQuery).toHaveBeenCalledWith(
'SELECT id, text, x, y, author, color FROM notes ORDER BY id'
);
});
it('should return an empty array when no notes exist', async () => {
mockQuery.mockResolvedValue({ rows: [] });
const result = await getAllNotes();
expect(result).toEqual([]);
});
it('should propagate database errors', async () => {
mockQuery.mockRejectedValue(new Error('connection refused'));
await expect(getAllNotes()).rejects.toThrow('connection refused');
});
});
describe('streamAllNotes', () => {
const mockClient = (rows: Note[]) => {
const release = vi.fn();
const client = {
release,
query: vi.fn(() => Readable.from(rows, { objectMode: true })),
};
mockConnect.mockResolvedValue(client);
return { client, release };
};
it('should stream rows without buffering them into an array', async () => {
mockClient(MOCK_ROWS);
const stream = await streamAllNotes();
const received: Note[] = [];
for await (const row of stream) received.push(row as Note);
expect(received).toEqual(MOCK_ROWS);
});
it('should release the pooled client once the stream ends', async () => {
const { release } = mockClient(MOCK_ROWS);
const stream = await streamAllNotes();
for await (const row of stream) void row;
expect(release).toHaveBeenCalledOnce();
});
it('should release the pooled client when a consumer destroys the stream early', async () => {
const { release } = mockClient(MOCK_ROWS);
const stream = await streamAllNotes();
stream.destroy();
await new Promise((resolve) => stream.once('close', resolve));
expect(release).toHaveBeenCalledOnce();
});
it('should release the pooled client when starting the query throws', async () => {
const release = vi.fn();
mockConnect.mockResolvedValue({
release,
query: vi.fn(() => { throw new Error('cursor failed'); }),
});
await expect(streamAllNotes()).rejects.toThrow('cursor failed');
expect(release).toHaveBeenCalledOnce();
});
});
describe('getNoteById', () => {
it('should return a single note when found', async () => {
mockQuery.mockResolvedValue({ rows: [MOCK_ROWS[0]] });
const result = await getNoteById('note_001');
expect(result).toEqual(MOCK_ROWS[0]);
expect(mockQuery).toHaveBeenCalledWith(
'SELECT id, text, x, y, author, color FROM notes WHERE id = $1',
['note_001']
);
});
it('should return null when note is not found', async () => {
mockQuery.mockResolvedValue({ rows: [] });
const result = await getNoteById('note_999');
expect(result).toBeNull();
});
});
describe('createNote', () => {
it('should insert a note and return it', async () => {
const input: NoteInput = { id: 'note_003', text: 'Export fails', x: 100, y: 200, author: 'user_1', color: 'blue' };
mockQuery.mockResolvedValue({ rows: [input] });
const result = await createNote(input);
expect(result).toEqual(input);
expect(mockQuery).toHaveBeenCalledOnce();
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('INSERT INTO notes');
expect(params).toEqual(['note_003', 'Export fails', 100, 200, 'user_1', 'blue', '{}']);
});
it('should serialize provenance into source_meta', async () => {
const input: NoteInput = {
id: 'note_005',
text: 'From Slack',
author: 'user_3',
sourceMeta: { provider: 'slack', externalId: 'msg_1' },
};
mockQuery.mockResolvedValue({ rows: [input] });
await createNote(input);
const params = mockQuery.mock.calls[0][1] as unknown[];
expect(params[6]).toBe('{"provider":"slack","externalId":"msg_1"}');
});
it('should use defaults for missing x, y, and color', async () => {
const input: NoteInput = { id: 'note_004', text: 'Needs fixing', author: 'user_2' };
mockQuery.mockResolvedValue({ rows: [{ ...input, x: 0, y: 0, color: 'yellow' }] });
await createNote(input);
const params = mockQuery.mock.calls[0][1] as unknown[];
expect(params[2]).toBe(0);
expect(params[3]).toBe(0);
expect(params[5]).toBe('yellow');
});
});
describe('createNotes', () => {
it('should batch-insert multiple notes and return them', async () => {
mockQuery.mockResolvedValue({ rows: MOCK_ROWS });
const result = await createNotes(MOCK_ROWS);
expect(result).toEqual(MOCK_ROWS);
expect(mockQuery).toHaveBeenCalledOnce();
const [sql, params] = mockQuery.mock.calls[0] as [string, unknown[]];
expect(sql).toContain('INSERT INTO notes');
expect(sql).toContain('ON CONFLICT (id) DO NOTHING');
expect(params).toHaveLength(14);
});
it('should return an empty array without querying when given no notes', async () => {
const result = await createNotes([]);
expect(result).toEqual([]);
expect(mockQuery).not.toHaveBeenCalled();
});
it('should split large inputs into multiple statements under the bind-parameter limit', async () => {
const many: NoteInput[] = Array.from({ length: 2500 }, (_, i) => ({
id: `note_${i}`,
text: `text ${i}`,
author: 'user_1',
}));
mockQuery.mockImplementation(async (_sql: string, params: unknown[]) => ({
rows: new Array(params.length / 7).fill(null).map((_, i) => ({ i })),
}));
const result = await createNotes(many);
expect(mockQuery).toHaveBeenCalledTimes(3);
expect(result).toHaveLength(2500);
for (const [, params] of mockQuery.mock.calls as [string, unknown[]][]) {
expect(params.length).toBeLessThan(65535);
}
});
});
});

View File

@@ -0,0 +1,284 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
const {
mockRequestJson,
mockFindByProviderWorkspace,
mockGetAccessToken,
mockGetRefreshToken,
mockUpdateTokens,
} = vi.hoisted(() => ({
mockRequestJson: vi.fn(),
mockFindByProviderWorkspace: vi.fn(),
mockGetAccessToken: vi.fn(),
mockGetRefreshToken: vi.fn(),
mockUpdateTokens: vi.fn(),
}));
vi.mock('../lib/httpClient.js', () => ({
requestJson: mockRequestJson,
}));
vi.mock('../db/integrations.dao.js', () => ({
findByProviderWorkspace: mockFindByProviderWorkspace,
getAccessToken: mockGetAccessToken,
getRefreshToken: mockGetRefreshToken,
updateTokens: mockUpdateTokens,
}));
import {
buildAuthorizeUrl,
exchangeCode,
refreshAccessToken,
getValidAccessToken,
OAuthError,
} from '../services/oauth.service.js';
import type { IntegrationRow } from '../types/integration.js';
const ENV_KEYS = [
'LINEAR_CLIENT_ID',
'LINEAR_CLIENT_SECRET',
'SLACK_CLIENT_ID',
'SLACK_CLIENT_SECRET',
] as const;
const integration = (overrides: Partial<IntegrationRow> = {}): IntegrationRow => ({
id: 11,
provider: 'linear',
externalWorkspaceId: 'ws_1',
displayName: 'Acme',
scopes: ['read'],
tokenExpiresAt: null,
...overrides,
});
describe('oauth.service', () => {
const originalEnv = new Map<string, string | undefined>();
beforeEach(() => {
vi.clearAllMocks();
for (const key of ENV_KEYS) originalEnv.set(key, process.env[key]);
process.env.LINEAR_CLIENT_ID = 'client-id-1';
process.env.LINEAR_CLIENT_SECRET = 'client-secret-1';
process.env.SLACK_CLIENT_ID = 'slack-id';
process.env.SLACK_CLIENT_SECRET = 'slack-secret';
});
afterEach(() => {
for (const key of ENV_KEYS) {
const value = originalEnv.get(key);
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
});
describe('buildAuthorizeUrl', () => {
it('should build an authorize url against the provider endpoint with all required params', () => {
const url = new URL(
buildAuthorizeUrl('linear', { redirectUri: 'https://app.test/cb', state: 'st_1' })
);
expect(`${url.origin}${url.pathname}`).toBe('https://linear.app/oauth/authorize');
expect(url.searchParams.get('client_id')).toBe('client-id-1');
expect(url.searchParams.get('redirect_uri')).toBe('https://app.test/cb');
expect(url.searchParams.get('response_type')).toBe('code');
expect(url.searchParams.get('state')).toBe('st_1');
expect(url.searchParams.get('scope')).toBe('read');
});
it('should join multiple configured scopes with spaces', () => {
const url = new URL(
buildAuthorizeUrl('slack', { redirectUri: 'https://app.test/cb', state: 'st_2' })
);
expect(url.searchParams.get('scope')).toBe('channels:history reactions:read users:read');
});
it('should throw an OAuthError when client credentials are not configured', () => {
delete process.env.LINEAR_CLIENT_SECRET;
expect(() =>
buildAuthorizeUrl('linear', { redirectUri: 'https://app.test/cb', state: 'st_1' })
).toThrow(OAuthError);
});
it('should throw an OAuthError for a provider without OAuth support', () => {
expect(() =>
buildAuthorizeUrl('rest', { redirectUri: 'https://app.test/cb', state: 'st_1' })
).toThrow(/does not support OAuth/);
});
});
describe('exchangeCode', () => {
it('should post a form-encoded authorization_code grant to the token endpoint', async () => {
mockRequestJson.mockResolvedValue({ access_token: 'at_1' });
await exchangeCode('linear', { code: 'code_1', redirectUri: 'https://app.test/cb' });
const [url, init] = mockRequestJson.mock.calls[0] as [string, RequestInit];
expect(url).toBe('https://api.linear.app/oauth/token');
expect(init.method).toBe('POST');
expect(init.headers).toMatchObject({
'content-type': 'application/x-www-form-urlencoded',
});
const form = new URLSearchParams(String(init.body));
expect(form.get('grant_type')).toBe('authorization_code');
expect(form.get('code')).toBe('code_1');
expect(form.get('redirect_uri')).toBe('https://app.test/cb');
expect(form.get('client_id')).toBe('client-id-1');
expect(form.get('client_secret')).toBe('client-secret-1');
});
it('should map the token response into a TokenSet', async () => {
mockRequestJson.mockResolvedValue({
access_token: 'at_1',
refresh_token: 'rt_1',
expires_in: 3600,
scope: 'read write',
});
const before = Date.now();
const tokens = await exchangeCode('linear', {
code: 'code_1',
redirectUri: 'https://app.test/cb',
});
expect(tokens.accessToken).toBe('at_1');
expect(tokens.refreshToken).toBe('rt_1');
expect(tokens.scopes).toEqual(['read', 'write']);
expect(tokens.expiresAt).toBeInstanceOf(Date);
expect(tokens.expiresAt?.getTime()).toBeGreaterThanOrEqual(before + 3_600_000);
expect(tokens.expiresAt?.getTime()).toBeLessThanOrEqual(Date.now() + 3_600_000);
});
it('should leave expiresAt undefined and scopes empty when the response omits them', async () => {
mockRequestJson.mockResolvedValue({ access_token: 'at_1' });
const tokens = await exchangeCode('linear', {
code: 'code_1',
redirectUri: 'https://app.test/cb',
});
expect(tokens.expiresAt).toBeUndefined();
expect(tokens.scopes).toEqual([]);
});
it('should throw an OAuthError when the payload carries no usable access token', async () => {
mockRequestJson.mockResolvedValue({ token_type: 'bearer' });
await expect(
exchangeCode('linear', { code: 'code_1', redirectUri: 'https://app.test/cb' })
).rejects.toThrow(OAuthError);
});
it('should throw an OAuthError when credentials are missing', async () => {
delete process.env.LINEAR_CLIENT_ID;
await expect(
exchangeCode('linear', { code: 'code_1', redirectUri: 'https://app.test/cb' })
).rejects.toThrow(/LINEAR_CLIENT_ID/);
expect(mockRequestJson).not.toHaveBeenCalled();
});
});
describe('refreshAccessToken', () => {
it('should post a refresh_token grant carrying the stored refresh token', async () => {
mockRequestJson.mockResolvedValue({ access_token: 'at_2' });
const tokens = await refreshAccessToken('linear', 'rt_1');
expect(tokens.accessToken).toBe('at_2');
const [, init] = mockRequestJson.mock.calls[0] as [string, RequestInit];
const form = new URLSearchParams(String(init.body));
expect(form.get('grant_type')).toBe('refresh_token');
expect(form.get('refresh_token')).toBe('rt_1');
});
});
describe('getValidAccessToken', () => {
it('should return the stored access token when it is not near expiry', async () => {
mockFindByProviderWorkspace.mockResolvedValue(
integration({ tokenExpiresAt: new Date(Date.now() + 3_600_000) })
);
mockGetAccessToken.mockResolvedValue('at_stored');
const result = await getValidAccessToken('linear', 'ws_1');
expect(result).toBe('at_stored');
expect(mockRequestJson).not.toHaveBeenCalled();
expect(mockUpdateTokens).not.toHaveBeenCalled();
});
it('should return the stored access token when no expiry is recorded', async () => {
mockFindByProviderWorkspace.mockResolvedValue(integration());
mockGetAccessToken.mockResolvedValue('at_stored');
const result = await getValidAccessToken('linear', 'ws_1');
expect(result).toBe('at_stored');
expect(mockGetAccessToken).toHaveBeenCalledWith(11);
});
it('should refresh and persist the new token when expiry is within the refresh margin', async () => {
mockFindByProviderWorkspace.mockResolvedValue(
integration({ tokenExpiresAt: new Date(Date.now() + 30_000) })
);
mockGetRefreshToken.mockResolvedValue('rt_1');
mockRequestJson.mockResolvedValue({
access_token: 'at_refreshed',
refresh_token: 'rt_2',
expires_in: 3600,
});
const result = await getValidAccessToken('linear', 'ws_1');
expect(result).toBe('at_refreshed');
expect(mockUpdateTokens).toHaveBeenCalledWith(
expect.objectContaining({ id: 11, accessToken: 'at_refreshed', refreshToken: 'rt_2' })
);
expect(mockGetAccessToken).not.toHaveBeenCalled();
});
it('should refresh when the token has already expired', async () => {
mockFindByProviderWorkspace.mockResolvedValue(
integration({ tokenExpiresAt: new Date(Date.now() - 1_000) })
);
mockGetRefreshToken.mockResolvedValue('rt_1');
mockRequestJson.mockResolvedValue({ access_token: 'at_refreshed' });
const result = await getValidAccessToken('linear', 'ws_1');
expect(result).toBe('at_refreshed');
});
it('should throw an OAuthError when the workspace has no integration', async () => {
mockFindByProviderWorkspace.mockResolvedValue(null);
await expect(getValidAccessToken('linear', 'ws_missing')).rejects.toThrow(OAuthError);
await expect(getValidAccessToken('linear', 'ws_missing')).rejects.toThrow(
/No linear integration for workspace ws_missing/
);
});
it('should throw an OAuthError when a refresh is needed but no refresh token is stored', async () => {
mockFindByProviderWorkspace.mockResolvedValue(
integration({ tokenExpiresAt: new Date(Date.now() + 1_000) })
);
mockGetRefreshToken.mockResolvedValue(null);
await expect(getValidAccessToken('linear', 'ws_1')).rejects.toThrow(
/no refresh token is stored/
);
expect(mockUpdateTokens).not.toHaveBeenCalled();
});
it('should throw an OAuthError when no access token is stored', async () => {
mockFindByProviderWorkspace.mockResolvedValue(integration());
mockGetAccessToken.mockResolvedValue(null);
await expect(getValidAccessToken('linear', 'ws_1')).rejects.toThrow(
/No access token stored for linear/
);
});
});
});

215
backend/tests/queue.test.ts Normal file
View File

@@ -0,0 +1,215 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { enqueue, size, drain, configureQueue } from '../lib/queue.js';
type Deferred = { promise: Promise<void>; resolve: () => void };
const deferred = (): Deferred => {
let resolve!: () => void;
const promise = new Promise<void>((res) => {
resolve = res;
});
return { promise, resolve };
};
// A macrotask boundary: enough for the queue to pick up work it deferred.
const flush = (): Promise<void> =>
new Promise((resolve) => {
setTimeout(resolve, 0);
});
describe('queue', () => {
let errors: ReturnType<typeof vi.spyOn>;
beforeEach(() => {
configureQueue({ maxAttempts: 3, baseDelayMs: 0 });
errors = vi.spyOn(console, 'error').mockImplementation(() => {});
});
afterEach(async () => {
await drain();
vi.useRealTimers();
vi.restoreAllMocks();
});
it('should run jobs one at a time in FIFO order', async () => {
const order: string[] = [];
const record = (id: string) => async (): Promise<void> => {
order.push(`${id}:start`);
await Promise.resolve();
order.push(`${id}:end`);
};
enqueue('a', record('a'));
enqueue('b', record('b'));
enqueue('c', record('c'));
await drain();
expect(order).toEqual([
'a:start', 'a:end',
'b:start', 'b:end',
'c:start', 'c:end',
]);
});
it('should not throw synchronously when a job throws synchronously', async () => {
const thrower = (): Promise<void> => {
throw new Error('sync boom');
};
expect(() => enqueue('sync-thrower', thrower)).not.toThrow();
await drain();
expect(errors).toHaveBeenCalled();
});
it('should retry a failing job up to the configured cap and then give up', async () => {
configureQueue({ maxAttempts: 4 });
let attempts = 0;
enqueue('always-failing', async () => {
attempts += 1;
throw new Error('boom');
});
await drain();
expect(attempts).toBe(4);
expect(errors).toHaveBeenCalledOnce();
expect(String(errors.mock.calls[0]?.[0])).toContain('always-failing');
});
it('should stop retrying as soon as an attempt succeeds', async () => {
let attempts = 0;
enqueue('flaky', async () => {
attempts += 1;
if (attempts < 2) throw new Error('transient');
});
await drain();
expect(attempts).toBe(2);
expect(errors).not.toHaveBeenCalled();
});
it('should keep running later jobs after one fails permanently', async () => {
const completed: string[] = [];
enqueue('doomed', async () => {
throw new Error('boom');
});
enqueue('survivor', async () => {
completed.push('survivor');
});
await drain();
expect(completed).toEqual(['survivor']);
});
it('should grow the backoff delay between attempts', async () => {
vi.useFakeTimers();
vi.spyOn(Math, 'random').mockReturnValue(0);
configureQueue({ maxAttempts: 4, baseDelayMs: 100 });
let attempts = 0;
enqueue('retrying', async () => {
attempts += 1;
throw new Error('boom');
});
await vi.advanceTimersByTimeAsync(0);
expect(attempts).toBe(1);
await vi.advanceTimersByTimeAsync(99);
expect(attempts).toBe(1);
await vi.advanceTimersByTimeAsync(2);
expect(attempts).toBe(2);
await vi.advanceTimersByTimeAsync(198);
expect(attempts).toBe(2);
await vi.advanceTimersByTimeAsync(2);
expect(attempts).toBe(3);
await vi.advanceTimersByTimeAsync(398);
expect(attempts).toBe(3);
await vi.advanceTimersByTimeAsync(2);
expect(attempts).toBe(4);
});
it('should apply jitter within the backoff window', async () => {
vi.useFakeTimers();
vi.spyOn(Math, 'random').mockReturnValue(0.75);
configureQueue({ maxAttempts: 2, baseDelayMs: 100 });
let attempts = 0;
enqueue('jittered', async () => {
attempts += 1;
throw new Error('boom');
});
await vi.advanceTimersByTimeAsync(174);
expect(attempts).toBe(1);
await vi.advanceTimersByTimeAsync(2);
expect(attempts).toBe(2);
});
it('should resolve drain() only once in-flight work has finished', async () => {
const gate = deferred();
let finished = false;
enqueue('blocked', async () => {
await gate.promise;
finished = true;
});
await flush();
expect(finished).toBe(false);
gate.resolve();
await drain();
expect(finished).toBe(true);
});
it('should resolve every concurrent drain() caller', async () => {
const gate = deferred();
enqueue('blocked', () => gate.promise);
const waiters = Promise.all([drain(), drain(), drain()]);
gate.resolve();
await expect(waiters).resolves.toEqual([undefined, undefined, undefined]);
});
it('should resolve drain() immediately when the queue is idle', async () => {
const winner = await Promise.race([
drain().then(() => 'drained'),
new Promise<string>((resolve) => {
setTimeout(() => resolve('timer'), 0);
}),
]);
expect(winner).toBe('drained');
});
it('should report the pending count excluding the job in flight', async () => {
const gate = deferred();
enqueue('blocked', () => gate.promise);
enqueue('second', async () => {});
enqueue('third', async () => {});
expect(size()).toBe(3);
await flush();
expect(size()).toBe(2);
gate.resolve();
await drain();
expect(size()).toBe(0);
});
});

View File

@@ -0,0 +1,304 @@
import { describe, it, expect, afterEach, vi } from 'vitest';
import { createHmac } from 'node:crypto';
import type { IncomingHttpHeaders } from 'node:http';
import { verifySignature } from '../lib/signatures.js';
import { providers } from '../config/providers.js';
import type { ProviderSlug } from '../types/integration.js';
const SECRET = 'top-secret-signing-key';
const RAW_BODY = Buffer.from(JSON.stringify({ action: 'create', id: 'c1' }), 'utf8');
const OTHER_BODY = Buffer.from(JSON.stringify({ action: 'remove', id: 'c1' }), 'utf8');
const hmacHex = (payload: string | Buffer, secret = SECRET): string =>
createHmac('sha256', secret).update(payload).digest('hex');
const verify = (
provider: ProviderSlug,
headers: IncomingHttpHeaders,
rawBody: Buffer = RAW_BODY,
toleranceSeconds?: number
) => verifySignature({ provider, rawBody, headers, secret: SECRET, toleranceSeconds });
const nowSeconds = (): string => Math.floor(Date.now() / 1000).toString();
const slackHeaders = (rawBody: Buffer, timestamp: string): IncomingHttpHeaders => ({
'x-slack-request-timestamp': timestamp,
'x-slack-signature': `v0=${hmacHex(`v0:${timestamp}:${rawBody.toString('utf8')}`)}`,
});
describe('verifySignature: linear-sha256', () => {
it('should accept a correct bare hex digest of the raw body', () => {
expect(verify('linear', { 'linear-signature': hmacHex(RAW_BODY) })).toEqual({ ok: true });
});
it('should read the first value when the header arrives as an array', () => {
expect(verify('linear', { 'linear-signature': [hmacHex(RAW_BODY)] })).toEqual({ ok: true });
});
it('should reject a signature computed over a different body', () => {
expect(verify('linear', { 'linear-signature': hmacHex(OTHER_BODY) })).toEqual({
ok: false,
reason: 'mismatch',
});
});
it('should reject a signature computed with a different secret', () => {
expect(verify('linear', { 'linear-signature': hmacHex(RAW_BODY, 'wrong') })).toEqual({
ok: false,
reason: 'mismatch',
});
});
it('should report a missing header', () => {
expect(verify('linear', {})).toEqual({ ok: false, reason: 'missing' });
expect(verify('linear', { 'linear-signature': '' })).toEqual({
ok: false,
reason: 'missing',
});
});
it('should report a malformed header', () => {
expect(verify('linear', { 'linear-signature': `sha256=${hmacHex(RAW_BODY)}` })).toEqual({
ok: false,
reason: 'malformed',
});
expect(verify('linear', { 'linear-signature': 'deadbeef' })).toEqual({
ok: false,
reason: 'malformed',
});
expect(verify('linear', { 'linear-signature': hmacHex(RAW_BODY).toUpperCase() })).toEqual({
ok: false,
reason: 'malformed',
});
});
it('should be case sensitive about the header name only', () => {
expect(verify('linear', { 'LINEAR-SIGNATURE': hmacHex(RAW_BODY) })).toEqual({
ok: false,
reason: 'missing',
});
});
});
describe('verifySignature: slack-v0', () => {
afterEach(() => {
vi.useRealTimers();
});
it('should accept a correct v0 signature over the timestamped base string', () => {
const timestamp = nowSeconds();
expect(verify('slack', slackHeaders(RAW_BODY, timestamp))).toEqual({ ok: true });
});
it('should accept a signature generated against a frozen clock', () => {
vi.useFakeTimers();
vi.setSystemTime(new Date('2026-08-01T12:00:00.000Z'));
const headers = slackHeaders(RAW_BODY, nowSeconds());
expect(verify('slack', headers)).toEqual({ ok: true });
});
it('should reject a signature computed over a different body', () => {
const timestamp = nowSeconds();
expect(verify('slack', slackHeaders(OTHER_BODY, timestamp))).toEqual({
ok: false,
reason: 'mismatch',
});
});
it('should reject a signature bound to a different timestamp', () => {
const timestamp = nowSeconds();
const headers = {
...slackHeaders(RAW_BODY, timestamp),
'x-slack-request-timestamp': (Number(timestamp) - 1).toString(),
};
expect(verify('slack', headers)).toEqual({ ok: false, reason: 'mismatch' });
});
it('should report a missing signature header', () => {
expect(verify('slack', { 'x-slack-request-timestamp': nowSeconds() })).toEqual({
ok: false,
reason: 'missing',
});
});
it('should report a missing timestamp header', () => {
const { 'x-slack-signature': signature } = slackHeaders(RAW_BODY, nowSeconds());
expect(verify('slack', { 'x-slack-signature': signature })).toEqual({
ok: false,
reason: 'missing',
});
});
it('should report a malformed signature header', () => {
const timestamp = nowSeconds();
expect(
verify('slack', { ...slackHeaders(RAW_BODY, timestamp), 'x-slack-signature': 'v1=abc' })
).toEqual({ ok: false, reason: 'malformed' });
expect(
verify('slack', {
...slackHeaders(RAW_BODY, timestamp),
'x-slack-signature': hmacHex(RAW_BODY),
})
).toEqual({ ok: false, reason: 'malformed' });
});
it('should reject a timestamp older than the tolerance window', () => {
const stale = (Math.floor(Date.now() / 1000) - 301).toString();
expect(verify('slack', slackHeaders(RAW_BODY, stale))).toEqual({
ok: false,
reason: 'stale',
});
});
it('should reject a timestamp too far in the future', () => {
const future = (Math.floor(Date.now() / 1000) + 301).toString();
expect(verify('slack', slackHeaders(RAW_BODY, future))).toEqual({
ok: false,
reason: 'stale',
});
});
it('should reject a request that goes stale while the clock advances', () => {
vi.useFakeTimers();
vi.setSystemTime(new Date('2026-08-01T12:00:00.000Z'));
const headers = slackHeaders(RAW_BODY, nowSeconds());
expect(verify('slack', headers)).toEqual({ ok: true });
vi.advanceTimersByTime(301_000);
expect(verify('slack', headers)).toEqual({ ok: false, reason: 'stale' });
});
it('should honor an explicit tolerance', () => {
const old = (Math.floor(Date.now() / 1000) - 600).toString();
expect(verify('slack', slackHeaders(RAW_BODY, old), RAW_BODY, 900)).toEqual({ ok: true });
expect(verify('slack', slackHeaders(RAW_BODY, old), RAW_BODY, 60)).toEqual({
ok: false,
reason: 'stale',
});
});
it('should reject a non-numeric timestamp as stale', () => {
expect(
verify('slack', {
'x-slack-request-timestamp': 'yesterday',
'x-slack-signature': `v0=${hmacHex('v0:yesterday:x')}`,
})
).toEqual({ ok: false, reason: 'stale' });
});
});
describe('verifySignature: github-sha256', () => {
it('should accept a correct sha256-prefixed signature', () => {
expect(verify('github', { 'x-hub-signature-256': `sha256=${hmacHex(RAW_BODY)}` })).toEqual({
ok: true,
});
});
it('should reject a signature computed over a different body', () => {
expect(verify('github', { 'x-hub-signature-256': `sha256=${hmacHex(OTHER_BODY)}` })).toEqual({
ok: false,
reason: 'mismatch',
});
});
it('should report a missing header', () => {
expect(verify('github', {})).toEqual({ ok: false, reason: 'missing' });
});
it('should report a malformed header', () => {
expect(verify('github', { 'x-hub-signature-256': hmacHex(RAW_BODY) })).toEqual({
ok: false,
reason: 'malformed',
});
expect(verify('github', { 'x-hub-signature-256': `sha1=${hmacHex(RAW_BODY)}` })).toEqual({
ok: false,
reason: 'malformed',
});
});
it('should reject a truncated but correctly prefixed signature', () => {
expect(
verify('github', { 'x-hub-signature-256': `sha256=${hmacHex(RAW_BODY).slice(0, 32)}` })
).toEqual({ ok: false, reason: 'mismatch' });
});
});
describe('verifySignature: none', () => {
it('should accept rest and jira without any header', () => {
expect(verify('rest', {})).toEqual({ ok: true });
expect(verify('jira', {})).toEqual({ ok: true });
});
it('should accept the none scheme even when a garbage header is present', () => {
expect(verify('rest', { 'linear-signature': 'nonsense' })).toEqual({ ok: true });
});
it('should cover every configured provider with a known scheme', () => {
const schemes = Object.values(providers).map((config) => config.signatureScheme);
expect(new Set(schemes)).toEqual(
new Set(['none', 'linear-sha256', 'slack-v0', 'github-sha256'])
);
});
});
describe('verifySignature: hostile input', () => {
const garbage: readonly string[] = [
'',
' ',
'v0=',
'sha256=',
':::',
'v0=zzzz',
'%%%%',
'0'.repeat(10_000),
'\u0000\u0000',
'null',
];
it('should never throw for any provider and any garbage header value', () => {
const slugs = Object.keys(providers) as ProviderSlug[];
for (const slug of slugs) {
for (const value of garbage) {
const headers: IncomingHttpHeaders = {
'linear-signature': value,
'x-hub-signature-256': value,
'x-slack-signature': value,
'x-slack-request-timestamp': value,
};
expect(() => verify(slug, headers)).not.toThrow();
expect(typeof verify(slug, headers).ok).toBe('boolean');
}
}
});
it('should never throw for an empty body or empty header set', () => {
const slugs = Object.keys(providers) as ProviderSlug[];
for (const slug of slugs) {
expect(() => verify(slug, {}, Buffer.alloc(0))).not.toThrow();
}
});
it('should tolerate array-valued and duplicated headers', () => {
expect(() =>
verify('slack', {
'x-slack-signature': ['v0=abc', 'v0=def'],
'x-slack-request-timestamp': [nowSeconds(), 'garbage'],
})
).not.toThrow();
});
});

View File

@@ -0,0 +1,82 @@
import { describe, it, expect } from 'vitest';
import { Readable, type Transform } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { batch, jsonArray } from '../lib/streams.js';
const collect = async <T>(source: Readable, transform: Transform): Promise<T[]> => {
const out: T[] = [];
await pipeline(source, transform, async (results: AsyncIterable<T>) => {
for await (const item of results) out.push(item);
});
return out;
};
describe('batch', () => {
it('should group items into fixed-size arrays', async () => {
const source = Readable.from([1, 2, 3, 4], { objectMode: true });
const result = await collect<number[]>(source, batch<number>(2));
expect(result).toEqual([[1, 2], [3, 4]]);
});
it('should flush a partial trailing batch', async () => {
const source = Readable.from([1, 2, 3, 4, 5], { objectMode: true });
const result = await collect<number[]>(source, batch<number>(2));
expect(result).toEqual([[1, 2], [3, 4], [5]]);
});
it('should emit nothing for an empty source', async () => {
const source = Readable.from([], { objectMode: true });
const result = await collect<number[]>(source, batch<number>(3));
expect(result).toEqual([]);
});
it('should reject a non-positive size', () => {
expect(() => batch(0)).toThrow(TypeError);
expect(() => batch(1.5)).toThrow(TypeError);
});
});
describe('jsonArray', () => {
const serialize = async (items: unknown[]): Promise<string> => {
const chunks = await collect<string>(
Readable.from(items, { objectMode: true }),
jsonArray()
);
return chunks.map(String).join('');
};
it('should serialize objects into a JSON array', async () => {
const items = [{ id: 'a' }, { id: 'b' }];
const output = await serialize(items);
expect(output).toBe('[{"id":"a"},{"id":"b"}]');
expect(JSON.parse(output)).toEqual(items);
});
it('should emit an empty array when the source yields nothing', async () => {
const output = await serialize([]);
expect(output).toBe('[]');
expect(JSON.parse(output)).toEqual([]);
});
it('should emit a valid single-element array', async () => {
const output = await serialize([{ id: 'only' }]);
expect(JSON.parse(output)).toEqual([{ id: 'only' }]);
});
it('should propagate serialization errors', async () => {
const circular: Record<string, unknown> = {};
circular.self = circular;
await expect(serialize([circular])).rejects.toThrow();
});
});

View File

@@ -0,0 +1,253 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import request from 'supertest';
import { createHmac } from 'node:crypto';
import app from '../app.js';
import { configureQueue, drain } from '../lib/queue.js';
vi.mock('../db/ingest_events.dao.js', () => ({
recordDelivery: vi.fn(),
markProcessing: vi.fn(),
markDone: vi.fn(),
markFailed: vi.fn(),
resetStaleProcessing: vi.fn(),
}));
vi.mock('../db/notes.dao.js', () => ({
createNotes: vi.fn(),
getAllNotes: vi.fn(),
streamAllNotes: vi.fn(),
}));
import {
recordDelivery,
markDone,
markFailed,
markProcessing,
} from '../db/ingest_events.dao.js';
import { createNotes } from '../db/notes.dao.js';
const mockRecordDelivery = vi.mocked(recordDelivery);
const mockCreateNotes = vi.mocked(createNotes);
const mockMarkDone = vi.mocked(markDone);
const mockMarkFailed = vi.mocked(markFailed);
const mockMarkProcessing = vi.mocked(markProcessing);
const SECRET = 'linear-test-secret';
// Deliberately irregular spacing: if any middleware parsed and re-serialized
// this body, the signature computed over these exact bytes would not verify.
const RAW_BODY = '{"action":"create", "type":"Comment","data":{"id":"cmt_1","body":"Deploys are scary","url":"https://linear.app/c/1","user":{"name":"jane"}} }';
const sign = (body: string, secret = SECRET): string =>
createHmac('sha256', secret).update(body).digest('hex');
const postLinear = (body: string, signature: string) =>
request(app)
.post('/v1/webhooks/linear')
.set('Content-Type', 'application/json')
.set('linear-signature', signature)
.send(body);
describe('POST /v1/webhooks/:provider', () => {
const originalSecret = process.env.LINEAR_SIGNING_SECRET;
beforeEach(() => {
vi.clearAllMocks();
process.env.LINEAR_SIGNING_SECRET = SECRET;
configureQueue({ baseDelayMs: 0, maxAttempts: 2 });
mockRecordDelivery.mockResolvedValue(42);
mockCreateNotes.mockResolvedValue([]);
});
afterEach(async () => {
await drain();
if (originalSecret === undefined) {
delete process.env.LINEAR_SIGNING_SECRET;
} else {
process.env.LINEAR_SIGNING_SECRET = originalSecret;
}
});
it('should accept a correctly signed delivery', async () => {
const res = await postLinear(RAW_BODY, sign(RAW_BODY));
expect(res.status).toBe(200);
expect(res.body).toEqual({ accepted: true, notes: 1 });
});
it('should preserve the exact request bytes for signature verification', async () => {
// The signature is over RAW_BODY verbatim. This passing is the regression
// test for express.raw being mounted ahead of express.json in app.ts.
const res = await postLinear(RAW_BODY, sign(RAW_BODY));
expect(res.status).toBe(200);
});
it('should insert the normalized note after responding', async () => {
await postLinear(RAW_BODY, sign(RAW_BODY));
await drain();
expect(mockCreateNotes).toHaveBeenCalledOnce();
const [notes] = mockCreateNotes.mock.calls[0];
expect(notes).toHaveLength(1);
expect(notes[0].text).toBe('Deploys are scary');
expect(notes[0].author).toBe('jane');
expect(notes[0].id).toBe('linear_cmt_1');
});
it('should record provenance on the ingested note', async () => {
await postLinear(RAW_BODY, sign(RAW_BODY));
await drain();
const [notes] = mockCreateNotes.mock.calls[0];
expect(notes[0].sourceMeta).toMatchObject({
provider: 'linear',
externalId: 'cmt_1',
permalink: 'https://linear.app/c/1',
});
});
it('should mark the delivery done once the insert succeeds', async () => {
await postLinear(RAW_BODY, sign(RAW_BODY));
await drain();
expect(mockMarkProcessing).toHaveBeenCalledWith(42);
expect(mockMarkDone).toHaveBeenCalledWith(42);
expect(mockMarkFailed).not.toHaveBeenCalled();
});
it('should mark the delivery failed when the insert keeps failing', async () => {
mockCreateNotes.mockRejectedValue(new Error('connection refused'));
await postLinear(RAW_BODY, sign(RAW_BODY));
await drain();
expect(mockMarkFailed).toHaveBeenCalledWith(42, 'connection refused');
expect(mockMarkDone).not.toHaveBeenCalled();
});
it('should respond without waiting for the insert to finish', async () => {
let finishInsert: () => void = () => {};
mockCreateNotes.mockImplementation(
() => new Promise((resolve) => {
finishInsert = () => resolve([]);
})
);
const res = await postLinear(RAW_BODY, sign(RAW_BODY));
expect(res.status).toBe(200);
expect(mockMarkDone).not.toHaveBeenCalled();
finishInsert();
await drain();
expect(mockMarkDone).toHaveBeenCalledWith(42);
});
it('should drop a redelivery without enqueueing work', async () => {
mockRecordDelivery.mockResolvedValue(null);
const res = await postLinear(RAW_BODY, sign(RAW_BODY));
await drain();
expect(res.status).toBe(200);
expect(res.body).toEqual({ duplicate: true });
expect(mockCreateNotes).not.toHaveBeenCalled();
});
it('should reject a signature computed with the wrong secret', async () => {
const res = await postLinear(RAW_BODY, sign(RAW_BODY, 'wrong-secret'));
expect(res.status).toBe(401);
expect(mockRecordDelivery).not.toHaveBeenCalled();
});
it('should reject a delivery whose body was altered after signing', async () => {
const signature = sign(RAW_BODY);
const tampered = RAW_BODY.replace('Deploys are scary', 'Deploys are fine');
const res = await postLinear(tampered, signature);
expect(res.status).toBe(401);
});
it('should reject a delivery with no signature header', async () => {
const res = await request(app)
.post('/v1/webhooks/linear')
.set('Content-Type', 'application/json')
.send(RAW_BODY);
expect(res.status).toBe(401);
});
it('should return 500 when no signing secret is configured', async () => {
delete process.env.LINEAR_SIGNING_SECRET;
const res = await postLinear(RAW_BODY, sign(RAW_BODY));
expect(res.status).toBe(500);
expect(mockRecordDelivery).not.toHaveBeenCalled();
});
it('should return 500 when recording the delivery fails', async () => {
mockRecordDelivery.mockRejectedValue(new Error('connection refused'));
const res = await postLinear(RAW_BODY, sign(RAW_BODY));
expect(res.status).toBe(500);
expect(mockCreateNotes).not.toHaveBeenCalled();
});
it('should return 404 for an unregistered provider', async () => {
const res = await request(app)
.post('/v1/webhooks/notion')
.set('Content-Type', 'application/json')
.send('{}');
expect(res.status).toBe(404);
});
it('should return 501 for a provider with no ingestion mapping yet', async () => {
process.env.GITHUB_WEBHOOK_SECRET = 'gh-secret';
const body = '{"action":"created"}';
const signature = `sha256=${createHmac('sha256', 'gh-secret').update(body).digest('hex')}`;
const res = await request(app)
.post('/v1/webhooks/github')
.set('Content-Type', 'application/json')
.set('x-hub-signature-256', signature)
.send(body);
delete process.env.GITHUB_WEBHOOK_SECRET;
expect(res.status).toBe(501);
});
it('should return 400 for a body that is not valid JSON', async () => {
const body = 'not json at all';
const res = await postLinear(body, sign(body));
expect(res.status).toBe(400);
});
it('should return 400 when the payload is missing the fields the provider guarantees', async () => {
const body = '{"action":"create","data":{}}';
const res = await postLinear(body, sign(body));
expect(res.status).toBe(400);
expect(mockRecordDelivery).not.toHaveBeenCalled();
});
it('should answer a Slack url_verification handshake without a signature', async () => {
const res = await request(app)
.post('/v1/webhooks/slack')
.set('Content-Type', 'application/json')
.send('{"type":"url_verification","challenge":"abc123"}');
expect(res.status).toBe(200);
expect(res.body).toEqual({ challenge: 'abc123' });
});
});

View File

@@ -0,0 +1,7 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": false
},
"exclude": ["node_modules", "dist", "tests"]
}

26
backend/tsconfig.json Normal file
View File

@@ -0,0 +1,26 @@
{
"compilerOptions": {
"target": "ES2022",
"lib": ["ES2022"],
"module": "NodeNext",
"moduleResolution": "NodeNext",
"types": ["node"],
"rootDir": ".",
"outDir": "dist",
"esModuleInterop": true,
"resolveJsonModule": true,
"skipLibCheck": true,
"sourceMap": true,
"noEmit": true,
/* Linting */
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noImplicitOverride": true,
"noFallthroughCasesInSwitch": true,
"noUncheckedSideEffectImports": true
},
"include": ["**/*.ts"],
"exclude": ["node_modules", "dist"]
}

55
backend/types/domain.ts Normal file
View File

@@ -0,0 +1,55 @@
export type Note = {
id: string;
text: string;
x: number;
y: number;
author: string;
color: string;
};
/** Shape accepted when writing a note; positional and color fields fall back to defaults. */
export type NoteInput = {
id: string;
text: string;
author: string;
x?: number;
y?: number;
color?: string;
sourceMeta?: Record<string, unknown>;
};
export type Cluster = {
label: string;
noteIds: string[];
};
export type ClusterResponse = {
clusters: Cluster[];
score: number;
};
export type ValidationResult = {
valid: boolean;
reasons: string[];
};
/** noteId to embedding vector. */
export type EmbeddingMap = Map<string, number[]>;
const isStringArray = (value: unknown): value is string[] =>
Array.isArray(value) && value.every((entry) => typeof entry === 'string');
/**
* Runtime guard for LLM output, which arrives as parsed JSON of unknown shape.
* Structural correctness beyond this (complete coverage, no duplicates) is the
* job of validateStructure.
*/
export const isClusterArray = (value: unknown): value is Cluster[] => {
if (!Array.isArray(value)) return false;
return value.every((entry) => {
if (typeof entry !== 'object' || entry === null) return false;
const candidate = entry as Record<string, unknown>;
return typeof candidate.label === 'string' && isStringArray(candidate.noteIds);
});
};

View File

@@ -0,0 +1,43 @@
import type { NoteInput } from './domain.js';
export type ProviderSlug = 'slack' | 'jira' | 'linear' | 'github' | 'rest';
export type SignatureScheme = 'slack-v0' | 'github-sha256' | 'linear-sha256' | 'none';
export type DeliveryStatus = 'pending' | 'processing' | 'done' | 'failed';
/**
* Integration as exposed to application code. Ciphertext columns are
* deliberately absent so a value of this type can never leak a secret.
*/
export type IntegrationRow = {
id: number;
provider: ProviderSlug;
externalWorkspaceId: string | null;
displayName: string | null;
scopes: string[];
tokenExpiresAt: Date | null;
};
export type SignatureResult =
| { ok: true }
| { ok: false; reason: 'missing' | 'malformed' | 'mismatch' | 'stale' };
export type NormalizedDelivery = {
externalId: string;
notes: NoteInput[];
};
/** Provenance recorded on each ingested note's source_meta column. */
export type NoteProvenance = {
provider: ProviderSlug;
externalId: string;
permalink?: string;
authorHandle?: string;
receivedAt: string;
};
const SLUGS: readonly string[] = ['slack', 'jira', 'linear', 'github', 'rest'];
export const isProviderSlug = (value: string): value is ProviderSlug =>
SLUGS.includes(value);

View File

@@ -0,0 +1,2 @@
VITE_APP_URL=localhost:3000
VITE_API_BASE=http://localhost:3001

2
frontend/.env.production Normal file
View File

@@ -0,0 +1,2 @@
VITE_APP_URL=https://example.com
VITE_API_BASE=https://www.example.com:4000

View File

@@ -1,5 +1,5 @@
{
"name": "congruity-frontend",
"name": "kongruity-frontend",
"private": true,
"version": "0.1.0",
"type": "module",
@@ -37,4 +37,3 @@
"vitest": "^4.0.18"
}
}

View File

@@ -1,7 +1,7 @@
import { useQuery, useMutation } from '@tanstack/react-query';
import type { Sticky, Cluster } from '../types/types';
import type { Sticky, ClusterResponse } from '../types/types';
const API_BASE = 'http://localhost:3001/v1/notes';
const API_BASE = `${import.meta.env.VITE_API_BASE}/v1/notes`;
const fetchStickies = async (): Promise<Sticky[]> => {
const response = await fetch(API_BASE);
@@ -11,7 +11,7 @@ const fetchStickies = async (): Promise<Sticky[]> => {
return response.json();
};
const fetchClusters = async (): Promise<Cluster[]> => {
const fetchClusters = async (): Promise<ClusterResponse> => {
const response = await fetch(`${API_BASE}/cluster`, {
method: 'POST',
});
@@ -29,7 +29,7 @@ export const useGetStickies = () => {
};
export const useClusterStickies = () => {
return useMutation<Cluster[]>({
return useMutation<ClusterResponse>({
mutationFn: fetchClusters,
});
};

View File

@@ -1,16 +1,43 @@
import { useState, useEffect, useRef } from 'react';
import type { DragEvent } from 'react';
import { useGetStickies, useClusterStickies } from '../api/api';
import type { Sticky as StickyType } from '../types/types';
import type { Sticky as StickyType, RankedCluster } from '../types/types';
import Sticky from './sticky';
import Button from './button';
import '../styles/stickies.css';
// In practice, silhouette on cosine distance between text embeddings occupies roughly
// [-0.05, 0.10], not strict theoretical [-1, 1]: near-orthogonal vectors put both the within- and
// nearest-cluster distances close to 0.8, and the coefficient divides their gap
// by the larger. These bands are calibrated to that range for voyage-3. see README, Reading the cohesion score
const scoreLabel = (score: number): string => {
if (score >= 0.07) return 'Strong';
if (score >= 0.04) return 'Moderate';
if (score >= 0.01) return 'Weak';
return 'Poor';
};
const Stickies = () => {
const { data: stickies, isLoading, error } = useGetStickies();
const { mutate: cluster, data: clusters, isPending } = useClusterStickies();
const { mutate: cluster, data: clusterResponse, isPending } = useClusterStickies();
const [rankedClusters, setRankedClusters] = useState<RankedCluster[]>([]);
const dragIndex = useRef<number | null>(null);
const [dragOverIndex, setDragOverIndex] = useState<number | null>(null);
useEffect(() => {
if (clusterResponse?.clusters) {
setRankedClusters(
clusterResponse.clusters.map((c, i) => ({ ...c, rank: i + 1 }))
);
}
}, [clusterResponse]);
if (isLoading) return <div>Loading...</div>;
if (error) return <div>Error: {error.message}</div>;
const score = clusterResponse?.score;
const handleCluster = () => {
cluster();
};
@@ -24,20 +51,78 @@ const Stickies = () => {
const stickyMap = buildStickyMap();
const renderStickies = (items: StickyType[]) =>
items.map((sticky) => <Sticky key={sticky.id} sticky={sticky} />);
items?.map((sticky) => <Sticky key={sticky.id} sticky={sticky} />);
const handleDragStart = (index: number) => {
dragIndex.current = index;
};
const handleDragOver = (e: DragEvent, index: number) => {
e.preventDefault();
setDragOverIndex(index);
};
const handleDragLeave = () => {
setDragOverIndex(null);
};
const handleDrop = (targetIndex: number) => {
const sourceIndex = dragIndex.current;
if (sourceIndex === null || sourceIndex === targetIndex) {
dragIndex.current = null;
setDragOverIndex(null);
return;
}
const reordered = [...rankedClusters];
const [moved] = reordered.splice(sourceIndex, 1);
reordered.splice(targetIndex, 0, moved);
setRankedClusters(reordered.map((c, i) => ({ ...c, rank: i + 1 })));
dragIndex.current = null;
setDragOverIndex(null);
};
const handleDragEnd = () => {
dragIndex.current = null;
setDragOverIndex(null);
};
console.log(score?.toFixed(2))
return (
<div className="stickies-container">
<Button onClick={handleCluster} isLoading={isPending} label="Group Stickies By Topic" />
{clusters ? (
{rankedClusters.length > 0 ? (
<div className="clusters-container">
{clusters.map((group) => (
<div key={group.label} className="cluster-group">
{score != null && (
<div className="cohesion-score">
Cluster cohesion: <strong>{scoreLabel(score)}</strong>
</div>
)}
{rankedClusters.map((group, index) => (
<div
key={group.label}
className={`cluster-group cluster-draggable${dragOverIndex === index ? ' cluster-drag-over' : ''}`}
draggable
onDragStart={() => handleDragStart(index)}
onDragOver={(e) => handleDragOver(e, index)}
onDragLeave={handleDragLeave}
onDrop={() => handleDrop(index)}
onDragEnd={handleDragEnd}
>
<div className="cluster-header">
<span className="cluster-rank" aria-label={`Priority ${group.rank}`}>
{group.rank}
</span>
{group.rank === 1 && (
<span className="cluster-reorder-hint">Drag and drop to reorganize cluster priority</span>
)}
<h3 className="cluster-label">{group.label}</h3>
<span className="cluster-drag-handle" aria-hidden="true">⠿</span>
</div>
<div className="stickies-grid">
{renderStickies(
group.noteIds
.map((id) => stickyMap.get(id))
group?.noteIds
.map((id) => stickyMap?.get(id))
.filter((s): s is StickyType => !!s)
)}
</div>

View File

@@ -14,5 +14,5 @@ createRoot(document.getElementById('root')!).render(
<App />
</BrowserRouter>
</QueryClientProvider>
</StrictMode>,
</StrictMode>
)

View File

@@ -4,14 +4,10 @@ import '../styles/home.css'
const Home = () => {
return (
<div>
<div>
<Navbar />
</div>
<div>
<Stickies />
</div>
</div>
);
};

View File

@@ -1,7 +1,7 @@
.main-head-box {
border-radius: 8px;
border: 1px solid #6dd6f4;
background-color: rgb(92, 0 91);
background-color: rgb(92, 0, 91);
display: flex;
}
@@ -37,5 +37,37 @@
'FILL' 0,
'wght' 300,
'GRAD' 0,
'opsz' 24
'opsz' 24;
}
@media screen and (max-width: 478px) {
.main-head-subbox-right {
display: none;
}
.main-head-subbox-left {
display: flex;
justify-content: center;
margin-left: auto;
margin-right: auto;
width: 100%;
}
.main-head {
font-size: 4rem;
color: #6dd6f4;
font-family: "Sulphur Point", sans-serif;
display: flex;
justify-content: center;
align-items: center;
margin-left: auto;
margin-right: auto;
font-weight: 400;
letter-spacing: 2px;
text-decoration: underline;
}
.material-symbols-outlined {
display: none !important;
}
}

View File

@@ -3,6 +3,7 @@
flex-wrap: wrap;
gap: 16px;
justify-content: center;
margin-top: 18px;
padding: 24px;
}
@@ -23,8 +24,84 @@
padding: 16px;
}
.cluster-draggable {
cursor: grab;
transition: box-shadow 0.2s ease, border-color 0.2s ease, transform 0.15s ease;
}
.cluster-draggable:active {
cursor: grabbing;
}
.cluster-drag-over {
border-color: #ffb7ce;
box-shadow: 0 0 12px rgba(255, 183, 206, 0.4);
transform: scale(1.01);
}
.cluster-header {
position: relative;
display: flex;
align-items: center;
gap: 12px;
margin-bottom: 16px;
min-height: 32px;
}
.cluster-rank {
display: flex;
align-items: center;
justify-content: center;
width: 32px;
height: 32px;
border-radius: 50%;
background: #6dd6f4;
color: #1a1a2e;
font-weight: 700;
font-size: 0.95em;
flex-shrink: 0;
}
.cluster-reorder-hint {
font-size: 0.8em;
color: #9ca3af;
font-style: italic;
white-space: nowrap;
flex-shrink: 0;
}
.cluster-label {
margin: 0 0 16px 0;
position: absolute;
left: 50%;
transform: translateX(-50%);
max-width: 50%;
margin: 0;
font-size: 1.2em;
font-weight: 600;
pointer-events: none;
}
.cluster-drag-handle {
margin-left: auto;
font-size: 1.4em;
color: #6dd6f4;
opacity: 0.4;
user-select: none;
transition: opacity 0.2s ease;
flex-shrink: 0;
}
.cluster-draggable:hover .cluster-drag-handle {
opacity: 0.8;
}
.cohesion-score {
text-align: center;
font-size: 0.95em;
color: #e0e0e0;
padding: 8px 16px;
background: rgba(109, 214, 244, 0.1);
border-radius: 6px;
width: fit-content;
margin: 0 auto;
}

View File

@@ -9,10 +9,13 @@ const MOCK_STICKIES = [
{ id: 'note_002', text: 'Export takes too long', x: 798, y: 211, author: 'user_2', color: 'green' },
];
const MOCK_CLUSTERS = [
const MOCK_CLUSTER_RESPONSE = {
clusters: [
{ label: 'Auth Issues', noteIds: ['note_001'] },
{ label: 'Export Issues', noteIds: ['note_002'] },
];
],
score: 0.09,
};
let fetchMock: ReturnType<typeof vi.fn>;
@@ -70,10 +73,9 @@ describe('Stickies', () => {
});
it('should show cluster labels after clustering succeeds', async () => {
// First call = GET notes, second call = POST cluster
fetchMock
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTERS));
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
const { Wrapper } = createTestWrapper();
render(<Stickies />, { wrapper: Wrapper });
@@ -90,7 +92,7 @@ describe('Stickies', () => {
it('should still render sticky note text inside clusters', async () => {
fetchMock
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTERS));
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
const { Wrapper } = createTestWrapper();
render(<Stickies />, { wrapper: Wrapper });
@@ -101,4 +103,41 @@ describe('Stickies', () => {
expect(await screen.findByText('Login flow feels confusing')).toBeInTheDocument();
expect(screen.getByText('Export takes too long')).toBeInTheDocument();
});
it('should display rank badges on clustered groups', async () => {
fetchMock
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
const { Wrapper } = createTestWrapper();
render(<Stickies />, { wrapper: Wrapper });
const btn = await screen.findByRole('button', { name: 'Group Stickies By Topic' });
await userEvent.click(btn);
await waitFor(() => {
expect(screen.getByLabelText('Priority 1')).toBeInTheDocument();
expect(screen.getByLabelText('Priority 2')).toBeInTheDocument();
});
});
it('should make cluster groups draggable', async () => {
fetchMock
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
const { Wrapper } = createTestWrapper();
render(<Stickies />, { wrapper: Wrapper });
const btn = await screen.findByRole('button', { name: 'Group Stickies By Topic' });
await userEvent.click(btn);
await waitFor(() => {
const groups = document.querySelectorAll('.cluster-draggable');
groups.forEach((group) => {
expect(group).toHaveAttribute('draggable', 'true');
});
expect(groups.length).toBe(2);
});
});
});

View File

@@ -11,3 +11,12 @@ export type Cluster = {
label: string;
noteIds: string[];
};
export type ClusterResponse = {
clusters: Cluster[];
score: number;
};
export type RankedCluster = Cluster & {
rank: number;
};

21
package-lock.json generated Normal file
View File

@@ -0,0 +1,21 @@
{
"name": "kongruity",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"dependencies": {
"split2": "^4.2.0"
}
},
"node_modules/split2": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
"integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
"license": "ISC",
"engines": {
"node": ">= 10.x"
}
}
}
}

5
package.json Normal file
View File

@@ -0,0 +1,5 @@
{
"dependencies": {
"split2": "^4.2.0"
}
}