202 lines
8.9 KiB
Markdown
202 lines
8.9 KiB
Markdown
# Low-level design
|
|
|
|
Phase 3 output. Exact interfaces for every module in [ROADMAP.md](ROADMAP.md), written before implementation. Error handling follows [agents.md](agents.md) section 2: async route handlers wrap in `try/catch`; services throw typed errors and let routes translate them into status codes.
|
|
|
|
## Shared types — `types/integration.ts`
|
|
|
|
```ts
|
|
export type ProviderSlug = 'slack' | 'jira' | 'linear' | 'github' | 'rest';
|
|
|
|
export type IntegrationRow = {
|
|
id: number;
|
|
provider: ProviderSlug;
|
|
externalWorkspaceId: string | null;
|
|
displayName: string | null;
|
|
scopes: string[];
|
|
tokenExpiresAt: Date | null;
|
|
};
|
|
|
|
export type SignatureResult =
|
|
| { ok: true }
|
|
| { ok: false; reason: 'missing' | 'malformed' | 'mismatch' | 'stale' };
|
|
|
|
export type NormalizedDelivery = {
|
|
externalId: string;
|
|
notes: NoteInput[];
|
|
};
|
|
|
|
export type DeliveryStatus = 'pending' | 'processing' | 'done' | 'failed';
|
|
```
|
|
|
|
String-literal unions rather than enums, per agents.md section 4.
|
|
|
|
## `lib/crypto.ts`
|
|
|
|
```ts
|
|
export const encryptSecret = (plaintext: string): string;
|
|
export const decryptSecret = (ciphertext: string): string;
|
|
export const hashApiKey = (key: string): string;
|
|
export const safeEquals = (a: string, b: string): boolean;
|
|
```
|
|
|
|
AES-256-GCM, serialized `base64(iv):base64(tag):base64(payload)`. `encryptSecret` throws if `TOKEN_ENCRYPTION_KEY` is missing or not 32 bytes after base64 decode. `safeEquals` pads to equal length before `timingSafeEqual` so it cannot leak length through an early throw.
|
|
|
|
Tests: round-trip fidelity, distinct ciphertext for identical plaintext (random IV), tamper detection on the auth tag, rejection of a short key, `safeEquals` true and false paths.
|
|
|
|
## `middleware/apiKey.ts`
|
|
|
|
```ts
|
|
declare module 'express-serve-static-core' {
|
|
interface Request { integration?: IntegrationRow }
|
|
}
|
|
|
|
export const requireApiKey: RequestHandler;
|
|
```
|
|
|
|
Reads `Authorization`, expects `Bearer <key>`. On success attaches `req.integration` and calls `next()`. On any failure responds `401 { error: 'Unauthorized' }` — identical body for missing, malformed, and unknown keys.
|
|
|
|
Tests: valid key passes and attaches, missing header 401s, wrong scheme 401s, unknown key 401s, database error surfaces as 500 via `next(err)`.
|
|
|
|
## `routes/ingest.routes.ts`
|
|
|
|
```ts
|
|
type IngestBody = { notes: NoteInput[] };
|
|
const isIngestBody = (value: unknown): value is IngestBody;
|
|
```
|
|
|
|
`POST /` returns `201 { inserted: number; notes: Note[] }`. Rejects an empty array and a batch over 5000 with `400`. The predicate checks `id`, `text`, and `author` are non-empty strings and that optional `x`, `y`, `color` have the right primitive types when present.
|
|
|
|
Tests: happy path inserts and echoes, missing `notes` key 400s, non-array 400s, element missing `text` 400s, empty array 400s, oversized batch 400s, unauthenticated 401s, DAO rejection 500s.
|
|
|
|
## `db/ingest_events.dao.ts`
|
|
|
|
```ts
|
|
export const recordDelivery = (input: {
|
|
provider: ProviderSlug;
|
|
externalId: string;
|
|
integrationId?: number;
|
|
}): Promise<number | null>;
|
|
|
|
export const markProcessing = (id: number): Promise<void>;
|
|
export const markDone = (id: number): Promise<void>;
|
|
export const markFailed = (id: number, error: string): Promise<void>;
|
|
export const resetStaleProcessing = (olderThanMs: number): Promise<number>;
|
|
```
|
|
|
|
`recordDelivery` resolves to `null` when the unique constraint fires, which is the caller's signal that this is a redelivery. `resetStaleProcessing` exists so a restart can recover rows the in-process queue was holding.
|
|
|
|
Tests: first delivery returns an id, identical redelivery returns null, different providers with the same external id both insert, status transitions persist, stale reset only touches `processing` rows past the cutoff.
|
|
|
|
## `lib/signatures.ts`
|
|
|
|
```ts
|
|
export const verifySignature = (input: {
|
|
provider: ProviderSlug;
|
|
rawBody: Buffer;
|
|
headers: IncomingHttpHeaders;
|
|
secret: string;
|
|
toleranceSeconds?: number;
|
|
}): SignatureResult;
|
|
```
|
|
|
|
Dispatches on the registry's `signatureScheme`. Never throws on bad input — a malformed header is a `{ ok: false }` result, because a thrown exception here would be an unhandled rejection path reachable by any anonymous caller.
|
|
|
|
Tests, per scheme: correct signature passes, altered body fails with `mismatch`, absent header fails with `missing`, timestamp outside tolerance fails with `stale`, and a signature of the right length but wrong content fails without a timing difference.
|
|
|
|
## `config/providers.ts`
|
|
|
|
```ts
|
|
export type ProviderConfig = {
|
|
slug: ProviderSlug;
|
|
signatureScheme: 'slack-v0' | 'github-sha256' | 'linear-sha256' | 'none';
|
|
signatureHeader: string;
|
|
timestampHeader?: string;
|
|
challenge?: (body: unknown) => { status: number; body: unknown } | null;
|
|
normalize: (payload: unknown) => NormalizedDelivery;
|
|
oauth?: {
|
|
authorizeUrl: string;
|
|
tokenUrl: string;
|
|
scopes: string[];
|
|
};
|
|
};
|
|
|
|
export const providers: Record<ProviderSlug, ProviderConfig>;
|
|
export const isProviderSlug = (value: string): value is ProviderSlug;
|
|
```
|
|
|
|
The `rest` entry uses scheme `none` and no normalizer of consequence — it exists so the registry is the complete inventory of inbound sources.
|
|
|
|
Tests: every slug resolves, `isProviderSlug` rejects unknown input, each entry declares a header when its scheme is not `none`.
|
|
|
|
## `services/normalize.service.ts`
|
|
|
|
```ts
|
|
export const normalize = (
|
|
provider: ProviderSlug,
|
|
payload: unknown
|
|
): NormalizedDelivery;
|
|
|
|
export const hasNormalizer = (provider: ProviderSlug): boolean;
|
|
```
|
|
|
|
Delegates to the registry entry. Throws `NormalizationError` when the payload lacks the fields that provider guarantees, and also when no normalizer is registered at all. Provenance written to `source_meta` as `{ provider, externalId, permalink, authorHandle, receivedAt }`.
|
|
|
|
Two normalizers ship with the infrastructure. `rest` is the generic bulk push and is a validation step rather than a translation. `linear` maps a comment webhook and exists so the webhook path is provable end to end; it is deliberately minimal and is not the Linear integration, which additionally needs OAuth install and label filtering. `slack`, `github`, and `jira` declare transport only.
|
|
|
|
Note ids are composed as `<provider>_<externalId>` and truncated to 64 characters, because `notes.id` is `VARCHAR(64)`.
|
|
|
|
Tests: a representative payload per shipped provider yields the expected notes, an empty message body yields zero notes rather than a blank note, provenance fields are populated, a long external id still fits the column, and a provider without a normalizer throws.
|
|
|
|
## `lib/queue.ts`
|
|
|
|
```ts
|
|
export type Job = () => Promise<void>;
|
|
|
|
export const enqueue = (name: string, job: Job): void;
|
|
export const size = (): number;
|
|
export const drain = (): Promise<void>;
|
|
export const configureQueue = (opts: {
|
|
maxAttempts?: number;
|
|
baseDelayMs?: number;
|
|
}): void;
|
|
```
|
|
|
|
Concurrency 1, exponential backoff with jitter, terminal failure logged and surfaced through the caller's `markFailed`. `drain` resolves when the queue is empty and no job is in flight, which is what makes the integration tests deterministic instead of timer-dependent.
|
|
|
|
Tests: jobs run in order, a failing job retries to the cap then stops, backoff delays grow, `drain` waits for in-flight work, `size` reflects pending count.
|
|
|
|
## `lib/httpClient.ts`
|
|
|
|
```ts
|
|
export const requestJson = <T>(url: string, init?: RequestInit & {
|
|
timeoutMs?: number;
|
|
maxAttempts?: number;
|
|
}): Promise<T>;
|
|
```
|
|
|
|
Retries 429 and 5xx, honors `Retry-After` when present, jittered exponential backoff otherwise, `AbortSignal.timeout` for the deadline. A 4xx other than 429 fails immediately — retrying a 401 just burns rate limit.
|
|
|
|
Tests: 200 parses, 429 with `Retry-After` waits then succeeds, 500 retries to cap, 400 fails without retry, timeout aborts.
|
|
|
|
## `routes/webhooks.routes.ts`
|
|
|
|
```ts
|
|
router.post('/:provider', rawBody, handler);
|
|
```
|
|
|
|
Handler order is fixed and each step has a distinct exit:
|
|
|
|
1. Unknown slug — `404`.
|
|
2. Body is not JSON — `400`.
|
|
3. Challenge request — provider-defined status and body.
|
|
4. No signing secret configured server-side — `500`, since that is our misconfiguration and not the caller's fault.
|
|
5. Signature invalid — `401`.
|
|
6. Provider registered but no normalizer yet — `501`. The transport is infrastructure and works; the payload mapping arrives with that provider's integration.
|
|
7. Payload fails normalization — `400`.
|
|
8. `recordDelivery` returns null — `200 { duplicate: true }`, no work enqueued.
|
|
9. Otherwise enqueue the insert and return `200 { accepted: true, notes: n }`.
|
|
|
|
Normalization runs *before* the ack rather than inside the queue job. It is a pure function, so it costs nothing on the request path, and doing it here means a malformed payload gets a `400` the sender can act on instead of failing silently in a background job. Only the database write is deferred.
|
|
|
|
Tests: each numbered branch, plus the ordering regression — sign a body with irregular internal whitespace and assert verification still succeeds, which can only happen if the exact bytes survived `express.raw` ahead of `express.json`.
|