Compare commits
52 Commits
refact3
...
minor-css-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c6b07ccb56 | ||
|
|
8e79e78006 | ||
| 19d4a82c90 | |||
| edeadd547a | |||
| 3b707a150b | |||
| c2d40fa409 | |||
|
|
6f09b6ecdc | ||
| ac75a30b61 | |||
| f1f2a93e4a | |||
|
|
68fed1a56e | ||
| e11122ed14 | |||
|
|
7b47e46852 | ||
|
|
62477d009c | ||
|
|
ee6fa9e576 | ||
| 90035b3568 | |||
|
|
9636be78c9 | ||
|
|
a50ca5c171 | ||
|
|
e271702bca | ||
|
|
d3d76b1d0d | ||
|
|
8cae0665db | ||
|
|
21453aab7d | ||
|
|
691186e02c | ||
|
|
fd48aaab4d | ||
|
|
0b56d0d6c4 | ||
|
|
0d22998c2e | ||
|
|
ce7bc2cde9 | ||
|
|
0ee426e1f3 | ||
|
|
90a19e4525 | ||
|
|
8b555bf729 | ||
|
|
c8ee194bc5 | ||
|
|
f77362e684 | ||
|
|
9ba13206c5 | ||
|
|
0c7baeb9d8 | ||
|
|
07b0309adb | ||
|
|
7ec074a043 | ||
|
|
b7757ca406 | ||
|
|
d55f58985b | ||
|
|
61835ae3aa | ||
|
|
091315c188 | ||
|
|
089b9e7e9d | ||
|
|
3e7978c918 | ||
|
|
317468b1ed | ||
|
|
a720673896 | ||
|
|
1a85c6e1e0 | ||
|
|
7f4c322280 | ||
|
|
5208327b5d | ||
|
|
ac76871d60 | ||
|
|
66c86c2625 | ||
|
|
56d0c21b79 | ||
|
|
37e81e773e | ||
|
|
534da11217 | ||
|
|
2212755772 |
21
LICENSE
Normal file
21
LICENSE
Normal file
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Steven Jannette
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
149
README.md
149
README.md
@@ -1,40 +1,147 @@
|
||||
# kongruity app
|
||||
# kongruity: Signal from noise
|
||||
|
||||
kongruity employs Large Language Model ("LLM") semantic grouping functionality to cluster large volumes of "to dos" or issue tags in development (or other) settings, according to thematic or topical similarity.
|
||||
“...All those moments will be lost in time, like tears in rain.”
|
||||
|
||||
(To learn more about this topic, see, e.g., [Kozlowski A., Boutyline A., Semantic Structure in Large Language Model Embeddings Aug. 2025, arXiv:2508.10003v1:04 Aug 2025](https://arxiv.org/html/2508.10003v1)).
|
||||
kongruity pulls in unstructured artifacts of the creative-engineering process -- to-dos, action items, agile tickets, Jira thread comments, Slack thread comments, retrospective notes -- and synthesizes them into semantically coherent, prioritized clusters that can be incorporated into implementation planning.
|
||||
|
||||
In the world of kongruity, these "to dos" are called "sticky notes." kongruity's React/Vite UI views a board of seemingly chaotic "sticky notes". But with one click, they are transformed into manageable, actionable groups, each with a header that explains the group semantic interrelation.
|
||||
In kongruity, the artifacts become "sticky notes." A board full of them looks chaotic.
|
||||
|
||||
The backend is an Express API that serves "sticky note" data and proxies semantic grouping requests to Antrhopic Claude.
|
||||
With a click, they are semantically evaluated, grouped into thematic clusters with descriptive headers, rankable and exportable to project planning and execution tools.
|
||||
|
||||
Developers may feel free to install other LLM SDKs and alter the syntax at backend/services/clustering.service.js to experiment with any LLM model/platform they prefer.
|
||||
## Clustering and evaluation: methodology
|
||||
|
||||
Two models run in parallel, and neither sees the other's work. Anthropic's `claude-sonnet-5` (`backend/services/clustering.service.js`) reads the raw text of every note and groups them into labeled thematic clusters.
|
||||
|
||||
At the same time, Voyage AI's voyage-3 model (`backend/services/embedding.service.js`) converts each note's text into a numeric representation of its semantic meaning aka vector.
|
||||
|
||||
Once the LLM returns, kongruity scores that grouping (`backend/services/validation.service.js`) using an established silhouette coefficient, with cosine distance rather than Euclidean as the proximity metric.
|
||||
|
||||
For each note, it weighs the average distance to the other notes in its own cluster against the average distance to the notes in the nearest neighboring cluster. Averaged across every note, this yields a single numeric cohesion score, displayed at the top of the results, along with plaintext: Strong, Moderate, Weak, Poor.
|
||||
|
||||
This yields an empirical groundedness evaluation. One model proposes the grouping; an independent model evaluates grouping accuracy.
|
||||
|
||||
Note that: before scoring, structural validation confirms that each note landed in exactly one cluster, that no cluster is empty, and that no hallucinated note IDs appear. A malformed response to the validation completely fails, rather than quietly returning a partial board.
|
||||
|
||||
## Reading the cohesion score
|
||||
|
||||
Average silhouette width is a widely-used measure of clustering quality. Higher values indicate:
|
||||
|
||||
1. The qualitative semantic cohesiveness of clusters, and:
|
||||
|
||||
2. How well-separated each cluster is from its nearest neighboring cluster.
|
||||
|
||||
Although the coefficient is mathematically bounded by [−1, 1], cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. Because silhouette divides the gap between them by the larger of the two, the practical range on embedding data is roughly [−0.05, 0.10] rather than the full interval.
|
||||
|
||||
The bands below are therefore calibrated against that observed range. On the seed board, the five ideal thematic clusters score 0.09; swapping a few notes between clusters drops it to 0.06; a scrambled assignment falls below zero.
|
||||
|
||||
The score appears above the results with a plain-language band:
|
||||
|
||||
- **0.07 and above** — Strong
|
||||
- **0.04 to 0.06** — Moderate
|
||||
- **0.01 to 0.03** — Weak
|
||||
- **Below 0.01** — Poor
|
||||
|
||||
A score near 0.00 means the grouping is no better than chance. Bands are specific to `voyage-3` cosine distance and would need recalibration behind a different embedding model. See Hugo Sträng, Tai Dinh. An upper bound on the silhouette evaluation metric for clustering. Pattern Recognition, Volume 178, 2026, 113402, ISSN 0031-3203.
|
||||
|
||||
## Organizing clusters, exporting to workflow software
|
||||
|
||||
Teams can drag-and-rank related task clusters by implementation priority - turning noise into an actionable workflow.
|
||||
|
||||
(Integrations with third-party project management, planning and workflow applications are action-items for next major version, see Roadmap, below)
|
||||
|
||||
## How it works
|
||||
|
||||
1. **Ingest** — Sticky notes are loaded and displayed on a board.
|
||||
2. **Cluster** — An LLM reads every note and groups them by semantic similarity (not keywords).
|
||||
3. **Evaluate** — In parallel, a separate embedding model (Voyage AI) generates vector representations of each note. A silhouette-based cohesion score measures how well-separated and internally consistent clusters are. The score is displayed alongside the results.
|
||||
4. **Validate** — Structural checks confirm every note is assigned to exactly one cluster, no clusters are empty, and labels are present.
|
||||
5. **Prioritize** — Clusters appear ranked and are drag-reorderable. Teams set implementation priority by dragging clusters into position.
|
||||
|
||||
## Dev implementation notes
|
||||
|
||||
Developers may swap in other LLM SDKs/APIs and alter prompt syntax in `backend/services/clustering.service.js` to experiment with LLMs and platforms of their choice.
|
||||
|
||||
## Development Roadmap
|
||||
|
||||
### Ingestion — pulling tagged artifacts in
|
||||
|
||||
- [ ] **Slack** — where decisions actually get made; a `:sticky:` emoji reaction fires an Events API webhook that pulls the message in.
|
||||
- [ ] **Microsoft Teams** — same capture gesture for enterprise shops; message extension plus Graph change notifications.
|
||||
- [ ] **Jira** — label- or mention-triggered webhook scoped by JQL. (This is where comments typically carry half the backlog's context.)
|
||||
- [ ] **Linear** — engineering-side tickets and threads; label-triggered GraphQL webhook.
|
||||
- [ ] **GitHub** — issue, PR review, and discussion comments; label- or mention-triggered webhook.
|
||||
- [ ] **Miro / FigJam** — REST API import
|
||||
- [ ] **Confluence / Notion** — page and inline-comment fetch. (Where retro and planning notes are born).
|
||||
- [ ] **Meeting transcripts (Granola, Otter, Zoom, Google Meet)** — where retros are now recorded, an option for action-item extraction from the transcript API.
|
||||
- [ ] **Generic REST, email, and Zapier** — authenticated bulk `POST /v1/notes`.
|
||||
|
||||
### Export — pushing ranked clusters to workflow tools
|
||||
|
||||
- [ ] **Jira** — drag-rank written through the Agile API's board rank endpoint.
|
||||
- [ ] **Asana** — drag-rank written as task order within the section.
|
||||
- [ ] **Rally** — drag-rank written as portfolio rank. (Clusters become features and notes, which become stories).
|
||||
- [ ] **Linear** — drag-rank written to issue `sortOrder`.
|
||||
- [ ] **Azure DevOps / GitHub Projects v2** — drag-rank written as project field ordering. Clusters become work-item parents.
|
||||
- [ ] **CSV, JSON, and Markdown** — direct download from the cluster view. (Should ship before any OAuth work.)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js (v18 or later recommended)
|
||||
- An [Anthropic API key](https://console.anthropic.com/)
|
||||
- PostgreSQL (v14 or later recommended)
|
||||
- An [Anthropic API key](https://console.anthropic.com/) (or other LLM platform, for clustering)
|
||||
- A [Voyage AI API key](https://dash.voyageai.com/) (for embedding-based evaluation)
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Unzip the project
|
||||
### 1. Clone the repository
|
||||
|
||||
```bash
|
||||
unzip kongruity.zip
|
||||
git clone https://github.com/kjannette/kongruity_
|
||||
cd kongruity
|
||||
```
|
||||
|
||||
### 2. Create an environment file
|
||||
|
||||
The backend expects a `.env` file containing an Anthropic API key in the root `backend/` directory. This file is git-ignored and must be created manually:
|
||||
The backend expects a `.env` file in the `backend/` directory. This file is git-ignored and must be created manually:
|
||||
|
||||
```bash
|
||||
echo 'ANTHROPIC_API_KEY=<your Anthropic API key>' > backend/.env
|
||||
cat > backend/.env << 'EOF'
|
||||
ANTHROPIC_API_KEY=<your Anthropic API key> (or other LLM platform key)
|
||||
VOYAGEAI_API_KEY=<your Voyage AI API key>
|
||||
DATABASE_URL=postgresql://<user>:<password>@localhost:5432/kongruity
|
||||
EOF
|
||||
```
|
||||
|
||||
Replace `<your Anthropic API key>` with your actual key.
|
||||
Replace placeholder values with your actual keys and database credentials.
|
||||
|
||||
### 3. Install dependencies
|
||||
### 3. Set up/run the database
|
||||
|
||||
Start DB for local development (assumes local dev env MacOS and Homebrew installed)
|
||||
|
||||
```bash
|
||||
brew services start postgresql@15
|
||||
```
|
||||
|
||||
Create a PostgreSQL database for the project:
|
||||
|
||||
```bash
|
||||
createdb kongruity
|
||||
```
|
||||
|
||||
Run the migration to create tables:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npm run db:migrate
|
||||
```
|
||||
|
||||
Seed the database with the sample sticky notes:
|
||||
|
||||
```bash
|
||||
npm run db:seed
|
||||
```
|
||||
|
||||
### 4. Install dependencies
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
@@ -48,7 +155,7 @@ npm install
|
||||
|
||||
## Run the app
|
||||
|
||||
### Start the backend - Production Mode
|
||||
### Start the backend — Production mode
|
||||
|
||||
From the `backend/` directory:
|
||||
|
||||
@@ -58,15 +165,15 @@ npm run start
|
||||
|
||||
The API server starts on **http://localhost:3001** (configurable via the `PORT` environment variable).
|
||||
|
||||
### Start the backend - Development mode
|
||||
### Start the backend — Development mode
|
||||
|
||||
To start using Nodemon for "hot reloads," if developing your own features:
|
||||
To start using Nodemon for hot reloads while developing:
|
||||
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Build the frontend - Production mode
|
||||
### Build the frontend — Production mode
|
||||
|
||||
From the `frontend/` directory:
|
||||
|
||||
@@ -74,13 +181,14 @@ From the `frontend/` directory:
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Start the frontend - Development mode
|
||||
### Start the frontend — Development mode
|
||||
|
||||
From the `frontend/` directory:
|
||||
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
The Vite dev server starts on **http://localhost:5173** by default. Open that URL in a browser.
|
||||
|
||||
## Running tests
|
||||
@@ -92,7 +200,8 @@ From the `backend/` directory:
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
Backend tests use Supertest for HTTP assertions.
|
||||
|
||||
Backend tests use Vitest with Supertest for HTTP assertions.
|
||||
|
||||
### Frontend tests
|
||||
|
||||
@@ -102,7 +211,7 @@ From the `frontend/` directory:
|
||||
npm test
|
||||
```
|
||||
|
||||
This runs `vitest run` with jsdom. For watch mode, during development:
|
||||
This runs Vitest with jsdom. For watch mode during development:
|
||||
|
||||
```bash
|
||||
npm run test:watch
|
||||
|
||||
@@ -6,11 +6,10 @@ const app = express();
|
||||
|
||||
app.use(cors());
|
||||
app.use(express.json());
|
||||
|
||||
app.use('/v1/notes', router);
|
||||
|
||||
app.use((req, res) => {
|
||||
res.status(404).json({ error: `Requested path is invalid or does not exist: ${req.method} ${req.originalUrl}` });
|
||||
});
|
||||
res.status(404).json({ error: `Requested path is invalid or does not exist: ${req.method} ${req.originalUrl}` });
|
||||
});
|
||||
|
||||
export default app;
|
||||
|
||||
@@ -1,402 +0,0 @@
|
||||
[
|
||||
{
|
||||
"id": "note_038",
|
||||
"text": "Live follow mode frequently breaks",
|
||||
"x": 710,
|
||||
"y": 579,
|
||||
"author": "user_9",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_005",
|
||||
"text": "SSO login loops back to the sign-in page",
|
||||
"x": 184,
|
||||
"y": 124,
|
||||
"author": "user_7",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_023",
|
||||
"text": "Undo/redo sometimes lags and applies late",
|
||||
"x": 236,
|
||||
"y": 687,
|
||||
"author": "user_9",
|
||||
"color": "purple"
|
||||
},
|
||||
{
|
||||
"id": "note_017",
|
||||
"text": "Exported file names are inconsistent and hard to track",
|
||||
"x": 676,
|
||||
"y": 201,
|
||||
"author": "user_4",
|
||||
"color": "blue"
|
||||
},
|
||||
{
|
||||
"id": "note_049",
|
||||
"text": "Need better controls for aligning and distributing notes",
|
||||
"x": 462,
|
||||
"y": 487,
|
||||
"author": "user_7",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_010",
|
||||
"text": "OAuth consent screen appears every time I sign in",
|
||||
"x": 228,
|
||||
"y": 124,
|
||||
"author": "user_9",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_036",
|
||||
"text": "Cursor presence is distracting and overlaps content",
|
||||
"x": 788,
|
||||
"y": 605,
|
||||
"author": "user_5",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_011",
|
||||
"text": "Need better export options (PDF quality is too low)",
|
||||
"x": 748,
|
||||
"y": 212,
|
||||
"author": "user_9",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_012",
|
||||
"text": "Exported PDF cuts off content near the edges",
|
||||
"x": 733,
|
||||
"y": 159,
|
||||
"author": "user_6",
|
||||
"color": "blue"
|
||||
},
|
||||
{
|
||||
"id": "note_019",
|
||||
"text": "Embedded exports don’t update when the board changes",
|
||||
"x": 662,
|
||||
"y": 132,
|
||||
"author": "user_1",
|
||||
"color": "blue"
|
||||
},
|
||||
{
|
||||
"id": "note_020",
|
||||
"text": "Can’t export comments and reactions with the content",
|
||||
"x": 739,
|
||||
"y": 139,
|
||||
"author": "user_7",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_039",
|
||||
"text": "Guest collaborators can’t see updates without refreshing",
|
||||
"x": 759,
|
||||
"y": 711,
|
||||
"author": "user_9",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_024",
|
||||
"text": "App crashes when opening a large board",
|
||||
"x": 239,
|
||||
"y": 597,
|
||||
"author": "user_10",
|
||||
"color": "purple"
|
||||
},
|
||||
{
|
||||
"id": "note_041",
|
||||
"text": "Hard to find the right template quickly",
|
||||
"x": 536,
|
||||
"y": 394,
|
||||
"author": "user_4",
|
||||
"color": "orange"
|
||||
},
|
||||
{
|
||||
"id": "note_040",
|
||||
"text": "Activity feed lacks context about what changed",
|
||||
"x": 710,
|
||||
"y": 771,
|
||||
"author": "user_7",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_014",
|
||||
"text": "Export takes too long and sometimes never finishes",
|
||||
"x": 798,
|
||||
"y": 211,
|
||||
"author": "user_2",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_031",
|
||||
"text": "Hard to tell who is editing what in real time",
|
||||
"x": 761,
|
||||
"y": 704,
|
||||
"author": "user_8",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_032",
|
||||
"text": "Comments get lost — no clear thread view",
|
||||
"x": 818,
|
||||
"y": 641,
|
||||
"author": "user_5",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_050",
|
||||
"text": "Can’t lock sections to prevent accidental edits during workshops",
|
||||
"x": 521,
|
||||
"y": 471,
|
||||
"author": "user_6",
|
||||
"color": "orange"
|
||||
},
|
||||
{
|
||||
"id": "note_001",
|
||||
"text": "Login flow feels confusing",
|
||||
"x": 193,
|
||||
"y": 191,
|
||||
"author": "user_5",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_025",
|
||||
"text": "Saving indicator spins but changes aren’t saved",
|
||||
"x": 129,
|
||||
"y": 781,
|
||||
"author": "user_3",
|
||||
"color": "purple"
|
||||
},
|
||||
{
|
||||
"id": "note_021",
|
||||
"text": "Board feels slow when there are many sticky notes",
|
||||
"x": 341,
|
||||
"y": 695,
|
||||
"author": "user_10",
|
||||
"color": "purple"
|
||||
},
|
||||
{
|
||||
"id": "note_002",
|
||||
"text": "Login flow is broken on mobile",
|
||||
"x": 214,
|
||||
"y": 281,
|
||||
"author": "user_9",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_046",
|
||||
"text": "Tags are missing — I need better categorization",
|
||||
"x": 472,
|
||||
"y": 435,
|
||||
"author": "user_6",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_018",
|
||||
"text": "No way to schedule recurring exports for stakeholders",
|
||||
"x": 661,
|
||||
"y": 229,
|
||||
"author": "user_9",
|
||||
"color": "blue"
|
||||
},
|
||||
{
|
||||
"id": "note_034",
|
||||
"text": "Too many notification emails for minor edits",
|
||||
"x": 769,
|
||||
"y": 739,
|
||||
"author": "user_9",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_003",
|
||||
"text": "When I enter my username and password I get an unknown error",
|
||||
"x": 189,
|
||||
"y": 193,
|
||||
"author": "user_2",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_008",
|
||||
"text": "Cannot change my password — save button does nothing",
|
||||
"x": 244,
|
||||
"y": 188,
|
||||
"author": "user_2",
|
||||
"color": "orange"
|
||||
},
|
||||
{
|
||||
"id": "note_015",
|
||||
"text": "Sharing link permissions are confusing",
|
||||
"x": 688,
|
||||
"y": 290,
|
||||
"author": "user_6",
|
||||
"color": "blue"
|
||||
},
|
||||
{
|
||||
"id": "note_033",
|
||||
"text": "Mentions (@) don’t notify the right people",
|
||||
"x": 696,
|
||||
"y": 669,
|
||||
"author": "user_5",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_006",
|
||||
"text": "Two-factor code is rejected even when correct",
|
||||
"x": 162,
|
||||
"y": 213,
|
||||
"author": "user_1",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_028",
|
||||
"text": "High CPU usage even when idle on a board",
|
||||
"x": 190,
|
||||
"y": 695,
|
||||
"author": "user_8",
|
||||
"color": "purple"
|
||||
},
|
||||
{
|
||||
"id": "note_009",
|
||||
"text": "Account gets locked too quickly after one failed attempt",
|
||||
"x": 280,
|
||||
"y": 255,
|
||||
"author": "user_10",
|
||||
"color": "orange"
|
||||
},
|
||||
{
|
||||
"id": "note_029",
|
||||
"text": "Offline mode loses edits when reconnecting",
|
||||
"x": 338,
|
||||
"y": 632,
|
||||
"author": "user_1",
|
||||
"color": "pink"
|
||||
},
|
||||
{
|
||||
"id": "note_035",
|
||||
"text": "No notification when someone resolves my comment",
|
||||
"x": 673,
|
||||
"y": 636,
|
||||
"author": "user_2",
|
||||
"color": "blue"
|
||||
},
|
||||
{
|
||||
"id": "note_037",
|
||||
"text": "Can’t easily hand off facilitation to another user",
|
||||
"x": 816,
|
||||
"y": 707,
|
||||
"author": "user_2",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_013",
|
||||
"text": "Cannot export selected area — only full board exports",
|
||||
"x": 696,
|
||||
"y": 205,
|
||||
"author": "user_4",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_016",
|
||||
"text": "Downloaded image is blurry compared to the canvas",
|
||||
"x": 677,
|
||||
"y": 245,
|
||||
"author": "user_5",
|
||||
"color": "blue"
|
||||
},
|
||||
{
|
||||
"id": "note_043",
|
||||
"text": "Can’t organize boards into folders the way I need",
|
||||
"x": 504,
|
||||
"y": 398,
|
||||
"author": "user_4",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_022",
|
||||
"text": "Canvas freezes for a few seconds when zooming",
|
||||
"x": 254,
|
||||
"y": 707,
|
||||
"author": "user_8",
|
||||
"color": "pink"
|
||||
},
|
||||
{
|
||||
"id": "note_026",
|
||||
"text": "Search is slow on boards with lots of content",
|
||||
"x": 253,
|
||||
"y": 658,
|
||||
"author": "user_2",
|
||||
"color": "pink"
|
||||
},
|
||||
{
|
||||
"id": "note_048",
|
||||
"text": "Duplicating a board loses some formatting",
|
||||
"x": 382,
|
||||
"y": 410,
|
||||
"author": "user_10",
|
||||
"color": "orange"
|
||||
},
|
||||
{
|
||||
"id": "note_007",
|
||||
"text": "I’m logged out unexpectedly after a few minutes",
|
||||
"x": 185,
|
||||
"y": 157,
|
||||
"author": "user_3",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_045",
|
||||
"text": "No bulk rename for multiple sticky notes",
|
||||
"x": 455,
|
||||
"y": 427,
|
||||
"author": "user_1",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_042",
|
||||
"text": "Template search results feel irrelevant",
|
||||
"x": 400,
|
||||
"y": 474,
|
||||
"author": "user_6",
|
||||
"color": "orange"
|
||||
},
|
||||
{
|
||||
"id": "note_044",
|
||||
"text": "Naming conventions aren’t enforced and things get messy",
|
||||
"x": 469,
|
||||
"y": 434,
|
||||
"author": "user_9",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_004",
|
||||
"text": "Password reset email never arrives",
|
||||
"x": 207,
|
||||
"y": 267,
|
||||
"author": "user_9",
|
||||
"color": "yellow"
|
||||
},
|
||||
{
|
||||
"id": "note_047",
|
||||
"text": "Hard to keep consistent styles across boards",
|
||||
"x": 420,
|
||||
"y": 426,
|
||||
"author": "user_8",
|
||||
"color": "green"
|
||||
},
|
||||
{
|
||||
"id": "note_030",
|
||||
"text": "I see random 'something went wrong' banners with no details",
|
||||
"x": 214,
|
||||
"y": 594,
|
||||
"author": "user_5",
|
||||
"color": "purple"
|
||||
},
|
||||
{
|
||||
"id": "note_027",
|
||||
"text": "Scrolling stutters on older laptops",
|
||||
"x": 178,
|
||||
"y": 586,
|
||||
"author": "user_7",
|
||||
"color": "pink"
|
||||
}
|
||||
]
|
||||
131
backend/db/README.md
Normal file
131
backend/db/README.md
Normal file
@@ -0,0 +1,131 @@
|
||||
# Database — kongruity backend
|
||||
|
||||
PostgreSQL database layer for the kongruity backend. All database modules live in this directory.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- PostgreSQL v14 or later
|
||||
- The `DATABASE_URL` environment variable set in `backend/.env`
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment variable
|
||||
|
||||
The connection pool reads a single env var:
|
||||
|
||||
```
|
||||
DATABASE_URL=postgresql://<user>:<password>@localhost:5432/kongruity
|
||||
```
|
||||
|
||||
This is loaded via `dotenv` from `backend/.env` (git-ignored). The pool is created once in `index.js` and shared across the application.
|
||||
|
||||
### Connection pool (`index.js`)
|
||||
|
||||
| Export | Description |
|
||||
|-------------|--------------------------------------------------|
|
||||
| `query` | Execute a parameterized SQL statement |
|
||||
| `getPool` | Return the underlying `pg.Pool` instance |
|
||||
| `close` | Gracefully shut down the pool (`pool.end()`) |
|
||||
| `default` | The pool itself (default export) |
|
||||
|
||||
## Schema
|
||||
|
||||
### `notes` table
|
||||
|
||||
Created by the migration in `migrate.js`. The DDL is idempotent (`CREATE TABLE IF NOT EXISTS`).
|
||||
|
||||
| Column | Type | Constraints / Defaults |
|
||||
|---------------|----------------|------------------------------|
|
||||
| `id` | `VARCHAR(64)` | `PRIMARY KEY` |
|
||||
| `text` | `TEXT` | `NOT NULL` |
|
||||
| `x` | `INTEGER` | `DEFAULT 0` |
|
||||
| `y` | `INTEGER` | `DEFAULT 0` |
|
||||
| `author` | `VARCHAR(128)` | |
|
||||
| `color` | `VARCHAR(32)` | `DEFAULT 'yellow'` |
|
||||
| `source_meta` | `JSONB` | `DEFAULT '{}'` |
|
||||
| `created_at` | `TIMESTAMPTZ` | `DEFAULT NOW()` |
|
||||
| `updated_at` | `TIMESTAMPTZ` | `DEFAULT NOW()` |
|
||||
|
||||
## npm scripts
|
||||
|
||||
Run these from the `backend/` directory.
|
||||
|
||||
### Migrate
|
||||
|
||||
Creates the `notes` table (safe to re-run):
|
||||
|
||||
```bash
|
||||
npm run db:migrate
|
||||
```
|
||||
|
||||
### Seed
|
||||
|
||||
Inserts 50 sample sticky notes. Uses `ON CONFLICT (id) DO NOTHING`, so re-running is safe and will not duplicate data:
|
||||
|
||||
```bash
|
||||
npm run db:seed
|
||||
```
|
||||
|
||||
## Data access layer (`notes.dao.js`)
|
||||
|
||||
| Function | Description |
|
||||
|---------------------------|--------------------------------------------------------------------|
|
||||
| `getAllNotes()` | Returns all notes ordered by `id` |
|
||||
| `getNoteById(id)` | Returns a single note by primary key, or `null` if not found |
|
||||
| `createNote(note)` | Inserts one note and returns the created row |
|
||||
| `createNotes(notes)` | Bulk-inserts an array of notes in a single query and returns rows |
|
||||
|
||||
All functions return plain objects with columns: `id`, `text`, `x`, `y`, `author`, `color`.
|
||||
|
||||
## File overview
|
||||
|
||||
```
|
||||
db/
|
||||
├── index.js # Connection pool and query helper
|
||||
├── migrate.js # Table creation (run via npm run db:migrate)
|
||||
├── notes.dao.js # Data access functions for the notes table
|
||||
├── seed.js # Sample data seeder (run via npm run db:seed)
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Useful psql commands
|
||||
|
||||
Connect to the database:
|
||||
|
||||
```bash
|
||||
psql kongruity
|
||||
```
|
||||
|
||||
Quick checks:
|
||||
|
||||
```sql
|
||||
-- Row count
|
||||
SELECT count(*) FROM notes;
|
||||
|
||||
-- Preview data
|
||||
SELECT id, text, color FROM notes LIMIT 10;
|
||||
|
||||
-- Full schema info
|
||||
\d notes
|
||||
|
||||
-- Drop and re-seed (destructive)
|
||||
TRUNCATE notes;
|
||||
```
|
||||
|
||||
Then re-seed:
|
||||
|
||||
```bash
|
||||
npm run db:seed
|
||||
```
|
||||
|
||||
## Resetting the database
|
||||
|
||||
To start completely fresh:
|
||||
|
||||
```bash
|
||||
dropdb kongruity
|
||||
createdb kongruity
|
||||
cd backend
|
||||
npm run db:migrate
|
||||
npm run db:seed
|
||||
```
|
||||
50
backend/db/fixtures/notes.jsonl
Normal file
50
backend/db/fixtures/notes.jsonl
Normal file
@@ -0,0 +1,50 @@
|
||||
{"id":"note_001","text":"Login flow feels confusing","x":193,"y":191,"author":"user_5","color":"yellow"}
|
||||
{"id":"note_002","text":"Login flow is broken on mobile","x":214,"y":281,"author":"user_9","color":"yellow"}
|
||||
{"id":"note_003","text":"When I enter my username and password I get an unknown error","x":189,"y":193,"author":"user_2","color":"yellow"}
|
||||
{"id":"note_004","text":"Password reset email never arrives","x":207,"y":267,"author":"user_9","color":"yellow"}
|
||||
{"id":"note_005","text":"SSO login loops back to the sign-in page","x":184,"y":124,"author":"user_7","color":"yellow"}
|
||||
{"id":"note_006","text":"Two-factor code is rejected even when correct","x":162,"y":213,"author":"user_1","color":"yellow"}
|
||||
{"id":"note_007","text":"I'm logged out unexpectedly after a few minutes","x":185,"y":157,"author":"user_3","color":"yellow"}
|
||||
{"id":"note_008","text":"Cannot change my password — save button does nothing","x":244,"y":188,"author":"user_2","color":"orange"}
|
||||
{"id":"note_009","text":"Account gets locked too quickly after one failed attempt","x":280,"y":255,"author":"user_10","color":"orange"}
|
||||
{"id":"note_010","text":"OAuth consent screen appears every time I sign in","x":228,"y":124,"author":"user_9","color":"yellow"}
|
||||
{"id":"note_011","text":"Need better export options (PDF quality is too low)","x":748,"y":212,"author":"user_9","color":"green"}
|
||||
{"id":"note_012","text":"Exported PDF cuts off content near the edges","x":733,"y":159,"author":"user_6","color":"blue"}
|
||||
{"id":"note_013","text":"Cannot export selected area — only full board exports","x":696,"y":205,"author":"user_4","color":"green"}
|
||||
{"id":"note_014","text":"Export takes too long and sometimes never finishes","x":798,"y":211,"author":"user_2","color":"green"}
|
||||
{"id":"note_015","text":"Sharing link permissions are confusing","x":688,"y":290,"author":"user_6","color":"blue"}
|
||||
{"id":"note_016","text":"Downloaded image is blurry compared to the canvas","x":677,"y":245,"author":"user_5","color":"blue"}
|
||||
{"id":"note_017","text":"Exported file names are inconsistent and hard to track","x":676,"y":201,"author":"user_4","color":"blue"}
|
||||
{"id":"note_018","text":"No way to schedule recurring exports for stakeholders","x":661,"y":229,"author":"user_9","color":"blue"}
|
||||
{"id":"note_019","text":"Embedded exports don't update when the board changes","x":662,"y":132,"author":"user_1","color":"blue"}
|
||||
{"id":"note_020","text":"Can't export comments and reactions with the content","x":739,"y":139,"author":"user_7","color":"green"}
|
||||
{"id":"note_021","text":"Board feels slow when there are many sticky notes","x":341,"y":695,"author":"user_10","color":"purple"}
|
||||
{"id":"note_022","text":"Canvas freezes for a few seconds when zooming","x":254,"y":707,"author":"user_8","color":"pink"}
|
||||
{"id":"note_023","text":"Undo/redo sometimes lags and applies late","x":236,"y":687,"author":"user_9","color":"purple"}
|
||||
{"id":"note_024","text":"App crashes when opening a large board","x":239,"y":597,"author":"user_10","color":"purple"}
|
||||
{"id":"note_025","text":"Saving indicator spins but changes aren't saved","x":129,"y":781,"author":"user_3","color":"purple"}
|
||||
{"id":"note_026","text":"Search is slow on boards with lots of content","x":253,"y":658,"author":"user_2","color":"pink"}
|
||||
{"id":"note_027","text":"Scrolling stutters on older laptops","x":178,"y":586,"author":"user_7","color":"pink"}
|
||||
{"id":"note_028","text":"High CPU usage even when idle on a board","x":190,"y":695,"author":"user_8","color":"purple"}
|
||||
{"id":"note_029","text":"Offline mode loses edits when reconnecting","x":338,"y":632,"author":"user_1","color":"pink"}
|
||||
{"id":"note_030","text":"I see random 'something went wrong' banners with no details","x":214,"y":594,"author":"user_5","color":"purple"}
|
||||
{"id":"note_031","text":"Hard to tell who is editing what in real time","x":761,"y":704,"author":"user_8","color":"yellow"}
|
||||
{"id":"note_032","text":"Comments get lost — no clear thread view","x":818,"y":641,"author":"user_5","color":"yellow"}
|
||||
{"id":"note_033","text":"Mentions (@) don't notify the right people","x":696,"y":669,"author":"user_5","color":"yellow"}
|
||||
{"id":"note_034","text":"Too many notification emails for minor edits","x":769,"y":739,"author":"user_9","color":"yellow"}
|
||||
{"id":"note_035","text":"No notification when someone resolves my comment","x":673,"y":636,"author":"user_2","color":"blue"}
|
||||
{"id":"note_036","text":"Cursor presence is distracting and overlaps content","x":788,"y":605,"author":"user_5","color":"yellow"}
|
||||
{"id":"note_037","text":"Can't easily hand off facilitation to another user","x":816,"y":707,"author":"user_2","color":"yellow"}
|
||||
{"id":"note_038","text":"Live follow mode frequently breaks","x":710,"y":579,"author":"user_9","color":"yellow"}
|
||||
{"id":"note_039","text":"Guest collaborators can't see updates without refreshing","x":759,"y":711,"author":"user_9","color":"yellow"}
|
||||
{"id":"note_040","text":"Activity feed lacks context about what changed","x":710,"y":771,"author":"user_7","color":"yellow"}
|
||||
{"id":"note_041","text":"Hard to find the right template quickly","x":536,"y":394,"author":"user_4","color":"orange"}
|
||||
{"id":"note_042","text":"Template search results feel irrelevant","x":400,"y":474,"author":"user_6","color":"orange"}
|
||||
{"id":"note_043","text":"Can't organize boards into folders the way I need","x":504,"y":398,"author":"user_4","color":"green"}
|
||||
{"id":"note_044","text":"Naming conventions aren't enforced and things get messy","x":469,"y":434,"author":"user_9","color":"green"}
|
||||
{"id":"note_045","text":"No bulk rename for multiple sticky notes","x":455,"y":427,"author":"user_1","color":"green"}
|
||||
{"id":"note_046","text":"Tags are missing — I need better categorization","x":472,"y":435,"author":"user_6","color":"green"}
|
||||
{"id":"note_047","text":"Hard to keep consistent styles across boards","x":420,"y":426,"author":"user_8","color":"green"}
|
||||
{"id":"note_048","text":"Duplicating a board loses some formatting","x":382,"y":410,"author":"user_10","color":"orange"}
|
||||
{"id":"note_049","text":"Need better controls for aligning and distributing notes","x":462,"y":487,"author":"user_7","color":"green"}
|
||||
{"id":"note_050","text":"Can't lock sections to prevent accidental edits during workshops","x":521,"y":471,"author":"user_6","color":"orange"}
|
||||
13
backend/db/index.js
Normal file
13
backend/db/index.js
Normal file
@@ -0,0 +1,13 @@
|
||||
import pg from 'pg';
|
||||
|
||||
const pool = new pg.Pool({
|
||||
connectionString: process.env.DATABASE_URL,
|
||||
});
|
||||
|
||||
export const query = (text, params) => pool.query(text, params);
|
||||
|
||||
export const getPool = () => pool;
|
||||
|
||||
export const close = () => pool.end();
|
||||
|
||||
export default pool;
|
||||
30
backend/db/migrate.js
Normal file
30
backend/db/migrate.js
Normal file
@@ -0,0 +1,30 @@
|
||||
import 'dotenv/config';
|
||||
import { query, close } from './index.js';
|
||||
|
||||
const up = `
|
||||
CREATE TABLE IF NOT EXISTS notes (
|
||||
id VARCHAR(64) PRIMARY KEY,
|
||||
text TEXT NOT NULL,
|
||||
x INTEGER DEFAULT 0,
|
||||
y INTEGER DEFAULT 0,
|
||||
author VARCHAR(128),
|
||||
color VARCHAR(32) DEFAULT 'yellow',
|
||||
source_meta JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
`;
|
||||
|
||||
const run = async () => {
|
||||
try {
|
||||
await query(up);
|
||||
console.log('Migration complete — notes table ready.');
|
||||
} catch (err) {
|
||||
console.error('Migration failed:', err.message);
|
||||
process.exit(1);
|
||||
} finally {
|
||||
await close();
|
||||
}
|
||||
};
|
||||
|
||||
run();
|
||||
110
backend/db/notes.dao.js
Normal file
110
backend/db/notes.dao.js
Normal file
@@ -0,0 +1,110 @@
|
||||
import QueryStream from 'pg-query-stream';
|
||||
import { pipeline } from 'node:stream/promises';
|
||||
import { Readable } from 'node:stream';
|
||||
import { query, getPool } from './index.js';
|
||||
import { batch } from '../lib/streams.js';
|
||||
|
||||
const SELECT_NOTES = 'SELECT id, text, x, y, author, color FROM notes ORDER BY id';
|
||||
|
||||
// Postgres caps a statement at 65535 bind parameters; six columns per note
|
||||
// leaves 10922 as the hard ceiling.
|
||||
const INSERT_BATCH_SIZE = 1000;
|
||||
|
||||
export const getAllNotes = async () => {
|
||||
const { rows } = await query(SELECT_NOTES);
|
||||
return rows;
|
||||
};
|
||||
|
||||
/**
|
||||
* Streams every note as an object-mode Readable. The pooled client is released on end, error, or
|
||||
* destruction by consumer.
|
||||
*
|
||||
* @returns {Promise<import('node:stream').Readable>}
|
||||
*/
|
||||
export const streamAllNotes = async () => {
|
||||
const client = await getPool().connect();
|
||||
|
||||
let released = false;
|
||||
const release = () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
client.release();
|
||||
};
|
||||
|
||||
try {
|
||||
const rows = client.query(new QueryStream(SELECT_NOTES));
|
||||
rows.once('end', release);
|
||||
rows.once('error', release);
|
||||
rows.once('close', release);
|
||||
return rows;
|
||||
} catch (err) {
|
||||
release();
|
||||
throw err;
|
||||
}
|
||||
};
|
||||
|
||||
export const getNoteById = async (id) => {
|
||||
const { rows } = await query(
|
||||
'SELECT id, text, x, y, author, color FROM notes WHERE id = $1',
|
||||
[id]
|
||||
);
|
||||
return rows[0] || null;
|
||||
};
|
||||
|
||||
export const createNote = async (note) => {
|
||||
const { rows } = await query(
|
||||
`INSERT INTO notes (id, text, x, y, author, color)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
RETURNING id, text, x, y, author, color`,
|
||||
[note.id, note.text, note.x ?? 0, note.y ?? 0, note.author, note.color ?? 'yellow']
|
||||
);
|
||||
return rows[0];
|
||||
};
|
||||
|
||||
const insertNoteBatch = async (notes) => {
|
||||
const values = [];
|
||||
const placeholders = [];
|
||||
|
||||
notes.forEach((note, i) => {
|
||||
const offset = i * 6;
|
||||
placeholders.push(
|
||||
`($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6})`
|
||||
);
|
||||
values.push(
|
||||
note.id,
|
||||
note.text,
|
||||
note.x ?? 0,
|
||||
note.y ?? 0,
|
||||
note.author,
|
||||
note.color ?? 'yellow'
|
||||
);
|
||||
});
|
||||
|
||||
const { rows } = await query(
|
||||
`INSERT INTO notes (id, text, x, y, author, color)
|
||||
VALUES ${placeholders.join(', ')}
|
||||
RETURNING id, text, x, y, author, color`,
|
||||
values
|
||||
);
|
||||
return rows;
|
||||
};
|
||||
|
||||
export const createNotes = async (notes) => {
|
||||
if (!notes || notes.length === 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const inserted = [];
|
||||
|
||||
await pipeline(
|
||||
Readable.from(notes, { objectMode: true }),
|
||||
batch(INSERT_BATCH_SIZE),
|
||||
async (batches) => {
|
||||
for await (const chunk of batches) {
|
||||
inserted.push(...await insertNoteBatch(chunk));
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
return inserted;
|
||||
};
|
||||
80
backend/db/seed.js
Normal file
80
backend/db/seed.js
Normal file
@@ -0,0 +1,80 @@
|
||||
import 'dotenv/config';
|
||||
import split2 from 'split2';
|
||||
import { from as copyFrom } from 'pg-copy-streams';
|
||||
import { createReadStream } from 'node:fs';
|
||||
import { Transform } from 'node:stream';
|
||||
import { pipeline } from 'node:stream/promises';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { getPool, close } from './index.js';
|
||||
|
||||
const FIXTURE = fileURLToPath(new URL('./fixtures/notes.jsonl', import.meta.url));
|
||||
|
||||
const COLUMNS = ['id', 'text', 'x', 'y', 'author', 'color'];
|
||||
|
||||
const DEFAULTS = { x: 0, y: 0, color: 'yellow' };
|
||||
|
||||
const csvField = (value) => {
|
||||
if (value === null || value === undefined) return '';
|
||||
return `"${String(value).replaceAll('"', '""')}"`;
|
||||
};
|
||||
|
||||
const toCsvRows = () => new Transform({
|
||||
writableObjectMode: true,
|
||||
transform(line, _encoding, callback) {
|
||||
if (line.trim().length === 0) {
|
||||
callback();
|
||||
return;
|
||||
}
|
||||
|
||||
let note;
|
||||
try {
|
||||
note = JSON.parse(line);
|
||||
} catch {
|
||||
callback(new Error(`Fixture contains a malformed JSON line: ${line.slice(0, 80)}`));
|
||||
return;
|
||||
}
|
||||
|
||||
const row = COLUMNS.map((col) => csvField(note[col] ?? DEFAULTS[col]));
|
||||
callback(null, `${row.join(',')}\n`);
|
||||
},
|
||||
});
|
||||
|
||||
const run = async () => {
|
||||
const client = await getPool().connect();
|
||||
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
|
||||
// COPY has no ON CONFLICT, so land the fixture in a temp table first and
|
||||
// let a single INSERT ... SELECT apply the existing idempotency.
|
||||
await client.query(
|
||||
'CREATE TEMP TABLE notes_import (LIKE notes INCLUDING DEFAULTS) ON COMMIT DROP'
|
||||
);
|
||||
|
||||
const copy = client.query(
|
||||
copyFrom(`COPY notes_import (${COLUMNS.join(', ')}) FROM STDIN WITH (FORMAT csv)`)
|
||||
);
|
||||
|
||||
await pipeline(createReadStream(FIXTURE), split2(), toCsvRows(), copy);
|
||||
|
||||
const { rowCount: staged } = await client.query('SELECT 1 FROM notes_import');
|
||||
const { rowCount: inserted } = await client.query(
|
||||
`INSERT INTO notes (${COLUMNS.join(', ')})
|
||||
SELECT ${COLUMNS.join(', ')} FROM notes_import
|
||||
ON CONFLICT (id) DO NOTHING`
|
||||
);
|
||||
|
||||
await client.query('COMMIT');
|
||||
|
||||
console.log(`Seed complete — ${inserted} notes inserted (${staged - inserted} already existed).`);
|
||||
} catch (err) {
|
||||
await client.query('ROLLBACK').catch(() => {});
|
||||
console.error('Seed failed:', err.message);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
client.release();
|
||||
await close();
|
||||
}
|
||||
};
|
||||
|
||||
run();
|
||||
68
backend/lib/streams.js
Normal file
68
backend/lib/streams.js
Normal file
@@ -0,0 +1,68 @@
|
||||
import { Transform } from 'node:stream';
|
||||
|
||||
/**
|
||||
* Backpressure on readable side is limits how many batches are in flight.
|
||||
*
|
||||
* @param {number} size - maximum items per emitted batch
|
||||
* @returns {Transform}
|
||||
*/
|
||||
export const batch = (size) => {
|
||||
if (!Number.isInteger(size) || size < 1) {
|
||||
throw new TypeError('batch(size) requires a positive integer size');
|
||||
}
|
||||
|
||||
let pending = [];
|
||||
|
||||
return new Transform({
|
||||
objectMode: true,
|
||||
transform(item, _encoding, callback) {
|
||||
pending.push(item);
|
||||
|
||||
if (pending.length < size) {
|
||||
callback();
|
||||
return;
|
||||
}
|
||||
|
||||
const full = pending;
|
||||
pending = [];
|
||||
callback(null, full);
|
||||
},
|
||||
flush(callback) {
|
||||
if (pending.length === 0) {
|
||||
callback();
|
||||
return;
|
||||
}
|
||||
|
||||
const remainder = pending;
|
||||
pending = [];
|
||||
callback(null, remainder);
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
/**
|
||||
* @returns {Transform}
|
||||
*/
|
||||
export const jsonArray = () => {
|
||||
let wroteFirst = false;
|
||||
|
||||
return new Transform({
|
||||
writableObjectMode: true,
|
||||
transform(item, _encoding, callback) {
|
||||
let serialized;
|
||||
try {
|
||||
serialized = JSON.stringify(item);
|
||||
} catch (err) {
|
||||
callback(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const prefix = wroteFirst ? ',' : '[';
|
||||
wroteFirst = true;
|
||||
callback(null, prefix + serialized);
|
||||
},
|
||||
flush(callback) {
|
||||
callback(null, wroteFirst ? ']' : '[]');
|
||||
},
|
||||
});
|
||||
};
|
||||
407
backend/package-lock.json
generated
407
backend/package-lock.json
generated
@@ -11,7 +11,11 @@
|
||||
"@anthropic-ai/sdk": "^0.74.0",
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^16.4.7",
|
||||
"express": "^4.21.2"
|
||||
"express": "^4.21.2",
|
||||
"pg": "^8.18.0",
|
||||
"pg-copy-streams": "^7.0.0",
|
||||
"pg-query-stream": "^4.16.0",
|
||||
"voyageai": "^0.1.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"nodemon": "^3.1.9",
|
||||
@@ -1013,6 +1017,18 @@
|
||||
"url": "https://opencollective.com/vitest"
|
||||
}
|
||||
},
|
||||
"node_modules/abort-controller": {
|
||||
"version": "3.0.0",
|
||||
"resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz",
|
||||
"integrity": "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"event-target-shim": "^5.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.5"
|
||||
}
|
||||
},
|
||||
"node_modules/accepts": {
|
||||
"version": "1.3.8",
|
||||
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
|
||||
@@ -1067,7 +1083,6 @@
|
||||
"version": "0.4.0",
|
||||
"resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz",
|
||||
"integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/balanced-match": {
|
||||
@@ -1077,6 +1092,26 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/base64-js": {
|
||||
"version": "1.5.1",
|
||||
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
|
||||
"integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
],
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/binary-extensions": {
|
||||
"version": "2.3.0",
|
||||
"resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz",
|
||||
@@ -1138,6 +1173,30 @@
|
||||
"node": ">=8"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer": {
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz",
|
||||
"integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"base64-js": "^1.3.1",
|
||||
"ieee754": "^1.2.1"
|
||||
}
|
||||
},
|
||||
"node_modules/bytes": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
|
||||
@@ -1215,7 +1274,6 @@
|
||||
"version": "1.0.8",
|
||||
"resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
|
||||
"integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"delayed-stream": "~1.0.0"
|
||||
@@ -1314,7 +1372,6 @@
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
|
||||
"integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.4.0"
|
||||
@@ -1432,7 +1489,6 @@
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz",
|
||||
"integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"es-errors": "^1.3.0",
|
||||
@@ -1511,6 +1567,24 @@
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/event-target-shim": {
|
||||
"version": "5.0.1",
|
||||
"resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz",
|
||||
"integrity": "sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
},
|
||||
"node_modules/events": {
|
||||
"version": "3.3.0",
|
||||
"resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz",
|
||||
"integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.8.x"
|
||||
}
|
||||
},
|
||||
"node_modules/expect-type": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.3.0.tgz",
|
||||
@@ -1609,7 +1683,6 @@
|
||||
"version": "4.0.5",
|
||||
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz",
|
||||
"integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"asynckit": "^0.4.0",
|
||||
@@ -1622,6 +1695,15 @@
|
||||
"node": ">= 6"
|
||||
}
|
||||
},
|
||||
"node_modules/formdata-node": {
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.npmjs.org/formdata-node/-/formdata-node-6.0.3.tgz",
|
||||
"integrity": "sha512-8e1++BCiTzUno9v5IZ2J6bv4RU+3UKDmqWUQD0MIMVCd9AdhWkO1gw57oo1mNEX1dMq2EGI+FbWz4B92pscSQg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
}
|
||||
},
|
||||
"node_modules/formidable": {
|
||||
"version": "3.5.4",
|
||||
"resolved": "https://registry.npmjs.org/formidable/-/formidable-3.5.4.tgz",
|
||||
@@ -1770,7 +1852,6 @@
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz",
|
||||
"integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"has-symbols": "^1.0.3"
|
||||
@@ -1826,6 +1907,26 @@
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ieee754": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
|
||||
"integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/feross"
|
||||
},
|
||||
{
|
||||
"type": "patreon",
|
||||
"url": "https://www.patreon.com/feross"
|
||||
},
|
||||
{
|
||||
"type": "consulting",
|
||||
"url": "https://feross.org/support"
|
||||
}
|
||||
],
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/ignore-by-default": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/ignore-by-default/-/ignore-by-default-1.0.1.tgz",
|
||||
@@ -1894,6 +1995,12 @@
|
||||
"node": ">=0.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/js-base64": {
|
||||
"version": "3.7.2",
|
||||
"resolved": "https://registry.npmjs.org/js-base64/-/js-base64-3.7.2.tgz",
|
||||
"integrity": "sha512-NnRs6dsyqUXejqk/yv2aiXlAvOs56sLkX6nUdeaNezI5LFFLlsZjOThmwnrcwh5ZZRwZlCMnVAY3CvhIhoVEKQ==",
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/json-schema-to-ts": {
|
||||
"version": "3.1.1",
|
||||
"resolved": "https://registry.npmjs.org/json-schema-to-ts/-/json-schema-to-ts-3.1.1.tgz",
|
||||
@@ -2033,6 +2140,26 @@
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/node-fetch": {
|
||||
"version": "2.7.0",
|
||||
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz",
|
||||
"integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"whatwg-url": "^5.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": "4.x || >=6.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"encoding": "^0.1.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"encoding": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/nodemon": {
|
||||
"version": "3.1.11",
|
||||
"resolved": "https://registry.npmjs.org/nodemon/-/nodemon-3.1.11.tgz",
|
||||
@@ -2173,6 +2300,123 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/pg": {
|
||||
"version": "8.18.0",
|
||||
"resolved": "https://registry.npmjs.org/pg/-/pg-8.18.0.tgz",
|
||||
"integrity": "sha512-xqrUDL1b9MbkydY/s+VZ6v+xiMUmOUk7SS9d/1kpyQxoJ6U9AO1oIJyUWVZojbfe5Cc/oluutcgFG4L9RDP1iQ==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"pg-connection-string": "^2.11.0",
|
||||
"pg-pool": "^3.11.0",
|
||||
"pg-protocol": "^1.11.0",
|
||||
"pg-types": "2.2.0",
|
||||
"pgpass": "1.0.5"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 16.0.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"pg-cloudflare": "^1.3.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"pg-native": ">=3.0.1"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"pg-native": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/pg-cloudflare": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.3.0.tgz",
|
||||
"integrity": "sha512-6lswVVSztmHiRtD6I8hw4qP/nDm1EJbKMRhf3HCYaqud7frGysPv7FYJ5noZQdhQtN2xJnimfMtvQq21pdbzyQ==",
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/pg-connection-string": {
|
||||
"version": "2.11.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.11.0.tgz",
|
||||
"integrity": "sha512-kecgoJwhOpxYU21rZjULrmrBJ698U2RxXofKVzOn5UDj61BPj/qMb7diYUR1nLScCDbrztQFl1TaQZT0t1EtzQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/pg-copy-streams": {
|
||||
"version": "7.0.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-copy-streams/-/pg-copy-streams-7.0.0.tgz",
|
||||
"integrity": "sha512-zBvnY6wtaBRE2ae2xXWOOGMaNVPkXh1vhypAkNSKgMdciJeTyIQAHZaEeRAxUjs/p1El5jgzYmwG5u871Zj3dQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/pg-cursor": {
|
||||
"version": "2.21.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-cursor/-/pg-cursor-2.21.0.tgz",
|
||||
"integrity": "sha512-IYvk/j+Suhtbo/C3uOf4JLsLK/gWxOTUOmYbDsbKnLaVJDq+KwhwK6ngpRfiCk8eDMS3AmGQABZCv0cREEzHQw==",
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"pg": "^8"
|
||||
}
|
||||
},
|
||||
"node_modules/pg-int8": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
|
||||
"integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=4.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/pg-pool": {
|
||||
"version": "3.11.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.11.0.tgz",
|
||||
"integrity": "sha512-MJYfvHwtGp870aeusDh+hg9apvOe2zmpZJpyt+BMtzUWlVqbhFmMK6bOBXLBUPd7iRtIF9fZplDc7KrPN3PN7w==",
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"pg": ">=8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/pg-protocol": {
|
||||
"version": "1.11.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.11.0.tgz",
|
||||
"integrity": "sha512-pfsxk2M9M3BuGgDOfuy37VNRRX3jmKgMjcvAcWqNDpZSf4cUmv8HSOl5ViRQFsfARFn0KuUQTgLxVMbNq5NW3g==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/pg-query-stream": {
|
||||
"version": "4.16.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-query-stream/-/pg-query-stream-4.16.0.tgz",
|
||||
"integrity": "sha512-vyqxAG4YVax43BCSfqcKxHSbh8YQshhVR0CjLNdo7MIM2UOqI+XsIYk0bVZJ0aKjQfjJQcGiGo9xK3hz0JcFyg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"pg-cursor": "^2.21.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"pg": "^8"
|
||||
}
|
||||
},
|
||||
"node_modules/pg-types": {
|
||||
"version": "2.2.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
|
||||
"integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"pg-int8": "1.0.1",
|
||||
"postgres-array": "~2.0.0",
|
||||
"postgres-bytea": "~1.0.0",
|
||||
"postgres-date": "~1.0.4",
|
||||
"postgres-interval": "^1.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=4"
|
||||
}
|
||||
},
|
||||
"node_modules/pgpass": {
|
||||
"version": "1.0.5",
|
||||
"resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
|
||||
"integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"split2": "^4.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/picocolors": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
|
||||
@@ -2222,6 +2466,54 @@
|
||||
"node": "^10 || ^12 || >=14"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-array": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
|
||||
"integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=4"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-bytea": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
|
||||
"integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-date": {
|
||||
"version": "1.0.7",
|
||||
"resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
|
||||
"integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-interval": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
|
||||
"integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"xtend": "^4.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/process": {
|
||||
"version": "0.11.10",
|
||||
"resolved": "https://registry.npmjs.org/process/-/process-0.11.10.tgz",
|
||||
"integrity": "sha512-cdGef/drWFoydD1JsMzuFf8100nZl+GT+yacc2bEced5f9Rjk4z+WtFUTBu9PhOi9j/jfmBPu0mMEY4wIdAF8A==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6.0"
|
||||
}
|
||||
},
|
||||
"node_modules/proxy-addr": {
|
||||
"version": "2.0.7",
|
||||
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
|
||||
@@ -2281,6 +2573,22 @@
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/readable-stream": {
|
||||
"version": "4.7.0",
|
||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz",
|
||||
"integrity": "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"abort-controller": "^3.0.0",
|
||||
"buffer": "^6.0.3",
|
||||
"events": "^3.3.0",
|
||||
"process": "^0.11.10",
|
||||
"string_decoder": "^1.3.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^12.22.0 || ^14.17.0 || >=16.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/readdirp": {
|
||||
"version": "3.6.0",
|
||||
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.6.0.tgz",
|
||||
@@ -2531,6 +2839,15 @@
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/split2": {
|
||||
"version": "4.2.0",
|
||||
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
|
||||
"integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">= 10.x"
|
||||
}
|
||||
},
|
||||
"node_modules/stackback": {
|
||||
"version": "0.0.2",
|
||||
"resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz",
|
||||
@@ -2554,6 +2871,15 @@
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/string_decoder": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
|
||||
"integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"safe-buffer": "~5.2.0"
|
||||
}
|
||||
},
|
||||
"node_modules/superagent": {
|
||||
"version": "10.3.0",
|
||||
"resolved": "https://registry.npmjs.org/superagent/-/superagent-10.3.0.tgz",
|
||||
@@ -2759,6 +3085,12 @@
|
||||
"nodetouch": "bin/nodetouch.js"
|
||||
}
|
||||
},
|
||||
"node_modules/tr46": {
|
||||
"version": "0.0.3",
|
||||
"resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz",
|
||||
"integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/ts-algebra": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/ts-algebra/-/ts-algebra-2.0.0.tgz",
|
||||
@@ -2794,6 +3126,12 @@
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/url-join": {
|
||||
"version": "4.0.1",
|
||||
"resolved": "https://registry.npmjs.org/url-join/-/url-join-4.0.1.tgz",
|
||||
"integrity": "sha512-jk1+QP6ZJqyOiuEI9AEWQfju/nB2Pw466kbA0LEZljHwKeMgd9WrAEgEGxjPDD2+TNbbb37rTyhEfrCXfuKXnA==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/utils-merge": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz",
|
||||
@@ -3011,6 +3349,52 @@
|
||||
"url": "https://github.com/sponsors/jonschlinkert"
|
||||
}
|
||||
},
|
||||
"node_modules/voyageai": {
|
||||
"version": "0.1.0",
|
||||
"resolved": "https://registry.npmjs.org/voyageai/-/voyageai-0.1.0.tgz",
|
||||
"integrity": "sha512-QnalSSHRigZqzKJpXLsgp3tKaKkSv2kGO4rcXQ6mr6PKHa0vRirs0jZ72olvFJrGJkbu1hHijcyr6/eI4L5aRw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"form-data": "^4.0.0",
|
||||
"formdata-node": "^6.0.3",
|
||||
"js-base64": "3.7.2",
|
||||
"node-fetch": "2.7.0",
|
||||
"qs": "6.11.2",
|
||||
"readable-stream": "^4.5.2",
|
||||
"url-join": "4.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/voyageai/node_modules/qs": {
|
||||
"version": "6.11.2",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.11.2.tgz",
|
||||
"integrity": "sha512-tDNIz22aBzCDxLtVH++VnTfzxlfeK5CbqohpSqpJgj1Wg/cQbStNAz3NuqCs5vV+pjBsK4x4pN9HlVh7rcYRiA==",
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"side-channel": "^1.0.4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.6"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/webidl-conversions": {
|
||||
"version": "3.0.1",
|
||||
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz",
|
||||
"integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==",
|
||||
"license": "BSD-2-Clause"
|
||||
},
|
||||
"node_modules/whatwg-url": {
|
||||
"version": "5.0.0",
|
||||
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",
|
||||
"integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"tr46": "~0.0.3",
|
||||
"webidl-conversions": "^3.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/why-is-node-running": {
|
||||
"version": "2.3.0",
|
||||
"resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz",
|
||||
@@ -3034,6 +3418,15 @@
|
||||
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/xtend": {
|
||||
"version": "4.0.2",
|
||||
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
|
||||
"integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.4"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,13 +7,19 @@
|
||||
"scripts": {
|
||||
"start": "node server.js",
|
||||
"dev": "nodemon server.js",
|
||||
"test": "vitest run"
|
||||
"test": "vitest run",
|
||||
"db:migrate": "node db/migrate.js",
|
||||
"db:seed": "node db/seed.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"@anthropic-ai/sdk": "^0.74.0",
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^16.4.7",
|
||||
"express": "^4.21.2"
|
||||
"express": "^4.21.2",
|
||||
"pg": "^8.18.0",
|
||||
"pg-copy-streams": "^7.0.0",
|
||||
"pg-query-stream": "^4.16.0",
|
||||
"voyageai": "^0.1.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"nodemon": "^3.1.9",
|
||||
|
||||
@@ -1,33 +1,40 @@
|
||||
import { Router } from 'express';
|
||||
import { readFile } from 'fs/promises';
|
||||
import { pipeline } from 'node:stream/promises';
|
||||
import { getAllNotes, streamAllNotes } from '../db/notes.dao.js';
|
||||
import { clusterNotes } from '../services/clustering.service.js';
|
||||
import { jsonArray } from '../lib/streams.js';
|
||||
|
||||
const router = Router();
|
||||
const DATA_PATH = '../data/notes.json'
|
||||
|
||||
const loadNotes = async () => {
|
||||
const raw = await readFile(DATA_PATH, 'utf-8');
|
||||
return JSON.parse(raw);
|
||||
};
|
||||
|
||||
router.get('/', async (req, res) => {
|
||||
try {
|
||||
const notes = await loadNotes();
|
||||
res.json(notes);
|
||||
const rows = await streamAllNotes();
|
||||
res.type('application/json');
|
||||
await pipeline(rows, jsonArray(), res);
|
||||
} catch (err) {
|
||||
console.error(`Error loading notes: ${err}`)
|
||||
console.error(`Error loading notes: ${err}`);
|
||||
if (res.headersSent) {
|
||||
res.destroy(err);
|
||||
return;
|
||||
}
|
||||
res.status(500).json({ error: 'Failed to load notes' });
|
||||
}
|
||||
});
|
||||
|
||||
router.post('/cluster', async (req, res) => {
|
||||
const controller = new AbortController();
|
||||
res.on('close', () => {
|
||||
if (!res.writableEnded) controller.abort();
|
||||
});
|
||||
|
||||
try {
|
||||
const notes = await loadNotes();
|
||||
const clusters = await clusterNotes(notes);
|
||||
res.json(clusters);
|
||||
const notes = await getAllNotes();
|
||||
const result = await clusterNotes(notes, { signal: controller.signal });
|
||||
res.json(result);
|
||||
} catch (err) {
|
||||
console.error(`Clustering failed: ${err}` )
|
||||
res.status(500).json({ error: `Clustering failed: ${err}` });
|
||||
if (controller.signal.aborted) return;
|
||||
console.error(`Clustering failed: ${err}`);
|
||||
res.status(500).json({ error: 'Clustering failed' });
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { Transform, Writable } from "node:stream";
|
||||
import { embedNotes } from "./embedding.service.js";
|
||||
import { validateStructure, computeCohesionScore } from "./validation.service.js";
|
||||
|
||||
const client = new Anthropic({
|
||||
apiKey: process.env.ANTHROPIC_API_KEY,
|
||||
@@ -31,15 +35,81 @@ Here are the notes:
|
||||
${notesJson}`;
|
||||
};
|
||||
|
||||
export const clusterNotes = async (notes) => {
|
||||
const response = await client.messages.create({
|
||||
model: "claude-sonnet-4-20250514",
|
||||
max_tokens: 4096,
|
||||
messages: [
|
||||
{ role: "user", content: buildPrompt(notes) },
|
||||
],
|
||||
});
|
||||
const textDeltas = () => new Transform({
|
||||
objectMode: true,
|
||||
transform(event, _encoding, callback) {
|
||||
if (event?.type === 'content_block_delta' && event.delta?.type === 'text_delta') {
|
||||
callback(null, event.delta.text);
|
||||
return;
|
||||
}
|
||||
callback();
|
||||
},
|
||||
});
|
||||
|
||||
const raw = response.content[0].text;
|
||||
return JSON.parse(raw);
|
||||
// Rejects as soon as the first non-whitespace character proves the response
|
||||
// is not the JSON array we asked for, rather than after the full generation.
|
||||
const collectClusterJson = (sink) => new Writable({
|
||||
objectMode: true,
|
||||
write(text, _encoding, callback) {
|
||||
if (!sink.sawOpeningBracket) {
|
||||
const leading = (sink.parts.join('') + text).trimStart();
|
||||
if (leading.length > 0) {
|
||||
if (!leading.startsWith('[')) {
|
||||
callback(new Error('LLM API returned non-JSON response'));
|
||||
return;
|
||||
}
|
||||
sink.sawOpeningBracket = true;
|
||||
}
|
||||
}
|
||||
sink.parts.push(text);
|
||||
callback();
|
||||
},
|
||||
});
|
||||
|
||||
const requestClusters = async (notes, signal) => {
|
||||
const sink = { parts: [], sawOpeningBracket: false };
|
||||
|
||||
const options = signal ? [{ signal }] : [];
|
||||
|
||||
const events = client.messages.stream(
|
||||
{
|
||||
model: "claude-sonnet-5",
|
||||
max_tokens: 4096,
|
||||
messages: [
|
||||
{ role: "user", content: buildPrompt(notes) },
|
||||
],
|
||||
},
|
||||
...options
|
||||
);
|
||||
|
||||
await pipeline(events, textDeltas(), collectClusterJson(sink), ...options);
|
||||
|
||||
const text = sink.parts.join('');
|
||||
|
||||
if (text.trim().length === 0) {
|
||||
throw new Error('Unexpected response from LLM API: no text content returned');
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
throw new Error('LLM API returned non-JSON response');
|
||||
}
|
||||
};
|
||||
|
||||
export const clusterNotes = async (notes, { signal } = {}) => {
|
||||
const [clusters, embeddingMap] = await Promise.all([
|
||||
requestClusters(notes, signal),
|
||||
embedNotes(notes),
|
||||
]);
|
||||
|
||||
const noteIds = notes.map((n) => n.id);
|
||||
const { valid, reasons } = validateStructure(clusters, noteIds);
|
||||
if (!valid) {
|
||||
throw new Error(`Cluster validation failed: ${reasons.join('; ')}`);
|
||||
}
|
||||
|
||||
const score = computeCohesionScore(clusters, embeddingMap);
|
||||
|
||||
return { clusters, score: Math.round(score * 100) / 100 };
|
||||
};
|
||||
|
||||
41
backend/services/embedding.service.js
Normal file
41
backend/services/embedding.service.js
Normal file
@@ -0,0 +1,41 @@
|
||||
import { VoyageAIClient } from "voyageai";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { Readable } from "node:stream";
|
||||
import { batch } from "../lib/streams.js";
|
||||
|
||||
const client = new VoyageAIClient({
|
||||
apiKey: process.env.VOYAGEAI_API_KEY,
|
||||
});
|
||||
|
||||
const EMBED_BATCH_SIZE = 128;
|
||||
|
||||
/**
|
||||
* @param {Array<{id: string, text: string}>} notes
|
||||
* @returns {Promise<Map<string, number[]>>} noteId → embedding vector
|
||||
*/
|
||||
export const embedNotes = async (notes) => {
|
||||
const embeddingMap = new Map();
|
||||
|
||||
if (!notes || notes.length === 0) {
|
||||
return embeddingMap;
|
||||
}
|
||||
|
||||
await pipeline(
|
||||
Readable.from(notes, { objectMode: true }),
|
||||
batch(EMBED_BATCH_SIZE),
|
||||
async (batches) => {
|
||||
for await (const chunk of batches) {
|
||||
const response = await client.embed({
|
||||
input: chunk.map((n) => n.text),
|
||||
model: "voyage-3",
|
||||
});
|
||||
|
||||
response.data.forEach((item, i) => {
|
||||
embeddingMap.set(chunk[i].id, item.embedding);
|
||||
});
|
||||
}
|
||||
}
|
||||
);
|
||||
|
||||
return embeddingMap;
|
||||
};
|
||||
128
backend/services/validation.service.js
Normal file
128
backend/services/validation.service.js
Normal file
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* Structural validation for LLM output
|
||||
*
|
||||
* @param {Array<{label: string, noteIds: string[]}>} clusters
|
||||
* @param {string[]} inputNoteIds - the original note IDs that were sent to the LLM
|
||||
* @returns {{valid: boolean, reasons: string[]}}
|
||||
*/
|
||||
export const validateStructure = (clusters, inputNoteIds) => {
|
||||
const reasons = [];
|
||||
|
||||
if (!Array.isArray(clusters) || clusters.length === 0) {
|
||||
return { valid: false, reasons: ['Response is not a non-empty array'] };
|
||||
}
|
||||
|
||||
const assignedIds = [];
|
||||
for (const cluster of clusters) {
|
||||
if (!cluster.label || typeof cluster.label !== 'string') {
|
||||
reasons.push(`Cluster missing a valid label`);
|
||||
}
|
||||
if (!Array.isArray(cluster.noteIds) || cluster.noteIds.length === 0) {
|
||||
reasons.push(`Cluster "${cluster.label ?? '(unlabeled)'}" has no noteIds`);
|
||||
}
|
||||
assignedIds.push(...(cluster.noteIds ?? []));
|
||||
}
|
||||
|
||||
const inputSet = new Set(inputNoteIds);
|
||||
const assignedSet = new Set(assignedIds);
|
||||
|
||||
if (assignedIds.length !== assignedSet.size) {
|
||||
reasons.push('One or more notes appear in multiple clusters');
|
||||
}
|
||||
|
||||
const missing = inputNoteIds.filter((id) => !assignedSet.has(id));
|
||||
if (missing.length > 0) {
|
||||
reasons.push(`Notes missing from clusters: ${missing.join(', ')}`);
|
||||
}
|
||||
|
||||
const extra = assignedIds.filter((id) => !inputSet.has(id));
|
||||
if (extra.length > 0) {
|
||||
reasons.push(`Unknown noteIds in clusters: ${[...new Set(extra)].join(', ')}`);
|
||||
}
|
||||
|
||||
if (clusters.length > inputNoteIds.length) {
|
||||
reasons.push(`More clusters (${clusters.length}) than notes (${inputNoteIds.length})`);
|
||||
}
|
||||
|
||||
return { valid: reasons.length === 0, reasons };
|
||||
};
|
||||
|
||||
const cosineSimilarity = (a, b) => {
|
||||
let dot = 0;
|
||||
let magA = 0;
|
||||
let magB = 0;
|
||||
for (let i = 0; i < a.length; i++) {
|
||||
dot += a[i] * b[i];
|
||||
magA += a[i] * a[i];
|
||||
magB += b[i] * b[i];
|
||||
}
|
||||
const denom = Math.sqrt(magA) * Math.sqrt(magB);
|
||||
return denom === 0 ? 0 : dot / denom;
|
||||
};
|
||||
|
||||
/**
|
||||
* Computes silhouette-style cohesion score for the clustering.
|
||||
*
|
||||
* For each note, measures how much more similar it is to its own cluster
|
||||
* versus the nearest neighboring cluster. Returns a score in [-1, 1]
|
||||
* where higher is better.
|
||||
*
|
||||
* @param {Array<{label: string, noteIds: string[]}>} clusters
|
||||
* @param {Map<string, number[]>} embeddingMap - noteId → vector
|
||||
* @returns {number} average silhouette score
|
||||
*/
|
||||
export const computeCohesionScore = (clusters, embeddingMap) => {
|
||||
if (clusters.length <= 1) return 1.0;
|
||||
|
||||
const scores = [];
|
||||
|
||||
for (let ci = 0; ci < clusters.length; ci++) {
|
||||
const clusterIds = clusters[ci].noteIds;
|
||||
if (clusterIds.length <= 1) {
|
||||
scores.push(0);
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const noteId of clusterIds) {
|
||||
const vec = embeddingMap.get(noteId);
|
||||
if (!vec) continue;
|
||||
|
||||
// a(i): avg distance to other notes in same cluster
|
||||
let intraSum = 0;
|
||||
let intraCount = 0;
|
||||
for (const otherId of clusterIds) {
|
||||
if (otherId === noteId) continue;
|
||||
const otherVec = embeddingMap.get(otherId);
|
||||
if (!otherVec) continue;
|
||||
intraSum += 1 - cosineSimilarity(vec, otherVec);
|
||||
intraCount++;
|
||||
}
|
||||
const a = intraCount > 0 ? intraSum / intraCount : 0;
|
||||
|
||||
// b(i): min avg distance to notes in any other cluster
|
||||
let b = Infinity;
|
||||
for (let oi = 0; oi < clusters.length; oi++) {
|
||||
if (oi === ci) continue;
|
||||
const otherClusterIds = clusters[oi].noteIds;
|
||||
let interSum = 0;
|
||||
let interCount = 0;
|
||||
for (const otherId of otherClusterIds) {
|
||||
const otherVec = embeddingMap.get(otherId);
|
||||
if (!otherVec) continue;
|
||||
interSum += 1 - cosineSimilarity(vec, otherVec);
|
||||
interCount++;
|
||||
}
|
||||
if (interCount > 0) {
|
||||
b = Math.min(b, interSum / interCount);
|
||||
}
|
||||
}
|
||||
if (b === Infinity) b = 0;
|
||||
|
||||
const max = Math.max(a, b);
|
||||
scores.push(max === 0 ? 0 : (b - a) / max);
|
||||
}
|
||||
}
|
||||
|
||||
if (scores.length === 0) return 0;
|
||||
return scores.reduce((sum, s) => sum + s, 0) / scores.length;
|
||||
};
|
||||
@@ -1,19 +1,27 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
|
||||
const { createMock } = vi.hoisted(() => {
|
||||
return { createMock: vi.fn() };
|
||||
const { streamMock, mockEmbeddings } = vi.hoisted(() => {
|
||||
const embeddings = new Map([
|
||||
['note_001', [1.0, 0.0, 0.0]],
|
||||
['note_002', [0.0, 1.0, 0.0]],
|
||||
]);
|
||||
return { streamMock: vi.fn(), mockEmbeddings: embeddings };
|
||||
});
|
||||
|
||||
vi.mock('@anthropic-ai/sdk', () => {
|
||||
return {
|
||||
default: class MockAnthropic {
|
||||
constructor() {
|
||||
this.messages = { create: createMock };
|
||||
this.messages = { stream: streamMock };
|
||||
}
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock('../services/embedding.service.js', () => ({
|
||||
embedNotes: vi.fn().mockResolvedValue(mockEmbeddings),
|
||||
}));
|
||||
|
||||
import { clusterNotes } from '../services/clustering.service.js';
|
||||
|
||||
const MOCK_NOTES = [
|
||||
@@ -26,6 +34,40 @@ const MOCK_CLUSTERS = [
|
||||
{ label: 'Export Issues', noteIds: ['note_002'] },
|
||||
];
|
||||
|
||||
// Splits text into several text_delta events so the service is exercised
|
||||
// against a genuinely incremental stream rather than one whole payload.
|
||||
const textEvents = (text, pieces = 4) => {
|
||||
const size = Math.max(1, Math.ceil(text.length / pieces));
|
||||
const events = [];
|
||||
for (let i = 0; i < text.length; i += size) {
|
||||
events.push({
|
||||
type: 'content_block_delta',
|
||||
delta: { type: 'text_delta', text: text.slice(i, i + size) },
|
||||
});
|
||||
}
|
||||
return events;
|
||||
};
|
||||
|
||||
const mockStreamOf = (text) => {
|
||||
const events = [
|
||||
{ type: 'message_start' },
|
||||
...textEvents(text),
|
||||
{ type: 'message_stop' },
|
||||
];
|
||||
streamMock.mockImplementation(() => ({
|
||||
async *[Symbol.asyncIterator]() {
|
||||
for (const event of events) yield event;
|
||||
},
|
||||
}));
|
||||
};
|
||||
|
||||
const mockStreamThrowing = (err) => {
|
||||
streamMock.mockImplementation(() => ({
|
||||
async *[Symbol.asyncIterator]() {
|
||||
throw err;
|
||||
},
|
||||
}));
|
||||
};
|
||||
|
||||
describe('clusterNotes service', () => {
|
||||
|
||||
@@ -33,54 +75,88 @@ describe('clusterNotes service', () => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it('should call Anthropic messages.create with the correct model', async () => {
|
||||
createMock.mockResolvedValue({
|
||||
content: [{ text: JSON.stringify(MOCK_CLUSTERS) }],
|
||||
});
|
||||
it('should call Anthropic messages.stream with the correct model', async () => {
|
||||
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
|
||||
|
||||
await clusterNotes(MOCK_NOTES);
|
||||
|
||||
expect(createMock).toHaveBeenCalledOnce();
|
||||
const callArgs = createMock.mock.calls[0][0];
|
||||
expect(callArgs.model).toBe('claude-sonnet-4-20250514');
|
||||
expect(streamMock).toHaveBeenCalledOnce();
|
||||
const callArgs = streamMock.mock.calls[0][0];
|
||||
expect(callArgs.model).toBe('claude-sonnet-5');
|
||||
expect(callArgs.max_tokens).toBe(4096);
|
||||
});
|
||||
|
||||
it('should forward an abort signal to the LLM request', async () => {
|
||||
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
|
||||
const controller = new AbortController();
|
||||
|
||||
await clusterNotes(MOCK_NOTES, { signal: controller.signal });
|
||||
|
||||
expect(streamMock.mock.calls[0][1]).toEqual({ signal: controller.signal });
|
||||
});
|
||||
|
||||
it('should include all note texts in prompt sent to the LLM API', async () => {
|
||||
createMock.mockResolvedValue({
|
||||
content: [{ text: JSON.stringify(MOCK_CLUSTERS) }],
|
||||
});
|
||||
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
|
||||
|
||||
await clusterNotes(MOCK_NOTES);
|
||||
|
||||
const prompt = createMock.mock.calls[0][0].messages[0].content;
|
||||
const prompt = streamMock.mock.calls[0][0].messages[0].content;
|
||||
expect(prompt).toContain('note_001');
|
||||
expect(prompt).toContain('Login is broken');
|
||||
expect(prompt).toContain('note_002');
|
||||
expect(prompt).toContain('Export fails');
|
||||
});
|
||||
|
||||
it('should parse and return the clustered JSON from the API response', async () => {
|
||||
createMock.mockResolvedValue({
|
||||
content: [{ text: JSON.stringify(MOCK_CLUSTERS) }],
|
||||
});
|
||||
it('should return clusters and a cohesion score', async () => {
|
||||
mockStreamOf(JSON.stringify(MOCK_CLUSTERS));
|
||||
|
||||
const result = await clusterNotes(MOCK_NOTES);
|
||||
|
||||
expect(result).toEqual(MOCK_CLUSTERS);
|
||||
expect(result.clusters).toEqual(MOCK_CLUSTERS);
|
||||
expect(typeof result.score).toBe('number');
|
||||
expect(result.score).toBeGreaterThanOrEqual(-1);
|
||||
expect(result.score).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
it('should reassemble clusters split across many stream deltas', async () => {
|
||||
const json = JSON.stringify(MOCK_CLUSTERS);
|
||||
streamMock.mockImplementation(() => ({
|
||||
async *[Symbol.asyncIterator]() {
|
||||
for (const char of json) {
|
||||
yield { type: 'content_block_delta', delta: { type: 'text_delta', text: char } };
|
||||
}
|
||||
},
|
||||
}));
|
||||
|
||||
const result = await clusterNotes(MOCK_NOTES);
|
||||
|
||||
expect(result.clusters).toEqual(MOCK_CLUSTERS);
|
||||
});
|
||||
|
||||
it('should throw error when the API returns non-JSON', async () => {
|
||||
createMock.mockResolvedValue({
|
||||
content: [{ text: 'An unknown error occured when generting structured response.' }],
|
||||
});
|
||||
mockStreamOf('An unknown error occured when generting structured response.');
|
||||
|
||||
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow();
|
||||
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('non-JSON response');
|
||||
});
|
||||
|
||||
it('should throw error when the API response has no text content', async () => {
|
||||
mockStreamOf('');
|
||||
|
||||
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('no text content returned');
|
||||
});
|
||||
|
||||
it('should throw error when Anthropic API authentication fails', async () => {
|
||||
createMock.mockRejectedValue(new Error('401 Unauthorized'));
|
||||
mockStreamThrowing(new Error('401 Unauthorized'));
|
||||
|
||||
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('401 Unauthorized');
|
||||
});
|
||||
|
||||
it('should throw a validation error when a note is missing from clusters', async () => {
|
||||
const incompleteClusters = [
|
||||
{ label: 'Auth Issues', noteIds: ['note_001'] },
|
||||
];
|
||||
mockStreamOf(JSON.stringify(incompleteClusters));
|
||||
|
||||
await expect(clusterNotes(MOCK_NOTES)).rejects.toThrow('Cluster validation failed');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,17 +1,19 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
import request from 'supertest';
|
||||
import { Readable } from 'node:stream';
|
||||
import app from '../app.js';
|
||||
|
||||
vi.mock('../db/notes.dao.js', () => ({
|
||||
getAllNotes: vi.fn(),
|
||||
streamAllNotes: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/clustering.service.js', () => ({
|
||||
clusterNotes: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('fs/promises', () => ({
|
||||
readFile: vi.fn(),
|
||||
}));
|
||||
|
||||
import { getAllNotes, streamAllNotes } from '../db/notes.dao.js';
|
||||
import { clusterNotes } from '../services/clustering.service.js';
|
||||
import { readFile } from 'fs/promises';
|
||||
|
||||
const MOCK_NOTES = [
|
||||
{ id: 'note_001', text: 'Login flow feels confusing', x: 193, y: 191, author: 'user_5', color: 'yellow' },
|
||||
@@ -24,6 +26,8 @@ const MOCK_CLUSTERS = [
|
||||
{ label: 'Export Problems', noteIds: ['note_003'] },
|
||||
];
|
||||
|
||||
const rowStream = (rows) => Readable.from(rows, { objectMode: true });
|
||||
|
||||
describe('GET /v1/notes', () => {
|
||||
|
||||
beforeEach(() => {
|
||||
@@ -31,7 +35,7 @@ describe('GET /v1/notes', () => {
|
||||
});
|
||||
|
||||
it('should return 200 and an array of notes', async () => {
|
||||
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
|
||||
streamAllNotes.mockResolvedValue(rowStream(MOCK_NOTES));
|
||||
|
||||
const res = await request(app).get('/v1/notes');
|
||||
|
||||
@@ -40,8 +44,26 @@ describe('GET /v1/notes', () => {
|
||||
expect(Array.isArray(res.body)).toBe(true);
|
||||
});
|
||||
|
||||
it('should send JSON incrementally rather than buffering the row set', async () => {
|
||||
streamAllNotes.mockResolvedValue(rowStream(MOCK_NOTES));
|
||||
|
||||
const res = await request(app).get('/v1/notes');
|
||||
|
||||
expect(res.headers['content-type']).toMatch(/application\/json/);
|
||||
expect(res.headers['content-length']).toBeUndefined();
|
||||
});
|
||||
|
||||
it('should return an empty array when there are no notes', async () => {
|
||||
streamAllNotes.mockResolvedValue(rowStream([]));
|
||||
|
||||
const res = await request(app).get('/v1/notes');
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual([]);
|
||||
});
|
||||
|
||||
it('should return notes with expected properties', async () => {
|
||||
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
|
||||
streamAllNotes.mockResolvedValue(rowStream(MOCK_NOTES));
|
||||
|
||||
const res = await request(app).get('/v1/notes');
|
||||
const note = res.body[0];
|
||||
@@ -54,8 +76,8 @@ describe('GET /v1/notes', () => {
|
||||
expect(note).toHaveProperty('color');
|
||||
});
|
||||
|
||||
it('should return 500 when the data file cannot be read', async () => {
|
||||
readFile.mockRejectedValue(new Error('ENOENT: file not found'));
|
||||
it('should return 500 when the database query fails', async () => {
|
||||
streamAllNotes.mockRejectedValue(new Error('connection refused'));
|
||||
|
||||
const res = await request(app).get('/v1/notes');
|
||||
|
||||
@@ -64,13 +86,17 @@ describe('GET /v1/notes', () => {
|
||||
expect(res.body.error).toBe('Failed to load notes');
|
||||
});
|
||||
|
||||
it('should return 500 when data file is invalid JSON', async () => {
|
||||
readFile.mockResolvedValue('{ this is not valid json }');
|
||||
it('should abort the response when the row stream fails mid-flight', async () => {
|
||||
const failing = new Readable({
|
||||
objectMode: true,
|
||||
read() {
|
||||
this.push(MOCK_NOTES[0]);
|
||||
this.destroy(new Error('connection lost'));
|
||||
},
|
||||
});
|
||||
streamAllNotes.mockResolvedValue(failing);
|
||||
|
||||
const res = await request(app).get('/v1/notes');
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body).toHaveProperty('error');
|
||||
await expect(request(app).get('/v1/notes')).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -81,7 +107,7 @@ describe('POST /v1/notes/cluster', () => {
|
||||
});
|
||||
|
||||
it('should return 200 and clustered results', async () => {
|
||||
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
|
||||
getAllNotes.mockResolvedValue(MOCK_NOTES);
|
||||
clusterNotes.mockResolvedValue(MOCK_CLUSTERS);
|
||||
|
||||
const res = await request(app).post('/v1/notes/cluster');
|
||||
@@ -91,7 +117,7 @@ describe('POST /v1/notes/cluster', () => {
|
||||
});
|
||||
|
||||
it('should return clusters with the expected shape (label, noteIds)', async () => {
|
||||
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
|
||||
getAllNotes.mockResolvedValue(MOCK_NOTES);
|
||||
clusterNotes.mockResolvedValue(MOCK_CLUSTERS);
|
||||
|
||||
const res = await request(app).post('/v1/notes/cluster');
|
||||
@@ -104,18 +130,33 @@ describe('POST /v1/notes/cluster', () => {
|
||||
});
|
||||
|
||||
it('should pass the loaded notes to clusterNotes', async () => {
|
||||
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
|
||||
getAllNotes.mockResolvedValue(MOCK_NOTES);
|
||||
clusterNotes.mockResolvedValue(MOCK_CLUSTERS);
|
||||
|
||||
await request(app).post('/v1/notes/cluster');
|
||||
|
||||
expect(clusterNotes).toHaveBeenCalledOnce();
|
||||
expect(clusterNotes).toHaveBeenCalledWith(MOCK_NOTES);
|
||||
expect(clusterNotes).toHaveBeenCalledWith(
|
||||
MOCK_NOTES,
|
||||
expect.objectContaining({ signal: expect.any(AbortSignal) })
|
||||
);
|
||||
});
|
||||
|
||||
it('should pass a signal that is not aborted while the request is open', async () => {
|
||||
getAllNotes.mockResolvedValue(MOCK_NOTES);
|
||||
clusterNotes.mockImplementation(async (_notes, { signal }) => {
|
||||
expect(signal.aborted).toBe(false);
|
||||
return MOCK_CLUSTERS;
|
||||
});
|
||||
|
||||
const res = await request(app).post('/v1/notes/cluster');
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
});
|
||||
|
||||
it('should return 500 when clusterNotes (API call) fails', async () => {
|
||||
readFile.mockResolvedValue(JSON.stringify(MOCK_NOTES));
|
||||
clusterNotes.mockRejectedValue(new Error('Anthropic API error'));
|
||||
getAllNotes.mockResolvedValue(MOCK_NOTES);
|
||||
clusterNotes.mockRejectedValue(new Error('LLM API error'));
|
||||
|
||||
const res = await request(app).post('/v1/notes/cluster');
|
||||
|
||||
@@ -124,8 +165,8 @@ describe('POST /v1/notes/cluster', () => {
|
||||
expect(res.body.error).toMatch(/^Clustering failed/);
|
||||
});
|
||||
|
||||
it('should return 500 when the data file cannot be read', async () => {
|
||||
readFile.mockRejectedValue(new Error('ENOENT: file not found'));
|
||||
it('should return 500 when the database query fails', async () => {
|
||||
getAllNotes.mockRejectedValue(new Error('connection refused'));
|
||||
|
||||
const res = await request(app).post('/v1/notes/cluster');
|
||||
|
||||
|
||||
200
backend/tests/notes.dao.test.js
Normal file
200
backend/tests/notes.dao.test.js
Normal file
@@ -0,0 +1,200 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
import { Readable } from 'node:stream';
|
||||
|
||||
const { mockQuery, mockConnect } = vi.hoisted(() => ({
|
||||
mockQuery: vi.fn(),
|
||||
mockConnect: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../db/index.js', () => ({
|
||||
query: mockQuery,
|
||||
getPool: () => ({ connect: mockConnect }),
|
||||
}));
|
||||
|
||||
import {
|
||||
getAllNotes,
|
||||
streamAllNotes,
|
||||
getNoteById,
|
||||
createNote,
|
||||
createNotes,
|
||||
} from '../db/notes.dao.js';
|
||||
|
||||
const MOCK_ROWS = [
|
||||
{ id: 'note_001', text: 'Login flow feels confusing', x: 193, y: 191, author: 'user_5', color: 'yellow' },
|
||||
{ id: 'note_002', text: 'Login flow is broken on mobile', x: 214, y: 281, author: 'user_9', color: 'yellow' },
|
||||
];
|
||||
|
||||
describe('notes.dao', () => {
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
describe('getAllNotes', () => {
|
||||
it('should return all notes ordered by id', async () => {
|
||||
mockQuery.mockResolvedValue({ rows: MOCK_ROWS });
|
||||
|
||||
const result = await getAllNotes();
|
||||
|
||||
expect(result).toEqual(MOCK_ROWS);
|
||||
expect(mockQuery).toHaveBeenCalledWith(
|
||||
'SELECT id, text, x, y, author, color FROM notes ORDER BY id'
|
||||
);
|
||||
});
|
||||
|
||||
it('should return an empty array when no notes exist', async () => {
|
||||
mockQuery.mockResolvedValue({ rows: [] });
|
||||
|
||||
const result = await getAllNotes();
|
||||
|
||||
expect(result).toEqual([]);
|
||||
});
|
||||
|
||||
it('should propagate database errors', async () => {
|
||||
mockQuery.mockRejectedValue(new Error('connection refused'));
|
||||
|
||||
await expect(getAllNotes()).rejects.toThrow('connection refused');
|
||||
});
|
||||
});
|
||||
|
||||
describe('streamAllNotes', () => {
|
||||
const mockClient = (rows) => {
|
||||
const release = vi.fn();
|
||||
const client = {
|
||||
release,
|
||||
query: vi.fn(() => Readable.from(rows, { objectMode: true })),
|
||||
};
|
||||
mockConnect.mockResolvedValue(client);
|
||||
return { client, release };
|
||||
};
|
||||
|
||||
it('should stream rows without buffering them into an array', async () => {
|
||||
mockClient(MOCK_ROWS);
|
||||
|
||||
const stream = await streamAllNotes();
|
||||
const received = [];
|
||||
for await (const row of stream) received.push(row);
|
||||
|
||||
expect(received).toEqual(MOCK_ROWS);
|
||||
});
|
||||
|
||||
it('should release the pooled client once the stream ends', async () => {
|
||||
const { release } = mockClient(MOCK_ROWS);
|
||||
|
||||
const stream = await streamAllNotes();
|
||||
for await (const _row of stream) { /* drain */ }
|
||||
|
||||
expect(release).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it('should release the pooled client when a consumer destroys the stream early', async () => {
|
||||
const { release } = mockClient(MOCK_ROWS);
|
||||
|
||||
const stream = await streamAllNotes();
|
||||
stream.destroy();
|
||||
await new Promise((resolve) => stream.once('close', resolve));
|
||||
|
||||
expect(release).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it('should release the pooled client when starting the query throws', async () => {
|
||||
const release = vi.fn();
|
||||
mockConnect.mockResolvedValue({
|
||||
release,
|
||||
query: vi.fn(() => { throw new Error('cursor failed'); }),
|
||||
});
|
||||
|
||||
await expect(streamAllNotes()).rejects.toThrow('cursor failed');
|
||||
expect(release).toHaveBeenCalledOnce();
|
||||
});
|
||||
});
|
||||
|
||||
describe('getNoteById', () => {
|
||||
it('should return a single note when found', async () => {
|
||||
mockQuery.mockResolvedValue({ rows: [MOCK_ROWS[0]] });
|
||||
|
||||
const result = await getNoteById('note_001');
|
||||
|
||||
expect(result).toEqual(MOCK_ROWS[0]);
|
||||
expect(mockQuery).toHaveBeenCalledWith(
|
||||
'SELECT id, text, x, y, author, color FROM notes WHERE id = $1',
|
||||
['note_001']
|
||||
);
|
||||
});
|
||||
|
||||
it('should return null when note is not found', async () => {
|
||||
mockQuery.mockResolvedValue({ rows: [] });
|
||||
|
||||
const result = await getNoteById('note_999');
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('createNote', () => {
|
||||
it('should insert a note and return it', async () => {
|
||||
const input = { id: 'note_003', text: 'Export fails', x: 100, y: 200, author: 'user_1', color: 'blue' };
|
||||
mockQuery.mockResolvedValue({ rows: [input] });
|
||||
|
||||
const result = await createNote(input);
|
||||
|
||||
expect(result).toEqual(input);
|
||||
expect(mockQuery).toHaveBeenCalledOnce();
|
||||
const [sql, params] = mockQuery.mock.calls[0];
|
||||
expect(sql).toContain('INSERT INTO notes');
|
||||
expect(params).toEqual(['note_003', 'Export fails', 100, 200, 'user_1', 'blue']);
|
||||
});
|
||||
|
||||
it('should use defaults for missing x, y, and color', async () => {
|
||||
const input = { id: 'note_004', text: 'Needs fixing', author: 'user_2' };
|
||||
mockQuery.mockResolvedValue({ rows: [{ ...input, x: 0, y: 0, color: 'yellow' }] });
|
||||
|
||||
await createNote(input);
|
||||
|
||||
const params = mockQuery.mock.calls[0][1];
|
||||
expect(params[2]).toBe(0);
|
||||
expect(params[3]).toBe(0);
|
||||
expect(params[5]).toBe('yellow');
|
||||
});
|
||||
});
|
||||
|
||||
describe('createNotes', () => {
|
||||
it('should batch-insert multiple notes and return them', async () => {
|
||||
mockQuery.mockResolvedValue({ rows: MOCK_ROWS });
|
||||
|
||||
const result = await createNotes(MOCK_ROWS);
|
||||
|
||||
expect(result).toEqual(MOCK_ROWS);
|
||||
expect(mockQuery).toHaveBeenCalledOnce();
|
||||
const [sql, params] = mockQuery.mock.calls[0];
|
||||
expect(sql).toContain('INSERT INTO notes');
|
||||
expect(params).toHaveLength(12);
|
||||
});
|
||||
|
||||
it('should return an empty array without querying when given no notes', async () => {
|
||||
const result = await createNotes([]);
|
||||
|
||||
expect(result).toEqual([]);
|
||||
expect(mockQuery).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('should split large inputs into multiple statements under the bind-parameter limit', async () => {
|
||||
const many = Array.from({ length: 2500 }, (_, i) => ({
|
||||
id: `note_${i}`,
|
||||
text: `text ${i}`,
|
||||
author: 'user_1',
|
||||
}));
|
||||
mockQuery.mockImplementation(async (_sql, params) => ({
|
||||
rows: new Array(params.length / 6).fill(null).map((_, i) => ({ i })),
|
||||
}));
|
||||
|
||||
const result = await createNotes(many);
|
||||
|
||||
expect(mockQuery).toHaveBeenCalledTimes(3);
|
||||
expect(result).toHaveLength(2500);
|
||||
for (const [, params] of mockQuery.mock.calls) {
|
||||
expect(params.length).toBeLessThan(65535);
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
82
backend/tests/streams.test.js
Normal file
82
backend/tests/streams.test.js
Normal file
@@ -0,0 +1,82 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { Readable } from 'node:stream';
|
||||
import { pipeline } from 'node:stream/promises';
|
||||
import { batch, jsonArray } from '../lib/streams.js';
|
||||
|
||||
const collect = async (source, transform) => {
|
||||
const out = [];
|
||||
await pipeline(source, transform, async (results) => {
|
||||
for await (const item of results) out.push(item);
|
||||
});
|
||||
return out;
|
||||
};
|
||||
|
||||
describe('batch', () => {
|
||||
it('should group items into fixed-size arrays', async () => {
|
||||
const source = Readable.from([1, 2, 3, 4], { objectMode: true });
|
||||
|
||||
const result = await collect(source, batch(2));
|
||||
|
||||
expect(result).toEqual([[1, 2], [3, 4]]);
|
||||
});
|
||||
|
||||
it('should flush a partial trailing batch', async () => {
|
||||
const source = Readable.from([1, 2, 3, 4, 5], { objectMode: true });
|
||||
|
||||
const result = await collect(source, batch(2));
|
||||
|
||||
expect(result).toEqual([[1, 2], [3, 4], [5]]);
|
||||
});
|
||||
|
||||
it('should emit nothing for an empty source', async () => {
|
||||
const source = Readable.from([], { objectMode: true });
|
||||
|
||||
const result = await collect(source, batch(3));
|
||||
|
||||
expect(result).toEqual([]);
|
||||
});
|
||||
|
||||
it('should reject a non-positive size', () => {
|
||||
expect(() => batch(0)).toThrow(TypeError);
|
||||
expect(() => batch(1.5)).toThrow(TypeError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('jsonArray', () => {
|
||||
const serialize = async (items) => {
|
||||
const chunks = await collect(
|
||||
Readable.from(items, { objectMode: true }),
|
||||
jsonArray()
|
||||
);
|
||||
return chunks.map(String).join('');
|
||||
};
|
||||
|
||||
it('should serialize objects into a JSON array', async () => {
|
||||
const items = [{ id: 'a' }, { id: 'b' }];
|
||||
|
||||
const output = await serialize(items);
|
||||
|
||||
expect(output).toBe('[{"id":"a"},{"id":"b"}]');
|
||||
expect(JSON.parse(output)).toEqual(items);
|
||||
});
|
||||
|
||||
it('should emit an empty array when the source yields nothing', async () => {
|
||||
const output = await serialize([]);
|
||||
|
||||
expect(output).toBe('[]');
|
||||
expect(JSON.parse(output)).toEqual([]);
|
||||
});
|
||||
|
||||
it('should emit a valid single-element array', async () => {
|
||||
const output = await serialize([{ id: 'only' }]);
|
||||
|
||||
expect(JSON.parse(output)).toEqual([{ id: 'only' }]);
|
||||
});
|
||||
|
||||
it('should propagate serialization errors', async () => {
|
||||
const circular = {};
|
||||
circular.self = circular;
|
||||
|
||||
await expect(serialize([circular])).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
2
frontend/.env.development
Normal file
2
frontend/.env.development
Normal file
@@ -0,0 +1,2 @@
|
||||
VITE_APP_URL=localhost:3000
|
||||
VITE_API_BASE=http://localhost:3001
|
||||
2
frontend/.env.production
Normal file
2
frontend/.env.production
Normal file
@@ -0,0 +1,2 @@
|
||||
VITE_APP_URL=https://example.com
|
||||
VITE_API_BASE=https://www.example.com:4000
|
||||
@@ -1,7 +1,7 @@
|
||||
import { useQuery, useMutation } from '@tanstack/react-query';
|
||||
import type { Sticky, Cluster } from '../types/types';
|
||||
import type { Sticky, ClusterResponse } from '../types/types';
|
||||
|
||||
const API_BASE = 'http://localhost:3001/v1/notes';
|
||||
const API_BASE = `${import.meta.env.VITE_API_BASE}/v1/notes`;
|
||||
|
||||
const fetchStickies = async (): Promise<Sticky[]> => {
|
||||
const response = await fetch(API_BASE);
|
||||
@@ -11,7 +11,7 @@ const fetchStickies = async (): Promise<Sticky[]> => {
|
||||
return response.json();
|
||||
};
|
||||
|
||||
const fetchClusters = async (): Promise<Cluster[]> => {
|
||||
const fetchClusters = async (): Promise<ClusterResponse> => {
|
||||
const response = await fetch(`${API_BASE}/cluster`, {
|
||||
method: 'POST',
|
||||
});
|
||||
@@ -29,7 +29,7 @@ export const useGetStickies = () => {
|
||||
};
|
||||
|
||||
export const useClusterStickies = () => {
|
||||
return useMutation<Cluster[]>({
|
||||
return useMutation<ClusterResponse>({
|
||||
mutationFn: fetchClusters,
|
||||
});
|
||||
};
|
||||
@@ -8,9 +8,9 @@ type ButtonProps = {
|
||||
|
||||
const Button = ({ onClick, isLoading, label }: ButtonProps) => {
|
||||
return (
|
||||
<button className="primaryButton" onClick={onClick} disabled={isLoading}>
|
||||
{isLoading ? 'Working...' : label}
|
||||
</button>
|
||||
<button className="primaryButton" onClick={onClick} disabled={isLoading}>
|
||||
{isLoading ? 'Working...' : label}
|
||||
</button>
|
||||
);
|
||||
};
|
||||
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
const Navbar = () => {
|
||||
return (
|
||||
<div className="main-head-box">
|
||||
<div className="main-head-subbox-left">
|
||||
<h1 className="main-head">kongruity</h1>
|
||||
</div>
|
||||
<div className="main-head-subbox-right">
|
||||
<span className="material-symbols-outlined">recenter</span>
|
||||
</div>
|
||||
return (
|
||||
<div className="main-head-box">
|
||||
<div className="main-head-subbox-left">
|
||||
<h1 className="main-head">kongruity</h1>
|
||||
</div>
|
||||
)
|
||||
<div className="main-head-subbox-right">
|
||||
<span className="material-symbols-outlined">recenter</span>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export default Navbar;
|
||||
@@ -1,56 +1,141 @@
|
||||
import { useState, useEffect, useRef } from 'react';
|
||||
import type { DragEvent } from 'react';
|
||||
import { useGetStickies, useClusterStickies } from '../api/api';
|
||||
import type { Sticky as StickyType } from '../types/types';
|
||||
import type { Sticky as StickyType, RankedCluster } from '../types/types';
|
||||
import Sticky from './sticky';
|
||||
import Button from './button';
|
||||
import '../styles/stickies.css';
|
||||
|
||||
// In practice, silhouette on cosine distance between text embeddings occupies roughly
|
||||
// [-0.05, 0.10], not strict theoretical [-1, 1]: near-orthogonal vectors put both the within- and
|
||||
// nearest-cluster distances close to 0.8, and the coefficient divides their gap
|
||||
// by the larger. These bands are calibrated to that range for voyage-3. see README, Reading the cohesion score
|
||||
const scoreLabel = (score: number): string => {
|
||||
if (score >= 0.07) return 'Strong';
|
||||
if (score >= 0.04) return 'Moderate';
|
||||
if (score >= 0.01) return 'Weak';
|
||||
return 'Poor';
|
||||
};
|
||||
|
||||
const Stickies = () => {
|
||||
const { data: stickies, isLoading, error } = useGetStickies();
|
||||
const { mutate: cluster, data: clusters, isPending } = useClusterStickies();
|
||||
const { data: stickies, isLoading, error } = useGetStickies();
|
||||
const { mutate: cluster, data: clusterResponse, isPending } = useClusterStickies();
|
||||
|
||||
if (isLoading) return <div>Loading...</div>;
|
||||
if (error) return <div>Error: {error.message}</div>;
|
||||
const [rankedClusters, setRankedClusters] = useState<RankedCluster[]>([]);
|
||||
const dragIndex = useRef<number | null>(null);
|
||||
const [dragOverIndex, setDragOverIndex] = useState<number | null>(null);
|
||||
|
||||
const handleCluster = () => {
|
||||
cluster();
|
||||
};
|
||||
useEffect(() => {
|
||||
if (clusterResponse?.clusters) {
|
||||
setRankedClusters(
|
||||
clusterResponse.clusters.map((c, i) => ({ ...c, rank: i + 1 }))
|
||||
);
|
||||
}
|
||||
}, [clusterResponse]);
|
||||
|
||||
const buildStickyMap = (): Map<string, StickyType> => {
|
||||
const map = new Map<string, StickyType>();
|
||||
stickies?.forEach((s) => map.set(s.id, s));
|
||||
return map;
|
||||
};
|
||||
if (isLoading) return <div>Loading...</div>;
|
||||
if (error) return <div>Error: {error.message}</div>;
|
||||
|
||||
const stickyMap = buildStickyMap();
|
||||
const score = clusterResponse?.score;
|
||||
|
||||
const renderStickies = (items: StickyType[]) =>
|
||||
items.map((sticky) => <Sticky key={sticky.id} sticky={sticky} />);
|
||||
const handleCluster = () => {
|
||||
cluster();
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="stickies-container">
|
||||
<Button onClick={handleCluster} isLoading={isPending} label="Group Stickies By Topic" />
|
||||
{clusters ? (
|
||||
<div className="clusters-container">
|
||||
{clusters.map((group) => (
|
||||
<div key={group.label} className="cluster-group">
|
||||
<h3 className="cluster-label">{group.label}</h3>
|
||||
<div className="stickies-grid">
|
||||
{renderStickies(
|
||||
group.noteIds
|
||||
.map((id) => stickyMap.get(id))
|
||||
.filter((s): s is StickyType => !!s)
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
const buildStickyMap = (): Map<string, StickyType> => {
|
||||
const map = new Map<string, StickyType>();
|
||||
stickies?.forEach((s) => map.set(s.id, s));
|
||||
return map;
|
||||
};
|
||||
|
||||
const stickyMap = buildStickyMap();
|
||||
|
||||
const renderStickies = (items: StickyType[]) =>
|
||||
items?.map((sticky) => <Sticky key={sticky.id} sticky={sticky} />);
|
||||
|
||||
const handleDragStart = (index: number) => {
|
||||
dragIndex.current = index;
|
||||
};
|
||||
|
||||
const handleDragOver = (e: DragEvent, index: number) => {
|
||||
e.preventDefault();
|
||||
setDragOverIndex(index);
|
||||
};
|
||||
|
||||
const handleDragLeave = () => {
|
||||
setDragOverIndex(null);
|
||||
};
|
||||
|
||||
const handleDrop = (targetIndex: number) => {
|
||||
const sourceIndex = dragIndex.current;
|
||||
if (sourceIndex === null || sourceIndex === targetIndex) {
|
||||
dragIndex.current = null;
|
||||
setDragOverIndex(null);
|
||||
return;
|
||||
}
|
||||
|
||||
const reordered = [...rankedClusters];
|
||||
const [moved] = reordered.splice(sourceIndex, 1);
|
||||
reordered.splice(targetIndex, 0, moved);
|
||||
|
||||
setRankedClusters(reordered.map((c, i) => ({ ...c, rank: i + 1 })));
|
||||
dragIndex.current = null;
|
||||
setDragOverIndex(null);
|
||||
};
|
||||
|
||||
const handleDragEnd = () => {
|
||||
dragIndex.current = null;
|
||||
setDragOverIndex(null);
|
||||
};
|
||||
console.log(score?.toFixed(2))
|
||||
return (
|
||||
<div className="stickies-container">
|
||||
<Button onClick={handleCluster} isLoading={isPending} label="Group Stickies By Topic" />
|
||||
{rankedClusters.length > 0 ? (
|
||||
<div className="clusters-container">
|
||||
{score != null && (
|
||||
<div className="cohesion-score">
|
||||
Cluster cohesion: <strong>{scoreLabel(score)}</strong>
|
||||
</div>
|
||||
)}
|
||||
{rankedClusters.map((group, index) => (
|
||||
<div
|
||||
key={group.label}
|
||||
className={`cluster-group cluster-draggable${dragOverIndex === index ? ' cluster-drag-over' : ''}`}
|
||||
draggable
|
||||
onDragStart={() => handleDragStart(index)}
|
||||
onDragOver={(e) => handleDragOver(e, index)}
|
||||
onDragLeave={handleDragLeave}
|
||||
onDrop={() => handleDrop(index)}
|
||||
onDragEnd={handleDragEnd}
|
||||
>
|
||||
<div className="cluster-header">
|
||||
<span className="cluster-rank" aria-label={`Priority ${group.rank}`}>
|
||||
{group.rank}
|
||||
</span>
|
||||
{group.rank === 1 && (
|
||||
<span className="cluster-reorder-hint">Drag and drop to reorganize cluster priority</span>
|
||||
)}
|
||||
<h3 className="cluster-label">{group.label}</h3>
|
||||
<span className="cluster-drag-handle" aria-hidden="true">⠿</span>
|
||||
</div>
|
||||
<div className="stickies-grid">
|
||||
{renderStickies(
|
||||
group?.noteIds
|
||||
.map((id) => stickyMap?.get(id))
|
||||
.filter((s): s is StickyType => !!s)
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
) : (
|
||||
<div className="stickies-grid">
|
||||
{renderStickies(stickies ?? [])}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
) : (
|
||||
<div className="stickies-grid">
|
||||
{renderStickies(stickies ?? [])}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
);
|
||||
};
|
||||
|
||||
export default Stickies;
|
||||
|
||||
@@ -14,5 +14,5 @@ createRoot(document.getElementById('root')!).render(
|
||||
<App />
|
||||
</BrowserRouter>
|
||||
</QueryClientProvider>
|
||||
</StrictMode>,
|
||||
</StrictMode>
|
||||
)
|
||||
|
||||
@@ -3,16 +3,12 @@ import Navbar from '../components/navbar'
|
||||
import '../styles/home.css'
|
||||
|
||||
const Home = () => {
|
||||
return (
|
||||
<div>
|
||||
<div>
|
||||
<Navbar />
|
||||
</div>
|
||||
<div>
|
||||
<Stickies />
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
return (
|
||||
<div>
|
||||
<Navbar />
|
||||
<Stickies />
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
export default Home;
|
||||
@@ -1,41 +1,73 @@
|
||||
.main-head-box {
|
||||
border-radius: 8px;
|
||||
border: 1px solid #6dd6f4;
|
||||
background-color: rgb(92, 0 91);
|
||||
display: flex;
|
||||
}
|
||||
border-radius: 8px;
|
||||
border: 1px solid #6dd6f4;
|
||||
background-color: rgb(92, 0, 91);
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.main-head-subbox-right {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
margin-right: 34px;
|
||||
width: 50%;
|
||||
}
|
||||
|
||||
.main-head-subbox-left {
|
||||
display: flex;
|
||||
justify-content: flex-start;
|
||||
margin-left: 34px;
|
||||
width: 50%;
|
||||
}
|
||||
|
||||
.main-head {
|
||||
font-size: 5rem;
|
||||
color: #6dd6f4;
|
||||
margin: 6px 0px 24px 22px;
|
||||
font-family: "Sulphur Point", sans-serif;
|
||||
font-weight: 500;
|
||||
letter-spacing: 4px;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.material-symbols-outlined {
|
||||
margin: 13px 74px 0px 0px;
|
||||
font-size: 82px;
|
||||
color: #6dd6f4;
|
||||
font-variation-settings:
|
||||
'FILL' 0,
|
||||
'wght' 300,
|
||||
'GRAD' 0,
|
||||
'opsz' 24;
|
||||
}
|
||||
|
||||
@media screen and (max-width: 478px) {
|
||||
.main-head-subbox-right {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
margin-right: 34px;
|
||||
width: 50%;
|
||||
display: none;
|
||||
}
|
||||
|
||||
.main-head-subbox-left {
|
||||
display: flex;
|
||||
justify-content: flex-start;
|
||||
margin-left: 34px;
|
||||
width: 50%;
|
||||
justify-content: center;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.main-head {
|
||||
font-size: 5rem;
|
||||
font-size: 4rem;
|
||||
color: #6dd6f4;
|
||||
margin: 6px 0px 24px 22px;
|
||||
font-family: "Sulphur Point", sans-serif;
|
||||
font-weight: 500;
|
||||
letter-spacing: 4px;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
font-weight: 400;
|
||||
letter-spacing: 2px;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.material-symbols-outlined {
|
||||
margin: 13px 74px 0px 0px;
|
||||
font-size: 82px;
|
||||
color: #6dd6f4;
|
||||
font-variation-settings:
|
||||
'FILL' 0,
|
||||
'wght' 300,
|
||||
'GRAD' 0,
|
||||
'opsz' 24
|
||||
display: none !important;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,30 +1,107 @@
|
||||
.stickies-grid {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 16px;
|
||||
justify-content: center;
|
||||
padding: 24px;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 16px;
|
||||
justify-content: center;
|
||||
margin-top: 18px;
|
||||
padding: 24px;
|
||||
}
|
||||
|
||||
.stickies-container {
|
||||
margin: 36px 0px 36px 0px;
|
||||
margin: 36px 0px 36px 0px;
|
||||
}
|
||||
|
||||
.clusters-container {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 32px;
|
||||
padding: 24px 0;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 32px;
|
||||
padding: 24px 0;
|
||||
}
|
||||
|
||||
.cluster-group {
|
||||
border: 1px solid #6dd6f4;
|
||||
border-radius: 8px;
|
||||
padding: 16px;
|
||||
border: 1px solid #6dd6f4;
|
||||
border-radius: 8px;
|
||||
padding: 16px;
|
||||
}
|
||||
|
||||
.cluster-draggable {
|
||||
cursor: grab;
|
||||
transition: box-shadow 0.2s ease, border-color 0.2s ease, transform 0.15s ease;
|
||||
}
|
||||
|
||||
.cluster-draggable:active {
|
||||
cursor: grabbing;
|
||||
}
|
||||
|
||||
.cluster-drag-over {
|
||||
border-color: #ffb7ce;
|
||||
box-shadow: 0 0 12px rgba(255, 183, 206, 0.4);
|
||||
transform: scale(1.01);
|
||||
}
|
||||
|
||||
.cluster-header {
|
||||
position: relative;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
margin-bottom: 16px;
|
||||
min-height: 32px;
|
||||
}
|
||||
|
||||
.cluster-rank {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
width: 32px;
|
||||
height: 32px;
|
||||
border-radius: 50%;
|
||||
background: #6dd6f4;
|
||||
color: #1a1a2e;
|
||||
font-weight: 700;
|
||||
font-size: 0.95em;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.cluster-reorder-hint {
|
||||
font-size: 0.8em;
|
||||
color: #9ca3af;
|
||||
font-style: italic;
|
||||
white-space: nowrap;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.cluster-label {
|
||||
margin: 0 0 16px 0;
|
||||
font-size: 1.2em;
|
||||
font-weight: 600;
|
||||
position: absolute;
|
||||
left: 50%;
|
||||
transform: translateX(-50%);
|
||||
max-width: 50%;
|
||||
margin: 0;
|
||||
font-size: 1.2em;
|
||||
font-weight: 600;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
.cluster-drag-handle {
|
||||
margin-left: auto;
|
||||
font-size: 1.4em;
|
||||
color: #6dd6f4;
|
||||
opacity: 0.4;
|
||||
user-select: none;
|
||||
transition: opacity 0.2s ease;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.cluster-draggable:hover .cluster-drag-handle {
|
||||
opacity: 0.8;
|
||||
}
|
||||
|
||||
.cohesion-score {
|
||||
text-align: center;
|
||||
font-size: 0.95em;
|
||||
color: #e0e0e0;
|
||||
padding: 8px 16px;
|
||||
background: rgba(109, 214, 244, 0.1);
|
||||
border-radius: 6px;
|
||||
width: fit-content;
|
||||
margin: 0 auto;
|
||||
}
|
||||
@@ -9,10 +9,13 @@ const MOCK_STICKIES = [
|
||||
{ id: 'note_002', text: 'Export takes too long', x: 798, y: 211, author: 'user_2', color: 'green' },
|
||||
];
|
||||
|
||||
const MOCK_CLUSTERS = [
|
||||
{ label: 'Auth Issues', noteIds: ['note_001'] },
|
||||
{ label: 'Export Issues', noteIds: ['note_002'] },
|
||||
];
|
||||
const MOCK_CLUSTER_RESPONSE = {
|
||||
clusters: [
|
||||
{ label: 'Auth Issues', noteIds: ['note_001'] },
|
||||
{ label: 'Export Issues', noteIds: ['note_002'] },
|
||||
],
|
||||
score: 0.09,
|
||||
};
|
||||
|
||||
let fetchMock: ReturnType<typeof vi.fn>;
|
||||
|
||||
@@ -70,10 +73,9 @@ describe('Stickies', () => {
|
||||
});
|
||||
|
||||
it('should show cluster labels after clustering succeeds', async () => {
|
||||
// First call = GET notes, second call = POST cluster
|
||||
fetchMock
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTERS));
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
|
||||
|
||||
const { Wrapper } = createTestWrapper();
|
||||
render(<Stickies />, { wrapper: Wrapper });
|
||||
@@ -90,7 +92,7 @@ describe('Stickies', () => {
|
||||
it('should still render sticky note text inside clusters', async () => {
|
||||
fetchMock
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTERS));
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
|
||||
|
||||
const { Wrapper } = createTestWrapper();
|
||||
render(<Stickies />, { wrapper: Wrapper });
|
||||
@@ -101,4 +103,41 @@ describe('Stickies', () => {
|
||||
expect(await screen.findByText('Login flow feels confusing')).toBeInTheDocument();
|
||||
expect(screen.getByText('Export takes too long')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('should display rank badges on clustered groups', async () => {
|
||||
fetchMock
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
|
||||
|
||||
const { Wrapper } = createTestWrapper();
|
||||
render(<Stickies />, { wrapper: Wrapper });
|
||||
|
||||
const btn = await screen.findByRole('button', { name: 'Group Stickies By Topic' });
|
||||
await userEvent.click(btn);
|
||||
|
||||
await waitFor(() => {
|
||||
expect(screen.getByLabelText('Priority 1')).toBeInTheDocument();
|
||||
expect(screen.getByLabelText('Priority 2')).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
it('should make cluster groups draggable', async () => {
|
||||
fetchMock
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_STICKIES))
|
||||
.mockReturnValueOnce(mockFetchOk(MOCK_CLUSTER_RESPONSE));
|
||||
|
||||
const { Wrapper } = createTestWrapper();
|
||||
render(<Stickies />, { wrapper: Wrapper });
|
||||
|
||||
const btn = await screen.findByRole('button', { name: 'Group Stickies By Topic' });
|
||||
await userEvent.click(btn);
|
||||
|
||||
await waitFor(() => {
|
||||
const groups = document.querySelectorAll('.cluster-draggable');
|
||||
groups.forEach((group) => {
|
||||
expect(group).toHaveAttribute('draggable', 'true');
|
||||
});
|
||||
expect(groups.length).toBe(2);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -11,3 +11,12 @@ export type Cluster = {
|
||||
label: string;
|
||||
noteIds: string[];
|
||||
};
|
||||
|
||||
export type ClusterResponse = {
|
||||
clusters: Cluster[];
|
||||
score: number;
|
||||
};
|
||||
|
||||
export type RankedCluster = Cluster & {
|
||||
rank: number;
|
||||
};
|
||||
21
package-lock.json
generated
Normal file
21
package-lock.json
generated
Normal file
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"name": "kongruity",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"dependencies": {
|
||||
"split2": "^4.2.0"
|
||||
}
|
||||
},
|
||||
"node_modules/split2": {
|
||||
"version": "4.2.0",
|
||||
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
|
||||
"integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">= 10.x"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
5
package.json
Normal file
5
package.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"dependencies": {
|
||||
"split2": "^4.2.0"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user