9 Commits

Author SHA1 Message Date
58e85c00f8 Update ReadMe.md 2026-07-28 07:32:07 +00:00
S Jannette
6edee1df6e Merge pull request #2 from kjannette/update-README
update README.md
2026-05-09 18:15:39 -04:00
KS Jannette
8d98d19e7e update README.md 2026-05-09 18:15:00 -04:00
S Jannette
01a08452b7 Update README for consistency in query pipeline steps 2026-05-09 17:55:16 -04:00
S Jannette
998b64f82d Update README to remove yt-dlp python installation instructions
Removed installation instructions for yt-dlp from prerequisites.
2026-05-09 17:29:45 -04:00
S Jannette
7f00e6d839 Improve client description in readme.md
Updated repository name in clone instructions and clarified client description.
2026-05-09 17:28:57 -04:00
S Jannette
0e9061c792 Update README.md for Voyage AI model 2026-05-09 17:26:23 -04:00
S Jannette
e6850fc6b8 Fix typo in README description 2026-05-09 17:24:46 -04:00
S Jannette
f570a1e711 Merge pull request #1 from kjannette/ts-refactor-two
addiitonal ts implementation
2026-05-09 17:18:32 -04:00

View File

@@ -1,6 +1,6 @@
# Citation Sentinel
A source-grounded research assistant that employs cosine similarity scoring for LLM generated query responses to provide a "groundedness" score. This is a metric in Retrieval-Augmented Generation (RAG) systems that quantifies how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated or pulled from the model's pre-training data.
A source-grounded research assistant that employs cosine similarity scoring for LLM-generated query responses to provide a "groundedness" score. This is a metric in Retrieval-Augmented Generation (RAG) systems that quantifies how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated or pulled from the model's pre-training data.
Users upload source documents, or provide links to online sources including audio/video (i.e. links to youtube videos). Users may then ask questions and receive answers (with inline citations) verifiably grounded in the provided information sources.
@@ -8,18 +8,18 @@ Built with a React/Vite frontend and a Typescript/Node/Express backend, using An
## Query pipeline
Source Ingestion -> Parsing -> Chunking -> Embedding (using Voayge AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> evaluation of user query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model)
Source Ingestion -> Parsing -> Chunking -> Embedding (using Voyage AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> Evaluation of User Query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model)
## Prerequisites
- **Node.js** (v18+)
- **yt-dlp** -- required for YouTube video source support (`brew install yt-dlp` or `pip install yt-dlp`)
- **yt-dlp** -- required for YouTube video source support (`brew install yt-dlp`)
## Getting Started
```bash
# clone and install
git clone <repo-url> && cd notebooklm_clone
git clone <repo-url> && cd citation_sentinel
cd server && npm install && cd ..
cd client && npm install && cd ..
@@ -42,15 +42,34 @@ This application is a RAG (Retrieval-Augmented Generation) system that allows us
When a user submits a query, the system enforces groundedness through a multi-layered strategy:
1. Retrieval constraint — The query is embedded (also via voyage-3) and compared against stored chunk vectors using cosine similarity, returning the top 20 candidates. Only user-supplied source material is searched; the system has no web search capability.
1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against
stored chunk vectors using cosine similarity, returning the top 20 candidates.
3. Reranking for precision — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder, which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder,
which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
5. Prompt-level constraint — The top 5 chunks are passed to the Primary LLM (Claude claude-opus-4-6) with an explicit system instruction: "Answer the user's question using ONLY the source documents provided below." The LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are insufficient.
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6).
The LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are
insufficient.
7. Schema enforcement — The LLM's response is constrained to a JSON schema requiring structured fields (answer, citedSourceIndices, followUpQuestions), and any cited source indices that don't correspond to real source groups are programmatically stripped out.
4. **Schema enforcement** — The LLM's response is constrained to a JSON schema requiring structured
fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not
correspond to real source groups are programmatically stripped out.
9. Post-generation groundedness scoring — After the answer is generated, it is split into individual sentences, each sentence is embedded via voyage-3, and each sentence embedding is compared (cosine similarity) against the vectors of the cited chunks. The raw similarities are calibrated to a 01 scale and averaged, producing a single groundedness score that is surfaced to the user as a visual indicator (green/gold/red).
5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage
AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors
of the cited chunks. The similarity is calibrated to a 01 scale and averaged, producing a single
groundedness score that is surfaced to the user with a visual indicator (green/yellow/red).
## Similarity Metrics for Semantic Understanding
Cosine similarity measures how closely two vectors (representing data like words, images, or preferences) are aligned in a multi-dimensional space by calculating the cosine of the angle between them.
Archetypical cosine scores range from -1 to 1.
1: Vectors point in the exact same direction (highly similar).
0: Vectors are at a 90-degree angle (orthogonal/unrelated).
-1: Vectors point in opposite directions.
## Design
@@ -58,7 +77,7 @@ Two-package monorepo:
- `server/` -- single Express backend with layered architecture (routes -> services -> stores). Routes orchestrate; services contain business logic; stores manage in-memory state.
- `client/` -- React 19 SPA via Vite. Two-panel layout: sidebar for notebooks/data sources, main area for LLM chat with explorable citations, groundedness badges (cosine similarity scoring of LLM responses), and follow-up question chips.
- `client/` -- React 19 SPA via Vite. Two-panel layout: sidebar for data sources, main area for LLM chat with explorable citations, groundedness badges (cosine similarity scoring of LLM responses), and follow-up question chips.
## License