diff --git a/README.md b/README.md index 8a6812c..0f71df7 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,19 @@ # Citation Sentinel -A source-grounded research assistant that employs cosine similarity scoring for LLM-generated query responses to provide a "groundedness" score. This is a metric in Retrieval-Augmented Generation (RAG) systems that quantifies how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated or pulled from the model's pre-training data. +A source-grounded research assistant that employs cosine similarity scoring for LLM-generated query responses to provide a "groundedness" score. This is a metric in Retrieval-Augmented Generation (RAG) systems that quantifies how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated or pulled from the model's pre-training data. -Users upload source documents, or provide links to online sources including audio/video (i.e. links to youtube videos). Users may then ask questions and receive answers (with inline citations) verifiably grounded in the provided information sources. +Users upload source documents, or provide links to online sources including audio/video (i.e. links to youtube videos). Users may then ask questions and receive answers (with inline citations) verifiably grounded in the provided information sources. Built with a React/Vite frontend and a Typescript/Node/Express backend, using Anthropic Claude for generation, OpenAI Whisper for video audio track transcription, Voyage AI for embeddings and response cosine similarity scoring (the "groundedness" score). ## Query pipeline -Source Ingestion -> Parsing -> Chunking -> Embedding (using Voyage AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> Evaluation of User Query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model) +Source Ingestion -> Parsing -> Chunking -> Embedding (using Voyage AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> Evaluation of User Query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model) ## Prerequisites - **Node.js** (v18+) -- **yt-dlp** -- required for YouTube video source support (`brew install yt-dlp`) +- **yt-dlp** -- Required for YouTube video source support (`brew install yt-dlp`) ## Getting Started @@ -38,39 +38,50 @@ This app demonstrates the core source-grounded Q&A pattern with transparent retr ## Methodology -This application is a RAG (Retrieval-Augmented Generation) system that allows users to upload source documents — PDFs, DOCX files, plain text, audio files, web URLs, and YouTube videos — which are then parsed, split into ~2000-character overlapping chunks, and converted into vector embeddings using Voyage AI's voyage-3 model. Those embeddings are stored in an in-memory vector store. +This is a RAG (Retrieval-Augmented Generation) and query-response source-groundedness assurance system allowing users to create a research knowledge corpus, including: -When a user submits a query, the system enforces groundedness through a multi-layered strategy: +1. Documents — PDFs, DOCX files, plain text audio files. +2. Internet sources: via web URLs. +2. Audio sources: i.e. YouTube videos, audio from which is transcribed to text. + +These are then parsed, split into ~2000-character overlapping chunks, and converted into vector embeddings using Voyage AI's voyage-3 model. + +The embeddings are stored in an in-memory vector store. + +When a user submits a query, the system enforces groundedness through a multi-layered strategy: 1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against - stored chunk vectors using cosine similarity, returning the top 20 candidates. + stored chunk vectors using cosine similarity, returning the top 20 candidates. 2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder, - which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive. + which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive. -3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6). - The LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are - insufficient. +3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6) + for natural language query response. + The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are + insufficient. -4. **Schema enforcement** — The LLM's response is constrained to a JSON schema requiring structured - fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not - correspond to real source groups are programmatically stripped out. +4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured + fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not + correspond to real source groups are programmatically removed. 5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage - AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors - of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a single - groundedness score that is surfaced to the user with a visual indicator (green/yellow/red). + AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors + of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a single + groundedness score that is surfaced to the user with a visual indicator (green/yellow/red). ## Similarity Metrics for Semantic Understanding Cosine similarity measures how closely two vectors (representing data like words, images, or preferences) are aligned in a multi-dimensional space by calculating the cosine of the angle between them. -Archetypical cosine scores range from -1 to 1. +Archetypical cosine scores range from -1 to 1. 1: Vectors point in the exact same direction (highly similar). 0: Vectors are at a 90-degree angle (orthogonal/unrelated). -1: Vectors point in opposite directions. +Cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. In this application, scores are normalized to a bounded range wherein unrelated query-response pair scores are nearly orthogonal. + ## Design Two-package monorepo: @@ -85,4 +96,4 @@ MIT. See [LICENSE](./LICENSE). ## Author -[@sjdev](https://sjdev.co) +@sjdev