kjannette/citation_sentinel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Readme updates #1

Closed
kjannette wants to merge 1 commits from FEAT-readme-update into master
pull from: FEAT-readme-update
merge into: kjannette:master
Conversation 1 Commits 1 Files Changed 1 +30 -19
1 changed files with 30 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
README.md
View File

@@ -8,12 +8,12 @@ Built with a React/Vite frontend and a Typescript/Node/Express backend, using An
## Query pipeline
Source Ingestion -> Parsing -> Chunking -> Embedding (using Voyage AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> Evaluation of User Query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model)
Source Ingestion -> Parsing -> Chunking -> Embedding (using Voyage AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> Evaluation of User Query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model)
## Prerequisites
- **Node.js** (v18+)
- **yt-dlp** -- required for YouTube video source support (`brew install yt-dlp`)
- **yt-dlp** -- Required for YouTube video source support (`brew install yt-dlp`)
## Getting Started
@@ -38,28 +38,37 @@ This app demonstrates the core source-grounded Q&A pattern with transparent retr
## Methodology
This application is a RAG (Retrieval-Augmented Generation) system that allows users to upload source documents — PDFs, DOCX files, plain text, audio files, web URLs, and YouTube videos — which are then parsed, split into ~2000-character overlapping chunks, and converted into vector embeddings using Voyage AI's voyage-3 model. Those embeddings are stored in an in-memory vector store.
This is a RAG (Retrieval-Augmented Generation) and query-response source-groundedness assurance system allowing users to create a research knowledge corpus, including:
When a user submits a query, the system enforces groundedness through a multi-layered strategy:
1. Documents — PDFs, DOCX files, plain text audio files.
2. Internet sources: via web URLs.
2. Audio sources: i.e. YouTube videos, audio from which is transcribed to text.
These are then parsed, split into ~2000-character overlapping chunks, and converted into vector embeddings using Voyage AI's voyage-3 model.
The embeddings are stored in an in-memory vector store.
When a user submits a query, the system enforces groundedness through a multi-layered strategy:
1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against
stored chunk vectors using cosine similarity, returning the top 20 candidates.
stored chunk vectors using cosine similarity, returning the top 20 candidates.
2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder,
which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6).
The LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are
insufficient.
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6)
for natural language query response.
The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are
insufficient.
4. **Schema enforcement** — The LLM's response is constrained to a JSON schema requiring structured
fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not
correspond to real source groups are programmatically stripped out.
4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured
fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not
correspond to real source groups are programmatically removed.
5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage
AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors
of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a single
groundedness score that is surfaced to the user with a visual indicator (green/yellow/red).
AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors
of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a single
groundedness score that is surfaced to the user with a visual indicator (green/yellow/red).
## Similarity Metrics for Semantic Understanding
@@ -71,6 +80,8 @@ Archetypical cosine scores range from -1 to 1.
0: Vectors are at a 90-degree angle (orthogonal/unrelated).
-1: Vectors point in opposite directions.
Cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. In this application, scores are normalized to a bounded range wherein unrelated query-response pair scores are nearly orthogonal.
## Design
Two-package monorepo:
@@ -85,4 +96,4 @@ MIT. See [LICENSE](./LICENSE).
## Author
[@sjdev](https://sjdev.co)
@sjdev