kjannette/citation_sentinel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Exlicative imporovements in readme

Browse Source
This commit is contained in:
KS Jannette
2026-08-06 02:41:23 -04:00
parent d481b79465
commit 62dd6de60c
1 changed files with 47 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
README.md
View File

@@ -1,99 +1,141 @@
# Citation Sentinel # Citation Sentinel © 2023 @sjDev
![Citation Sentinel UI showing the source sidebar, chat answer with citations, and groundedness badge](demo.jpg) ![Citation Sentinel UI showing the source sidebar, chat answer with citations, and groundedness badge](demo.jpg)
A source-grounded research assistant that empowers users to transform an otherwise-unmanageably-large corpus of data on any topic of interest into refined, easily-digested subtopics and pose focused inquiries, to instantly receive concise, quantifiably-evaluated, accurate responses. The system also provides suggested follow-up questions further refining in user inquiries by detecting the query goals.
Image one: source-grounded research assistant that empowers users to transform an otherwise-unmanageably-large corpus of data on any topic of interest into refined, easily-digested subtopics and pose focused inquiries, to instantly receive concise, quantifiably-evaluated, accurate responses. The system also provides suggested follow-up questions further refining in user inquiries by detecting the query goals.
Image Two: demo co clickable inline citations that take user to source-grounded research basis/knowledge corpus supporting query response, with rated “groundedness” score.
# Reliability and Understandability # Reliability and Understandability
Citation Sentinel's Retrieval-Augmented Generation (RAG) pipeline uses two-stage pass-through to Voyage AI embedding and ranking models, which uses cosine similarity methodology to scoring to evaluate query-source embeddings in multi-dimensional vector space. This yields a "groundedness" score. Citation Sentinel's Retrieval-Augmented Generation (RAG) pipeline uses two-stage pass-through to Voyage AI embedding and ranking models, which uses cosine similarity methodology to scoring to evaluate query-source embeddings in multi-dimensional vector space. This yields a "groundedness" score.
Groundedness metrics quantify how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated, inaccurate or stale (pulled from the model's training data). Groundedness metrics quantify how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated, inaccurate or stale (pulled from the model's training data).
# Architecture - Basic Overview # Architecture - Basic Overview
Built with a React/Vite frontend and a Typescript/Node/Express backend, using Anthropic Claude for generation, OpenAI Whisper for video audio track transcription, Voyage AI voyage-3 for embeddings and Voyage AI rerank-2 for response cosine similarity scoring, to generate the "groundedness" score, presented as an easily-understandable red/yellow/green "badge" style tooltip in the response (see above.)
Built with a React/Vote frontend and a Typescript/Node/Express backend, using Anthropic Claude for generation, OpenAI Whisper for video audio track transcription, Voyage AI voyage-3 for embeddings and Voyage AI rerank-2 for response cosine similarity scoring, to generate the "groundedness" score, presented as an easily-understandable red/yellow/green "badge" style tooltip in the response (see above.)
## Query pipeline ## Query pipeline
Source Ingestion -> Parsing -> Chunking -> Embedding (using Voyage AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> Evaluation of User Query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model) Source Ingestion -> Parsing -> Chunking -> Embedding (using Voyage AI voyage-3 model) -> Storage (vector store) -> { user query submission } -> Evaluation of User Query -> Retrieval -> Ranking -> Response Generation (Using Anthopic's claude-opus-4-6 model) -> Response Groundedness Scoring (using Voyage AI rerank-r model)
## Prerequisites ## Prerequisites
- **Node.js** (v18+) - **Node.js** (v18+)
- **yt-dlp** -- Required for YouTube video source support (`brew install yt-dlp`) - **yt-dlp** -- Required for YouTube video source support (`brew install yt-dlp`)
## Getting Started ## Getting Started
```bash ```bash
# clone and install # clone and install
git clone <repo-url> && cd citation_sentinel git clone <repo-url> && cd citation_sentinel
cd server && npm install && cd .. cd server && npm install && cd ..
cd client && npm install && cd .. cd client && npm install && cd ..
# configure # configure
cp server/.env.example server/.env cp server/.env.example server/.env
# edit server/.env and add your ANTHROPIC_API_KEY, VOYAGE_API_KEY, # edit server/.env and add your ANTHROPIC_API_KEY, VOYAGE_API_KEY,
# and optionally OPENAI_API_KEY (only needed for audio source transcription via Whisper) # and optionally OPENAI_API_KEY (only needed for audio source transcription via Whisper)
# run # run
make dev make dev
``` ```
## Rationale ## Rationale
This app demonstrates the core source-grounded Q&A pattern with transparent retrieval, generation, and groundedness scoring (LLM response quality cosine scoring) -- all with swappable models and is fully open source. This app demonstrates the core source-grounded Q&A pattern with transparent retrieval, generation, and groundedness scoring (LLM response quality cosine scoring) -- all with swappable models and is fully open source.
## Methodology ## Methodology
This is a RAG (Retrieval-Augmented Generation) and query-response source-groundedness assurance system allowing users to create a research knowledge corpus, including: This is a RAG (Retrieval-Augmented Generation) and query-response source-groundedness assurance system allowing users to create a research knowledge corpus, including:
1. Documents — PDFs, DOCX files, plain text audio files. 1. Documents — PDFs, DOCX files, plain text audio files.
2. Internet sources: via web URLs. 2. Internet sources: via web URLs.
2. Audio sources: i.e. YouTube videos, audio from which is transcribed to text. 2. Audio sources: i.e. YouTube videos, audio from which is transcribed to text.
These are then parsed, split into ~2000-character overlapping chunks, and converted into vector embeddings using Voyage AI's voyage-3 model. These are then parsed, split into ~2000-character overlapping chunks, and converted into vector embeddings using Voyage AI's voyage-3 model.
The embeddings are stored in an in-memory vector store. The embeddings are stored in an in-memory vector store.
When a user submits a query, the system enforces groundedness through a multi-layered strategy: When a user submits a query, the system enforces groundedness through a multi-layered strategy:
1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against stored chunk vectors using cosine similarity, returning the top 20 candidates. 1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against stored chunk vectors using cosine similarity, returning the top 20 candidates.
2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder, which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive. 2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder, which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6) for natural language query response. The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are insufficient. 3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6) for natural language query response. The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are insufficient.
4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not correspond to real source groups are programmatically removed. 4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not correspond to real source groups are programmatically removed.
5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a groundedness score surfaced to the user with a visual indicator (“badge” in green/yellow/red). 5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a groundedness score surfaced to the user with a visual indicator (“badge” in green/yellow/red).
## Similarity Metrics for Semantic Understanding ## Similarity Metrics for Semantic Understanding
Cosine similarity measures how closely two vectors (representing data like words, images, or preferences) are aligned in a multi-dimensional space by calculating the cosine of the angle between them. Cosine similarity measures how closely two vectors (representing data like words, images, or preferences) are aligned in a multi-dimensional space by calculating the cosine of the angle between them.
Archetypical cosine scores range from -1 to 1. Archetypical cosine scores range from -1 to 1.
1: Vectors point in the exact same direction (highly similar). 1: Vectors point in the exact same direction (highly similar).
0: Vectors are at a 90-degree angle (orthogonal/unrelated). 0: Vectors are at a 90-degree angle (orthogonal/unrelated).
-1: Vectors point in opposite directions. -1: Vectors point in opposite directions.
Cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. In this application, scores are normalized to a bounded range wherein unrelated query-response pair scores are nearly orthogonal. Cosine distance between high-dimensional text embeddings is compressed: unrelated notes sit close to orthogonal, so both the within-cluster and nearest-cluster distances land near 0.8. In this application, scores are normalized to a bounded range wherein unrelated query-response pair scores are nearly orthogonal.
## Design ## Design
Two-package monorepo: Two-package monorepo:
- `server/` -- single Express backend with layered architecture (routes -> services -> stores). Routes orchestrate; services contain business logic; stores manage in-memory state. - `server/` -- single Express backend with layered architecture (routes -> services -> stores). Routes orchestrate; services contain business logic; stores manage in-memory state.
- `client/` -- React 19 SPA via Vite. Two-panel layout: sidebar for data sources, main area for LLM chat with explorable citations, groundedness badges (cosine similarity scoring of LLM responses), and follow-up question chips. - `client/` -- React 19 SPA via Vite. Two-panel layout: sidebar for data sources, main area for LLM chat with explorable citations, groundedness badges (cosine similarity scoring of LLM responses), and follow-up question chips.
## License ## License
MIT. See [LICENSE](./LICENSE). MIT. See [LICENSE](./LICENSE).
## Author ## Author
@sjdev @sjdev