3 Commits

Author SHA1 Message Date
KS Jannette
4f803c14f4 hotfix 2026-08-03 01:40:18 -04:00
KS Jannette
720d34b663 hotfix 2026-08-03 01:37:45 -04:00
KS Jannette
bf89c7544f Enhanced readme.md 2026-08-03 01:16:07 -04:00
2 changed files with 18 additions and 18 deletions

View File

@@ -1,10 +1,18 @@
# Citation Sentinel
A source-grounded research assistant that employs cosine similarity scoring for LLM-generated query responses to provide a "groundedness" score. This is a metric in Retrieval-Augmented Generation (RAG) systems that quantifies how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated or pulled from the model's pre-training data.
![Citation Sentinel UI showing the source sidebar, chat answer with citations, and groundedness badge](demo.jpg)
Users upload source documents, or provide links to online sources including audio/video (i.e. links to youtube videos). Users may then ask questions and receive answers (with inline citations) verifiably grounded in the provided information sources.
A source-grounded research assistant that empowers users to transform an otherwise-unmanageably-large corpus of data on any topic of interest into refined, easily-digested subtopics and pose focused inquiries, to instantly receive concise, quantifiably-evaluated, accurate responses. The system also provides suggested follow-up questions further refining in user inquiries by detecting the query goals.
Built with a React/Vite frontend and a Typescript/Node/Express backend, using Anthropic Claude for generation, OpenAI Whisper for video audio track transcription, Voyage AI for embeddings and response cosine similarity scoring (the "groundedness" score).
# Reliability and Understandability
Citation Sentinel's Retrieval-Augmented Generation (RAG) pipeline uses two-stage pass-through to Voyage AI embedding and ranking models, which uses cosine similarity methodology to scoring to evaluate query-source embeddings in multi-dimensional vector space. This yields a "groundedness" score.
Groundedness metrics quantify how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated, inaccurate or stale (pulled from the model's training data).
# Architecture - Basic Overview
Built with a React/Vite frontend and a Typescript/Node/Express backend, using Anthropic Claude for generation, OpenAI Whisper for video audio track transcription, Voyage AI voyage-3 for embeddings and Voyage AI rerank-2 for response cosine similarity scoring, to generate the "groundedness" score, presented as an easily-understandable red/yellow/green "badge" style tooltip in the response (see above.)
## Query pipeline
@@ -50,25 +58,15 @@ The embeddings are stored in an in-memory vector store.
When a user submits a query, the system enforces groundedness through a multi-layered strategy:
1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against
stored chunk vectors using cosine similarity, returning the top 20 candidates.
1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against stored chunk vectors using cosine similarity, returning the top 20 candidates.
2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder,
which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder, which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6)
for natural language query response.
The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are
insufficient.
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6) for natural language query response. The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are insufficient.
4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured
fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not
correspond to real source groups are programmatically removed.
4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not correspond to real source groups are programmatically removed.
5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage
AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors
of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a single
groundedness score that is surfaced to the user with a visual indicator (green/yellow/red).
5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a groundedness score surfaced to the user with a visual indicator (“badge” in green/yellow/red).
## Similarity Metrics for Semantic Understanding
@@ -97,3 +95,5 @@ MIT. See [LICENSE](./LICENSE).
## Author
@sjdev

BIN
demo.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 109 KiB