Compare commits
4 Commits
CHORE-upda
...
FEAT-user-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4f803c14f4 | ||
|
|
720d34b663 | ||
|
|
bf89c7544f | ||
| 8ec849b103 |
36
README.md
36
README.md
@@ -1,10 +1,18 @@
|
||||
# Citation Sentinel
|
||||
|
||||
A source-grounded research assistant that employs cosine similarity scoring for LLM-generated query responses to provide a "groundedness" score. This is a metric in Retrieval-Augmented Generation (RAG) systems that quantifies how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated or pulled from the model's pre-training data.
|
||||

|
||||
|
||||
Users upload source documents, or provide links to online sources including audio/video (i.e. links to youtube videos). Users may then ask questions and receive answers (with inline citations) verifiably grounded in the provided information sources.
|
||||
A source-grounded research assistant that empowers users to transform an otherwise-unmanageably-large corpus of data on any topic of interest into refined, easily-digested subtopics and pose focused inquiries, to instantly receive concise, quantifiably-evaluated, accurate responses. The system also provides suggested follow-up questions further refining in user inquiries by detecting the query goals.
|
||||
|
||||
Built with a React/Vite frontend and a Typescript/Node/Express backend, using Anthropic Claude for generation, OpenAI Whisper for video audio track transcription, Voyage AI for embeddings and response cosine similarity scoring (the "groundedness" score).
|
||||
# Reliability and Understandability
|
||||
|
||||
Citation Sentinel's Retrieval-Augmented Generation (RAG) pipeline uses two-stage pass-through to Voyage AI embedding and ranking models, which uses cosine similarity methodology to scoring to evaluate query-source embeddings in multi-dimensional vector space. This yields a "groundedness" score.
|
||||
|
||||
Groundedness metrics quantify how well an AI-generated answer is supported by retrieved context. It measures "faithfulness" to source documents, ensuring the answer is not hallucinated, inaccurate or stale (pulled from the model's training data).
|
||||
|
||||
# Architecture - Basic Overview
|
||||
|
||||
Built with a React/Vite frontend and a Typescript/Node/Express backend, using Anthropic Claude for generation, OpenAI Whisper for video audio track transcription, Voyage AI voyage-3 for embeddings and Voyage AI rerank-2 for response cosine similarity scoring, to generate the "groundedness" score, presented as an easily-understandable red/yellow/green "badge" style tooltip in the response (see above.)
|
||||
|
||||
## Query pipeline
|
||||
|
||||
@@ -50,25 +58,15 @@ The embeddings are stored in an in-memory vector store.
|
||||
|
||||
When a user submits a query, the system enforces groundedness through a multi-layered strategy:
|
||||
|
||||
1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against
|
||||
stored chunk vectors using cosine similarity, returning the top 20 candidates.
|
||||
1. **Retrieval constraint** — The query is embedded (via Voyage AI voyage-3) and compared against stored chunk vectors using cosine similarity, returning the top 20 candidates.
|
||||
|
||||
2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder,
|
||||
which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
|
||||
2. **Reranking for precision** — Those 20 candidates are sent to Voyage AI's rerank-2 cross-encoder, which re-scores each query-chunk pair with deeper semantic analysis. Only the top 5 survive.
|
||||
|
||||
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6)
|
||||
for natural language query response.
|
||||
The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are
|
||||
insufficient.
|
||||
3. **Prompt-level constraint** — The top 5 chunks are passed to the "Primary LLM" (Claude opus-4-6) for natural language query response. The Primary LLM must cite sources using bracketed indices (e.g., [1], [2]) and admit when sources are insufficient.
|
||||
|
||||
4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured
|
||||
fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not
|
||||
correspond to real source groups are programmatically removed.
|
||||
4. **Schema enforcement** — The Primary LLM's response is constrained to a JSON schema requiring structured fields (answer, citedSourceIndices, followUpQuestions). Any cited source indices that do not correspond to real source groups are programmatically removed.
|
||||
|
||||
5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage
|
||||
AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors
|
||||
of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a single
|
||||
groundedness score that is surfaced to the user with a visual indicator (green/yellow/red).
|
||||
5. **Post-generation groundedness scoring** — The answer is split into individual sentences. Voyage AI voyage-3 embeds each sentence, and compares it (using cosine similarity) against the vectors of the cited chunks. The similarity is calibrated to a 0–1 scale and averaged, producing a groundedness score surfaced to the user with a visual indicator (“badge” in green/yellow/red).
|
||||
|
||||
## Similarity Metrics for Semantic Understanding
|
||||
|
||||
@@ -97,3 +95,5 @@ MIT. See [LICENSE](./LICENSE).
|
||||
## Author
|
||||
|
||||
@sjdev
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user