RAG Lambdas - Reliable, Source Grounded Serverless Retrevial Augmented Generation Pipeline

AWS Lambda functions for source-grounded Q&A: ingest documents from S3 to Pinecone, answer questions with citations and a groundedness score.

Embeddings and reranking use the Voyage API; generation and NLI use Anthropic. Lambdas need in/outbound HTTPS to api.voyageai.com and api.anthropic.com for REST API interactions.

Pipeline

ingest:
  S3 object
    → overlapping ~2000-char chunks
    → Voyage voyage-3 (input_type=document)
    → Pinecone upsert { text, source, chunkIndex }

query:
  question
    → Voyage voyage-3 (input_type=query)
    → Pinecone top-20 (cosine)
    → Voyage rerank-2 → top-5
    → Claude opus-4-6 (JSON answer + citations + follow-ups)
    → strip citation indices that are not in the retrieved set
    → groundedness:
         A. per sentence, rerank-2(sentence, cited chunk texts) → max score
         B. optional Haiku NLI: supported | contradicted | unsupported
         C. linear ramp on rerank scores (0.2–0.8, not cosine 0.35–0.65)
    → { answer, citations, groundednessScore, followUpQuestions, contextUsed }

Note: Groundedness is not bi-encoder cosine. Cosine is used only for Pinecone ANN retrieval. The badge is Voyage rerank-2 over cited chunk text, with an optional Haiku entailment pass.

Optional Rerank

GROUNDEDNESS_MODE=rerank skips the Haiku LLM-judge (less expensive/lower latency). rerank_nli (default) runs both.

Raw per-sentence rerank scores are logged as sentence_groundedness (for planned isotonic calibrator fit to labeled pairs - future update).

Pinecone index

Voyage-3 embeddings are 1024-dimensional. Create a new index, then ingest.

metric: cosine
dimension: 1024

Environment

Copy .env.example and set:

Variable	Purpose
`VOYAGE_API_KEY`	voyage-3 embed + rerank-2
`ANTHROPIC_API_KEY`	opus-4-6 generation + Haiku NLI
`PINECONE_API_KEY`	vector store
`PINECONE_INDEX_NAME`	1024-d cosine index
`GROUNDEDNESS_MODE`	`rerank` or `rerank_nli`

Lambdas

ingest.js — S3 trigger. Chunks, embeds, upserts.
query.js — API Gateway / HTTP. Body: { "question": "..." }.

Shared code lives in lib/. Package both handlers with node_modules (Node 18+, ESM: "type": "module" in package.json).

npm install

2.4 KiB Raw Blame History Unescape Escape