# RAG Lambdas - Reliable, Source Grounded Serverless Retrevial Augmented Generation Pipeline

AWS Lambda functions for source-grounded Q&A: ingest documents from S3 to Pinecone,  answer questions with citations and a groundedness score.

Embeddings and reranking use the Voyage API; generation and NLI use Anthropic. Lambdas need in/outbound HTTPS to `api.voyageai.com` and `api.anthropic.com` for REST API interactions.

## Pipeline

```
ingest:
  S3 object
    → overlapping ~2000-char chunks
    → Voyage voyage-3 (input_type=document)
    → Pinecone upsert { text, source, chunkIndex }

query:
  question
    → Voyage voyage-3 (input_type=query)
    → Pinecone top-20 (cosine)
    → Voyage rerank-2 → top-5
    → Claude opus-4-6 (JSON answer + citations + follow-ups)
    → strip citation indices that are not in the retrieved set
    → groundedness:
         A. per sentence, rerank-2(sentence, cited chunk texts) → max score
         B. optional Haiku NLI: supported | contradicted | unsupported
         C. linear ramp on rerank scores (0.2–0.8, not cosine 0.35–0.65)
    → { answer, citations, groundednessScore, followUpQuestions, contextUsed }
```

Note: Groundedness is not bi-encoder cosine. Cosine is used only for Pinecone ANN retrieval. The badge is Voyage `rerank-2` over cited chunk text, with an optional Haiku entailment pass. 

## Optional Rerank

`GROUNDEDNESS_MODE=rerank` skips the Haiku LLM-judge (less expensive/lower latency). `rerank_nli` (default) runs both.

Raw per-sentence rerank scores are logged as `sentence_groundedness` (for planned isotonic calibrator fit to labeled pairs - future update).

## Pinecone index

Voyage-3 embeddings are **1024-dimensional**.  Create a new index, then ingest.

- metric: `cosine`
- dimension: `1024`

## Environment

Copy `.env.example` and set:

| Variable | Purpose |
|---|---|
| `VOYAGE_API_KEY` | voyage-3 embed + rerank-2 |
| `ANTHROPIC_API_KEY` | opus-4-6 generation + Haiku NLI |
| `PINECONE_API_KEY` | vector store |
| `PINECONE_INDEX_NAME` | 1024-d cosine index |
| `GROUNDEDNESS_MODE` | `rerank` or `rerank_nli` |

## Lambdas

- `ingest.js` — S3 trigger. Chunks, embeds, upserts.
- `query.js` — API Gateway / HTTP. Body: `{ "question": "..." }`.

Shared code lives in `lib/`. Package both handlers with `node_modules` (Node 18+, ESM: `"type": "module"` in `package.json`).

```bash
npm install
```