Add shared library for APIs/chunking

README.md

@@ -0,0 +1,66 @@
+# RAG Lambdas - Reliable, Source Grounded Serverless Retrevial Augmented Generation Pipeline
+
+AWS Lambda functions for source-grounded Q&A: ingest documents from S3 to Pinecone,  answer questions with citations and a groundedness score.
+
+Embeddings and reranking use the Voyage API; generation and NLI use Anthropic. Lambdas need in/outbound HTTPS to `api.voyageai.com` and `api.anthropic.com` for REST API interactions.
+
+## Pipeline
+
+```
+ingest:
+  S3 object
+    → overlapping ~2000-char chunks
+    → Voyage voyage-3 (input_type=document)
+    → Pinecone upsert { text, source, chunkIndex }
+
+query:
+  question
+    → Voyage voyage-3 (input_type=query)
+    → Pinecone top-20 (cosine)
+    → Voyage rerank-2 → top-5
+    → Claude opus-4-6 (JSON answer + citations + follow-ups)
+    → strip citation indices that are not in the retrieved set
+    → groundedness:
+         A. per sentence, rerank-2(sentence, cited chunk texts) → max score
+         B. optional Haiku NLI: supported | contradicted | unsupported
+         C. linear ramp on rerank scores (0.2–0.8, not cosine 0.35–0.65)
+    → { answer, citations, groundednessScore, followUpQuestions, contextUsed }
+```
+
+Note: Groundedness is not bi-encoder cosine. Cosine is used only for Pinecone ANN retrieval. The badge is Voyage `rerank-2` over cited chunk text, with an optional Haiku entailment pass. 
+
+## Optional Rerank
+
+`GROUNDEDNESS_MODE=rerank` skips the Haiku LLM-judge (less expensive/lower latency). `rerank_nli` (default) runs both.
+
+Raw per-sentence rerank scores are logged as `sentence_groundedness` (for planned isotonic calibrator fit to labeled pairs - future update).
+
+## Pinecone index
+
+Voyage-3 embeddings are **1024-dimensional**.  Create a new index, then ingest.
+
+- metric: `cosine`
+- dimension: `1024`
+
+## Environment
+
+Copy `.env.example` and set:
+
+| Variable | Purpose |
+|---|---|
+| `VOYAGE_API_KEY` | voyage-3 embed + rerank-2 |
+| `ANTHROPIC_API_KEY` | opus-4-6 generation + Haiku NLI |
+| `PINECONE_API_KEY` | vector store |
+| `PINECONE_INDEX_NAME` | 1024-d cosine index |
+| `GROUNDEDNESS_MODE` | `rerank` or `rerank_nli` |
+
+## Lambdas
+
+- `ingest.js` — S3 trigger. Chunks, embeds, upserts.
+- `query.js` — API Gateway / HTTP. Body: `{ "question": "..." }`.
+
+Shared code lives in `lib/`. Package both handlers with `node_modules` (Node 18+, ESM: `"type": "module"` in `package.json`).
+
+```bash
+npm install
+```