kjannette/Semantic-Cache-Verifier
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Steven Jannette
2026-08-27 20:24:10 +00:00
parent b1c7655ecb
commit 8dbf2dd65e
1 changed files with 114 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

112
README.md
View File

@@ -1,57 +1,84 @@
# [Sem_Cache](https://github.com/kjannette/semantic-cache-script)
![Script usage screenshot](semcache.jpg)
Sem_Cache is a tiny command-line tool written in [Python](https://www.python.org/) that
demonstrates semantic caching for Large Language Model (LLM) queries. It uses the
[Sentence Transformers](https://www.sbert.net/) library with [NumPy](https://numpy.org/)
to compute vector embeddings and [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)
between text inputs, determining whether an incoming query is similar enough to a
cached query to return a stored response instead of making a new LLM Application
Programming Interface (API) call.
A tiny CLI tool to test reliability/performance of your Semantic Caching layer (such as RedisElastiCache, Momento).
It generates a tuned cosine similarity score for LLM queries, and tells you, in plain English, if a hypothetical cached vector should have been returned upon submission of a proposed query vector.
Written in [Python](https://www.python.org/), it uses the
[Sentence Transformers](https://www.sbert.net/) library with [NumPy](https://numpy.org/) to compute vector embeddings and [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)
between text inputs, determining whether an incoming query is similar enough to a cached query to return a stored response instead of incurring the computational and other expenses associated with a new LLM Application Programming Interface (API) call.
Released under the
[GPL Version 3](https://opensource.org/license/gpl-3-0). Initial release: August 2026.
This project is intended for developers building LLM applications who want
to reduce API costs, latency, and redundant computations by caching semantically
equivalent queries.
It can be used to quickly verify the reliability of a semantic caching layer by generating a cosine score using two test sentences, input as command line strings.
The output can be diff’d against the behavior of the system under development for an instant “sanity check.”
---
## How It Works
If two queries mean the same thing, they should return the same answer. Rather than comparing strings character-by-character, it converts each query into a dense vector embedding using the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model (a lightweight transformer that runs locally). It then calculates the cosine similarity between the cached query embedding and the new query embedding.
If the similarity score meets or exceeds the threshold (default: 0.92), the system
declares a **cache hit** and would return the cached response. Otherwise, it declares
a **cache miss** and routes the request to the LLM.
---
## Getting Started
### Prerequisites
- [Python](https://www.python.org/) 3.10 or higher
- [pip](https://pip.pypa.io/) (Python package installer)
### Installation
Clone the repository and set up a virtual environment:
```bash
git clone https://github.com/kjannette/semantic-cache-script.git
cd semantic-cache-script
@@ -59,80 +86,109 @@ python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
```
Install the required dependencies:
```bash
pip install numpy sentence-transformers
```
The first run will download the all-MiniLM-L6-v2 model (approximately 90 MB).
### Usage
Run the script from the command line:
```bash
python sem_cache.py
```
The program will prompt you to enter two sentences:
1. **Sentence 1**: The baseline cached query (simulating a query already in the cache)
2. **Sentence 2**: The new incoming query (simulating a user's new request)
### Example Session
```
============================================================
LOCAL SEMANTIC CACHE SIMULATOR
============================================================
Loading local embedding model (all-MiniLM-L6-v2)...
Model loaded successfully.
------------------------------------------------------------
Enter Sentence 1 (The Baseline Cached Query):
> What is the weather like today?
Enter Sentence 2 (The New Incoming Query):
> How's the weather today?
Generating local embeddings and performing vector math...
------------------------------------------------------------
RESULTS:
-> Calculated Cosine Similarity: 0.9412
-> Target Safety Threshold: 0.9200
[CACHE HIT] Returning cached response.
============================================================
```
In this example, the two queries are semantically equivalent, so the system returns
a cache hit with a similarity score of 0.9412.
---
## Configuration
The similarity threshold is set to 0.92 by default. This is a highly-conservative value, set to reduce false positives (treating dissimilar queries as matches). To adjust the threshold,
modify the `THRESHOLD` constant in `sem_cache.py`:
```python
THRESHOLD = 0.92
```
Lower values increase cache hit rates but risk returning incorrect cached responses.
Higher values reduce false positives but may miss valid semantic matches.
---
## Project Structure
```
semantic-cache-script/
├── sem_cache.py # Main application script
@@ -141,67 +197,101 @@ semantic-cache-script/
└── .venv/ # Python virtual environment (not tracked)
```
---
## Participation
### Bug Reports
Bug reports are accepted via [Git issues](https://github.com/kjannette/semantic-cache-script/issues).
Include the Python version, operating system, input sentences, and full error output
when reporting issues.
### Pull Requests
Pull requests are accepted for review. Project author makes no guarantee that
contributions will be merged. No Contributor License Agreement (CLA) is required.
### Code Style
This project follows [PEP 8](https://peps.python.org/pep-0008/) style guidelines.
---
## Author
- @ sjDev
---
## Ideology
This project does not have a formal Code of Conduct.
Sem_Cache is a standalone, free software project. It is not associated with a
for-profit company or "open core" offering. The software runs locally and does
not transmit activity data off the device where it runs (the embedding model
executes on-device; no external API calls are made by the caching logic itself).
---
## Roadmap / TO-DO
(This README also serves as a development notebook.)
### Planned Features
- [ ] Print result to command line (completed in current version)
- [ ] Perform and output quantified metrics of what a cache hit conserves
### Possible Metrics to Implement
1. **Estimated completion tokens** — Calculate tokens saved by popular model
(assuming cache did not exist or cache misses)
2. **Latency** — Measure the API call latency to the LLM that would be avoided
@@ -218,10 +308,16 @@ executes on-device; no external API calls are made by the caching logic itself).
identical queries (e.g., 50 users click the same button, ask the same
question; without cache, cost = 50x)
---
## License
This softeare is released under the [GNU General Public License Version 3](https://opensource.org/license/gpl-3-0).
This software is released under the [GNU General Public License Version 3](https://opensource.org/license/gpl-3-0).