Sem_Cache
A tiny CLI tool to test reliability/performance of your Semantic Caching layer (such as RedisElastiCache, Momento).
It generates a tuned cosine similarity score for LLM queries, and tells you, in plain English, if a hypothetical cached vector should have been returned upon submission of a proposed query vector.
Written in Python, it uses the Sentence Transformers library with NumPy to compute vector embeddings and cosine similarity between text inputs, determining whether an incoming query is similar enough to a cached query to return a stored response instead of incurring the computational and other expenses associated with a new LLM Application Programming Interface (API) call.
Released under the GPL Version 3. Initial release: August 2026.
This project is intended for developers building LLM applications who want to reduce API costs, latency, and redundant computations by caching semantically equivalent queries.
It can be used to quickly verify the reliability of a semantic caching layer by generating a cosine score using two test sentences, input as command line strings.
The output can be diff’d against the behavior of the system under development for an instant “sanity check.”
How It Works
If two queries mean the same thing, they should return the same answer. Rather than comparing strings character-by-character, it converts each query into a dense vector embedding using the all-MiniLM-L6-v2 model (a lightweight transformer that runs locally). It then calculates the cosine similarity between the cached query embedding and the new query embedding.
If the similarity score meets or exceeds the threshold (default: 0.92), the system declares a cache hit and would return the cached response. Otherwise, it declares a cache miss and routes the request to the LLM.
Getting Started
Prerequisites
Installation
Clone the repository and set up a virtual environment:
git clone https://github.com/kjannette/semantic-cache-script.git
cd semantic-cache-script
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
Install the required dependencies:
pip install numpy sentence-transformers
The first run will download the all-MiniLM-L6-v2 model (approximately 90 MB).
Usage
Run the script from the command line:
python sem_cache.py
The program will prompt you to enter two sentences:
- Sentence 1: The baseline cached query (simulating a query already in the cache)
- Sentence 2: The new incoming query (simulating a user's new request)
Example Session
============================================================
LOCAL SEMANTIC CACHE SIMULATOR
============================================================
Loading local embedding model (all-MiniLM-L6-v2)...
Model loaded successfully.
------------------------------------------------------------
Enter Sentence 1 (The Baseline Cached Query):
> What is the weather like today?
Enter Sentence 2 (The New Incoming Query):
> How's the weather today?
Generating local embeddings and performing vector math...
------------------------------------------------------------
RESULTS:
-> Calculated Cosine Similarity: 0.9412
-> Target Safety Threshold: 0.9200
[CACHE HIT] Returning cached response.
============================================================
In this example, the two queries are semantically equivalent, so the system returns a cache hit with a similarity score of 0.9412.
Configuration
The similarity threshold is set to 0.92 by default. This is a highly-conservative value, set to reduce false positives (treating dissimilar queries as matches). To adjust the threshold,
modify the THRESHOLD constant in sem_cache.py:
THRESHOLD = 0.92
Lower values increase cache hit rates but risk returning incorrect cached responses. Higher values reduce false positives but may miss valid semantic matches.
Project Structure
semantic-cache-script/
├── sem_cache.py # Main application script
├── README.md # This file
├── .gitignore # Git ignore patterns
└── .venv/ # Python virtual environment (not tracked)
Participation
Bug Reports
Bug reports are accepted via Git issues. Include the Python version, operating system, input sentences, and full error output when reporting issues.
Pull Requests
Pull requests are accepted for review. Project author makes no guarantee that contributions will be merged. No Contributor License Agreement (CLA) is required.
Code Style
This project follows PEP 8 style guidelines.
Author
- @ sjDev
Ideology
This project does not have a formal Code of Conduct.
Sem_Cache is a standalone, free software project. It is not associated with a for-profit company or "open core" offering. The software runs locally and does not transmit activity data off the device where it runs (the embedding model executes on-device; no external API calls are made by the caching logic itself).
Roadmap / TO-DO
(This README also serves as a development notebook.)
Planned Features
- Print result to command line (completed in current version)
- Perform and output quantified metrics of what a cache hit conserves
Possible Metrics to Implement
- Estimated completion tokens — Calculate tokens saved by popular model (assuming cache did not exist or cache misses)
- Latency — Measure the API call latency to the LLM that would be avoided on cache hit
- Rate limit / quota impact — Theoretical calculation of quota preservation (low priority)
- Cost of redundant embedding — Quantify savings from not re-embedding identical strings across runs, retries, and re-indexes
- Provider outage resilience — Document how cached responses provide continuity when upstream services are unavailable
- Reduction of nondeterminism — Address unreliable tests and unreproducible bugs by enabling diff comparisons between runs when model output varies
- Concurrency waste avoidance — Quantify savings when N users submit identical queries (e.g., 50 users click the same button, ask the same question; without cache, cost = 50x)
License
This software is released under the GNU General Public License Version 3.