kjannette/Semantic-Cache-Verifier
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

quick edit readme

Browse Source
This commit is contained in:
KS Jannette
2026-08-26 22:03:38 -04:00
parent cae6edfb78
commit b1c7655ecb
1 changed files with 31 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
README.md
View File

@@ -1,5 +1,7 @@
# [Sem_Cache](https://github.com/kjannette/semantic-cache-script) # [Sem_Cache](https://github.com/kjannette/semantic-cache-script)
![Script usage screenshot](semcache.jpg)
Sem_Cache is a tiny command-line tool written in [Python](https://www.python.org/) that Sem_Cache is a tiny command-line tool written in [Python](https://www.python.org/) that
demonstrates semantic caching for Large Language Model (LLM) queries. It uses the demonstrates semantic caching for Large Language Model (LLM) queries. It uses the
[Sentence Transformers](https://www.sbert.net/) library with [NumPy](https://numpy.org/) [Sentence Transformers](https://www.sbert.net/) library with [NumPy](https://numpy.org/)
@@ -21,6 +23,8 @@ The output can be diff’d against the behavior of the system under development
--- ---
## How It Works ## How It Works
If two queries mean the same thing, they should return the same answer. Rather than comparing strings character-by-character, it converts each query into a dense vector embedding using the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model (a lightweight transformer that runs locally). It then calculates the cosine similarity between the cached query embedding and the new query embedding. If two queries mean the same thing, they should return the same answer. Rather than comparing strings character-by-character, it converts each query into a dense vector embedding using the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model (a lightweight transformer that runs locally). It then calculates the cosine similarity between the cached query embedding and the new query embedding.
@@ -31,13 +35,19 @@ a **cache miss** and routes the request to the LLM.
--- ---
## Getting Started ## Getting Started
### Prerequisites ### Prerequisites
- [Python](https://www.python.org/) 3.10 or higher - [Python](https://www.python.org/) 3.10 or higher
- [pip](https://pip.pypa.io/) (Python package installer) - [pip](https://pip.pypa.io/) (Python package installer)
### Installation ### Installation
Clone the repository and set up a virtual environment: Clone the repository and set up a virtual environment:
@@ -70,6 +80,8 @@ The program will prompt you to enter two sentences:
1. **Sentence 1**: The baseline cached query (simulating a query already in the cache) 1. **Sentence 1**: The baseline cached query (simulating a query already in the cache)
2. **Sentence 2**: The new incoming query (simulating a user's new request) 2. **Sentence 2**: The new incoming query (simulating a user's new request)
### Example Session ### Example Session
``` ```
@@ -101,6 +113,8 @@ a cache hit with a similarity score of 0.9412.
--- ---
## Configuration ## Configuration
The similarity threshold is set to 0.92 by default. This is a highly-conservative value, set to reduce false positives (treating dissimilar queries as matches). To adjust the threshold, The similarity threshold is set to 0.92 by default. This is a highly-conservative value, set to reduce false positives (treating dissimilar queries as matches). To adjust the threshold,
@@ -115,6 +129,8 @@ Higher values reduce false positives but may miss valid semantic matches.
--- ---
## Project Structure ## Project Structure
``` ```
@@ -127,8 +143,12 @@ semantic-cache-script/
--- ---
## Participation ## Participation
### Bug Reports ### Bug Reports
Bug reports are accepted via [Git issues](https://github.com/kjannette/semantic-cache-script/issues). Bug reports are accepted via [Git issues](https://github.com/kjannette/semantic-cache-script/issues).
@@ -146,12 +166,16 @@ This project follows [PEP 8](https://peps.python.org/pep-0008/) style guidelines
--- ---
## Author ## Author
- @ sjDev - @ sjDev
--- ---
## Ideology ## Ideology
This project does not have a formal Code of Conduct. This project does not have a formal Code of Conduct.
@@ -163,6 +187,8 @@ executes on-device; no external API calls are made by the caching logic itself).
--- ---
## Roadmap / TO-DO ## Roadmap / TO-DO
(This README also serves as a development notebook.) (This README also serves as a development notebook.)
@@ -172,6 +198,8 @@ executes on-device; no external API calls are made by the caching logic itself).
- [ ] Print result to command line (completed in current version) - [ ] Print result to command line (completed in current version)
- [ ] Perform and output quantified metrics of what a cache hit conserves - [ ] Perform and output quantified metrics of what a cache hit conserves
### Possible Metrics to Implement ### Possible Metrics to Implement
1. **Estimated completion tokens** — Calculate tokens saved by popular model 1. **Estimated completion tokens** — Calculate tokens saved by popular model
@@ -192,6 +220,8 @@ executes on-device; no external API calls are made by the caching logic itself).
--- ---
## License ## License
This softeare is released under the [GNU General Public License Version 3](https://opensource.org/license/gpl-3-0). This softeare is released under the [GNU General Public License Version 3](https://opensource.org/license/gpl-3-0).