diff --git a/README.md b/README.md index 99d516f..c3e8c75 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # [Sem_Cache](https://github.com/kjannette/semantic-cache-script) +![Script usage screenshot](semcache.jpg) + Sem_Cache is a tiny command-line tool written in [Python](https://www.python.org/) that demonstrates semantic caching for Large Language Model (LLM) queries. It uses the [Sentence Transformers](https://www.sbert.net/) library with [NumPy](https://numpy.org/) @@ -21,6 +23,8 @@ The output can be diff’d against the behavior of the system under development --- + + ## How It Works If two queries mean the same thing, they should return the same answer. Rather than comparing strings character-by-character, it converts each query into a dense vector embedding using the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model (a lightweight transformer that runs locally). It then calculates the cosine similarity between the cached query embedding and the new query embedding. @@ -31,13 +35,19 @@ a **cache miss** and routes the request to the LLM. --- + + ## Getting Started + + ### Prerequisites - [Python](https://www.python.org/) 3.10 or higher - [pip](https://pip.pypa.io/) (Python package installer) + + ### Installation Clone the repository and set up a virtual environment: @@ -70,6 +80,8 @@ The program will prompt you to enter two sentences: 1. **Sentence 1**: The baseline cached query (simulating a query already in the cache) 2. **Sentence 2**: The new incoming query (simulating a user's new request) + + ### Example Session ``` @@ -101,6 +113,8 @@ a cache hit with a similarity score of 0.9412. --- + + ## Configuration The similarity threshold is set to 0.92 by default. This is a highly-conservative value, set to reduce false positives (treating dissimilar queries as matches). To adjust the threshold, @@ -115,6 +129,8 @@ Higher values reduce false positives but may miss valid semantic matches. --- + + ## Project Structure ``` @@ -127,8 +143,12 @@ semantic-cache-script/ --- + + ## Participation + + ### Bug Reports Bug reports are accepted via [Git issues](https://github.com/kjannette/semantic-cache-script/issues). @@ -146,12 +166,16 @@ This project follows [PEP 8](https://peps.python.org/pep-0008/) style guidelines --- + + ## Author - @ sjDev --- + + ## Ideology This project does not have a formal Code of Conduct. @@ -163,6 +187,8 @@ executes on-device; no external API calls are made by the caching logic itself). --- + + ## Roadmap / TO-DO (This README also serves as a development notebook.) @@ -172,6 +198,8 @@ executes on-device; no external API calls are made by the caching logic itself). - [ ] Print result to command line (completed in current version) - [ ] Perform and output quantified metrics of what a cache hit conserves + + ### Possible Metrics to Implement 1. **Estimated completion tokens** — Calculate tokens saved by popular model @@ -192,6 +220,8 @@ executes on-device; no external API calls are made by the caching logic itself). --- + + ## License -This softeare is released under the [GNU General Public License Version 3](https://opensource.org/license/gpl-3-0). +This softeare is released under the [GNU General Public License Version 3](https://opensource.org/license/gpl-3-0). \ No newline at end of file