I need a proxy that caches LLM responses (semantic caching) to reduce API costs.

To implement semantic caching for your LLM calls, you need a reverse proxy or AI gateway that intercepts incoming prompts, converts them into vector embeddings, performs a vector similarity search, and either returns the cached answer or forwards the request to the upstream LLM provider.

The top open-source proxies and libraries providing semantic caching include:

• LiteLLM Proxy : An extremely popular, production-ready, OpenAI-compatible proxy. It natively supports redis-semantic and qdrant-semantic caching tiers. It handles load balancing, virtual keys, and drop-in replacements for 100+ LLM providers while checking vector similarity thresholds before hitting external APIs. Check out the setup steps in the LiteLLM Caching Documentation.
• GPTCache : Developed by Zilliz, this is a dedicated library/service specifically built to create semantic caches for LLM applications. It allows you to plug in various embedding models (like HuggingFace, OpenAI) and vector databases (Milvus, Chroma, Redis, FAISS) to sit as a caching layer right in front of your application logic or custom proxy setup. Learn more at the GPTCache GitHub Repository.
• PromptCache : A lightweight, self-hosted semantic cache built explicitly as a middle-tier service for GenAI workloads to cut down latency and token expenditure. It features runtime threshold configurations and SSE streaming support. Explore the code on the PromptCache GitHub Repository.
• Kong AI Gateway : If you are already running Kong for API management, it includes an AI Semantic Cache plugin that translates prompts into vector embeddings and caches responses natively at the gateway infrastructure layer.

If you want to narrow down the best choice, tell me:

• Are you looking for a standalone proxy container (like LiteLLM) or a Python library to embed directly in your code (like GPTCache)?
• Which vector database or cache backend (e.g., Redis, Qdrant) do you already use in your stack?

Yes. What you want is essentially an OpenAI-compatible LLM gateway/proxy with semantic response caching: the proxy embeds an incoming prompt, searches previous requests by similarity, and returns the stored response when the similarity exceeds a threshold.

My shortlist

• docs.semcache.io — Probably the closest match if your primary requirement is “a proxy whose main job is semantic caching.” It exposes HTTP/OpenAI-compatible routes and supports OpenAI, Anthropic, DeepSeek, Mistral, and custom upstreams.
• docs.litellm.ai — Best if you also want a self-hosted gateway for multiple providers, fallbacks, routing, budgets, etc. Its caching ecosystem supports semantic backends such as Redis + embeddings and Qdrant.
• portkey.ai — Good managed option. Its gateway supports semantic caching based on cosine similarity, with TTL configuration; semantic caching is aimed at production/enterprise deployments.
• helicone.ai — Worth considering if observability, cost tracking, and debugging are as important as caching. Its proxy supports caching, although its current direction is broader AI-gateway functionality.

What I'd choose

If you're building your own infrastructure, I'd start with:

Client → LiteLLM/Semcache → Redis/Qdrant → OpenAI/Anthropic/etc.

For a simple dedicated caching layer, Semcache is particularly interesting. For a broader production gateway, LiteLLM + Redis/Qdrant gives you more control.

One important design point: don't use semantic matching blindly. You generally want the cache key to incorporate things like model, system prompt/version, relevant generation parameters, tenant, and application/version, while the user prompt gets semantic matching. Otherwise you can return an old answer after changing your system prompt or model.

A typical flow is:

                         ┌── cache hit ──→ cached response
                         │
App → LLM Proxy → embed → similarity search
                         │
                         └── cache miss → LLM API
                                             │
                                             └→ store response + embedding

For factual or highly dynamic queries, I'd also use a lower similarity threshold or bypass semantic caching entirely; semantic caching is most valuable for repetitive workloads such as support questions, documentation Q&A, and common instructions. Research has found substantial reductions in redundant LLM calls in repetitive workloads, though the actual savings depend heavily on your traffic.

If you tell me (1) self-hosted vs managed, (2) OpenAI only vs multiple providers, and (3) Redis/Qdrant/Postgres available, I can recommend one architecture and give you a ready-to-run Docker setup.

If you want a proxy layer in front of your LLM APIs that does semantic caching, you have a few good options depending on whether you want self-hosted, OpenAI-compatible, or managed.

Self-hosted semantic caching proxies

• Semcache — A lightweight HTTP proxy that can sit between your app and providers. It supports OpenAI-compatible usage, semantic similarity matching, multiple providers, and metrics/dashboard features.

• Good fit: you want base_url replacement with minimal app changes.

• Typical flow: ``` App → Semcache → (cache hit?) → return cached response ↓ LLM provider

- PromptCache — A provider-agnostic Go proxy with semantic matching, OpenAI-compatible endpoints, streaming support, and configurable thresholds. [GitHub](https://github.com/messkan/prompt-cache?utm_source=chatgpt.com)

 - Good fit: production gateway where you need control over cache policies.
- GPTCache — One of the earlier open-source semantic caching projects. It can sit before your LLM calls and reuse previous answers based on similarity. [ACL Anthology](https://aclanthology.org/2023.nlposs-1.24.pdf?utm_source=chatgpt.com)

 - Good fit: embedding caching directly into an application rather than running a standalone gateway.

## Managed proxies

- SemaCache — A hosted drop-in proxy. You point your OpenAI-compatible client at its endpoint and it performs exact-match and semantic cache lookups before forwarding misses upstream. [SemaCache](https://www.semacache.io/?utm_source=chatgpt.com)
- ProxyLLM — A managed proxy offering semantic caching plus routing and cost tracking. [ProxyLLM](https://www.proxyllm.dev/?utm_source=chatgpt.com)

## Architecture I would use for a production setup

            ┌─────────────┐
            │ Your App     │
            └──────┬──────┘
                   │ OpenAI SDK
                   ▼
          ┌────────────────┐
          │ LLM Proxy      │
          │                │
          │ 1. Exact hash  │
          │ 2. Embed query │
          │ 3. Vector search
          │ 4. Similarity check
          └──────┬─────────┘
                 │ miss
                 ▼
      ┌────────────────────┐
      │ OpenAI/Anthropic/etc│
      └────────────────────┘

Backend pieces:

- **Vector store:** pgvector, Qdrant, Redis Vector, Milvus
- **Embeddings:** small/cheap embedding model
- **Cache metadata:** - model name
- system prompt hash
- temperature
- tools/function schema
- tenant/user scope
- TTL
- **Similarity threshold:** - high threshold → automatic hit
- medium → optional verification model
- low → bypass cache

## Important production considerations

1. **Do not cache blindly**

 - A prompt like “summarize this private document” should usually include the document hash and user scope in the cache key.
2. **Include generation parameters**
 Cache keys should include:

model temperature top_p system prompt tool definitions user context

3. **Handle streaming**
 If your app uses SSE streaming, make sure the proxy can replay cached streams. Some proxies support this explicitly. [GitHub](https://github.com/messkan/prompt-cache?utm_source=chatgpt.com)
4. **Measure hit rate**
 A semantic cache is only valuable if:

savings = cache_hit_rate × average_LLM_cost

 Track:

 - hit %
- false-hit rate
- latency saved
- tokens avoided

For a typical SaaS app using OpenAI-compatible APIs, I would start with an **OpenAI-compatible self-hosted proxy (Semcache or PromptCache) backed by Redis/Postgres vector storage**, then add verification logic if incorrect cache hits are costly. [GitHub](https://github.com/sensoris/semcache?utm_source=chatgpt.com) [GitHub](https://github.com/messkan/prompt-cache?utm_source=chatgpt.com)

To implement a semantic caching proxy for LLMs, you have a few robust, production-ready open-source and managed gateway options. Instead of exact string matching, these tools generate vector embeddings of incoming prompts and query a vector database or specialized cache backend (like Redis or Qdrant) to return identical or conceptually similar answers.

Here are the top choices to set this up:

• LiteLLM Proxy : An open-source, OpenAI-compatible AI gateway that supports native redis-semantic and qdrant-semantic caching . It standardizes calls across 100+ providers and allows you to configure similarity thresholds and TTLs easily. Check out the setup steps in the LiteLLM Caching Documentation.
• Portkey AI Gateway : A production-grade AI gateway with out-of-the-box semantic caching capabilities, automatic fallbacks, and load balancing. It integrates directly into your request flow so you don't have to manage a separate standalone caching layer. You can review implementation options via Portkey.
• GPTCache (by Zilliz) : A dedicated, highly adaptable open-source library specifically built to create semantic caches for LLM apps. It integrates seamlessly with frameworks like LangChain and LlamaIndex and lets you plug in various vector stores and embedding models. Explore the codebase on the GPTCache GitHub Repository.
• LLM Cache Proxy : A lightweight, high-performance open-source caching proxy written in Rust that uses Redis and Qdrant to drop right into existing OpenAI or Groq SDK calls for massive latency and cost drops. Details are available on the LLM Cache Proxy GitHub.

If you'd like to narrow this down, tell me:

• Are you looking for a self-hosted open-source proxy (like LiteLLM/GPTCache) or a managed gateway?
• What infrastructure (like Redis, PostgreSQL/pgvector, or Qdrant) do you already run in your stack?

I can help you map out the exact configuration file or setup commands.

Yes. If you specifically want a proxy/gateway that sits in front of your LLM APIs and does semantic response caching, there are several good options.

My shortlist

• semacache.io — Probably the closest match to your request. It's a managed, drop-in LLM proxy supporting OpenAI, Claude, Gemini, Grok and others. You change the API base URL, and it performs semantic matching before forwarding misses to the provider. It advertises configurable semantic caching and cost/latency analytics.
• proxyllm.dev — Another drop-in managed proxy, with semantic caching plus model routing and cost tracking. It supports OpenAI- and Anthropic-compatible clients.
• portkey.ai — More of a full LLM gateway than a cache. It has both exact and semantic caching, plus routing, fallbacks, observability, etc. Good if caching is only one part of what you need.
• redis.io — My choice if you want to own the caching layer or already use Redis. Redis supports vector similarity lookup, TTLs, metadata filtering, and managed LangCache. You can put a thin proxy in front of it.
• github.com — Open-source Go middleware specifically designed as a semantic caching proxy. Useful if you want to self-host rather than put requests through another SaaS.
• github.com — Lightweight open-source HTTP proxy with semantic caching, multiple LLM-provider support, metrics, and an admin dashboard. It's currently beta.

What I'd choose

One important caveat: semantic caching isn't appropriate for every LLM request. You generally want to cache deterministic-ish questions where two semantically equivalent prompts should receive the same answer. Personalized responses, current-data queries, authorization-sensitive requests, and rapidly changing information should normally bypass the cache. Redis, for example, supports TTLs and metadata filters specifically to help control these issues.

If you tell me your stack (Python/Node/Go), which providers you use (OpenAI/Anthropic/etc.), and whether you want managed or self-hosted, I can narrow this to the best 2–3 options and show the integration architecture.

To slash LLM API costs and reduce latency using semantic caching, you can choose between managed drop-in proxies, open-source AI gateways, or modular python libraries . Semantic caching intercepts your prompts, converts them into embeddings, and performs a vector similarity search (like cosine distance) against past queries. If the meaning matches above a certain threshold (e.g., 0.88), it serves the cached response instantly without hitting the LLM provider.

The leading options fit different architectural preferences:

• LiteLLM Proxy : An open-source, highly popular Python-based proxy that gives you a unified OpenAI-compatible endpoint. It supports semantic caching natively by integrating with vector/cache backends like Redis (using RediSearch) or Qdrant. It handles routing, virtual keys, and spend tracking.
• Kong AI Gateway / AI Semantic Cache Plugin : If you already manage microservices with Kong, their enterprise AI gateway includes native semantic caching and routing capabilities. It embeds incoming prompts and cross-references them via Redis to instantly serve hits.
• Bifrost : A high-performance, native Go-based AI gateway option known for lower baseline overhead compared to Python frameworks. It features dual-layer caching (L1 exact hash → L2 semantic vector search), supports multiple vector stores (Redis, Qdrant, Weaviate, Pinecone), and allows per-request overrides via custom headers.
• GPTCache : An open-source library by Zilliz that you can use to build your own custom semantic caching proxy service. It integrates smoothly with LangChain and LlamaIndex and lets you plug in various embedding models and vector databases.
• ProxyLLM : A managed, drop-in cloud proxy service that sits between your application and AI providers, requiring just a base URL swap to immediately enable semantic caching and cost dashboards.

To help narrow down the right choice, tell me:

• Do you want to self-host an open-source gateway or use a managed service?
• What is your preferred infrastructure stack (e.g., do you already run Redis/Postgres)?
• Are you handling streaming responses?

If you want a proxy layer in front of your LLM APIs that does semantic response caching, you have several good options. The right choice depends on whether you want a drop-in OpenAI-compatible gateway, a library, or a full LLM gateway.

Drop-in semantic caching proxies

If you are building your own proxy

A typical architecture:

Your App
   |
   | OpenAI-compatible API
   v
Semantic Cache Proxy
   |
   +--> Embed request
   |
   +--> Vector similarity search
   |
   +--> Cache hit?
   |       |
   |       +--> return stored completion
   |
   +--> Cache miss
           |
           v
       LLM Provider
           |
           v
       Store prompt embedding + response

Common components:

• HTTP layer: FastAPI, Go, Rust, Envoy, Traefik
• Embedding model: OpenAI embeddings, BGE, E5, or local embedding models
• Vector store: Qdrant, Redis Vector Search, Milvus, pgvector
• Cache metadata: TTL, model name, temperature, system prompt hash, tenant ID
• Safety controls:
  • Only cache deterministic requests (temperature=0)
  • Include system prompt and tool schema in cache keys
  • Use similarity thresholds + optional verification for borderline matches
  • Partition cache by user/tenant when responses contain private data

Production settings I would recommend

For a customer-facing app:

• Exact-match cache first (Redis) → cheapest lookup
• Semantic cache second (vector DB)
• Only reuse responses when:
  • same model
  • same system prompt
  • same tools/functions
  • same permissions context
  • similarity score above a tuned threshold

Example flow:

request
  |
  +-- Redis exact hash?
  |       yes -> return
  |
  +-- Vector similarity > 0.92?
  |       yes -> return cached answer
  |
  +-- otherwise call LLM

My shortlist

• Fastest path: Semcache or PromptCache — change your OpenAI base_url and test.
• Need routing + multiple providers: ThriftLLM.
• Need maximum control: build around Redis + Qdrant/pgvector and keep the cache policy in your service.

If you tell me your stack (OpenAI API, Anthropic, Azure OpenAI, self-hosted models, Kubernetes, language/runtime), expected QPS, and whether responses are private per user, I can narrow this down.

To cut API costs and latency using semantic caching, you can drop a specialized open-source AI gateway or proxy between your application and your LLM providers (OpenAI, Anthropic, etc.). Instead of an exact character match, these proxies convert incoming prompts into vector embeddings and check for high cosine similarity against previous interactions.

The top production-ready options available depend on your preferred infrastructure stack:

• LiteLLM Proxy : An open-source, Python-based universal proxy that exposes an OpenAI-compatible endpoint for over 100 providers. It natively supports semantic caching via a redis-semantic (Redis Stack with RediSearch) or qdrant-semantic configuration block.
• Bifrost : A high-performance, open-source AI gateway written in Go. It features a native dual-layer caching architecture (exact hash matching plus semantic similarity search) and supports backends like Redis, Qdrant, Weaviate, and Pinecone. Ideal if you require minimal baseline overhead and maximum throughput.
• Kong AI Gateway : An enterprise-grade API gateway featuring an ai-semantic-cache plugin. If your infrastructure already relies on Kong for API management, adding its native plugin layer allows you to leverage Redis as a vector store without spinning up an extra standalone proxy service.
• GPTCache : A lightweight, modular open-source caching library by Zilliz rather than a full monolithic gateway. It embeds smoothly into custom Python applications or LangChain/LlamaIndex pipelines if you prefer embedding the cache logic directly into your application code instead of running an independent proxy.

To help you select and configure the best setup, tell me:

• Are you looking for a stand-alone proxy/gateway (like LiteLLM or Bifrost) or an embedded code library (like GPTCache)?
• What is your preferred vector/cache backend (e.g., Redis Stack, Qdrant, Postgres/pgvector)?

Several production-ready AI gateways and open-source proxies natively support semantic caching by embedding incoming prompts, evaluating vector similarity against a store (like Redis or Qdrant), and short-circuiting redundant API calls. The top choices include the LiteLLM Proxy with its built-in redis-semantic or qdrant-semantic modes, the Kong AI Gateway via its AI Semantic Caching plugin, and open-source specialized setups like Bifrost or Semcache.

Top LLM Proxies with Semantic Caching

• LiteLLM Proxy : An open-source Python-based proxy featuring a unified OpenAI-compatible interface. You can activate semantic caching by configuring a vector backend (Redis/Qdrant) inside its config YAML file.
• Kong AI Gateway : Enterprise API gateway that offers an AI Semantic Cache Plugin to automatically embed incoming prompts and query a Redis vector store before hitting upstream vendors.
• Bifrost : A high-performance, open-source AI gateway built in Go that implements a dual-layer caching strategy combining exact hash matches and semantic similarity search.
• Semcache : A dedicated lightweight semantic caching proxy layer designed to let you point your existing LLM SDKs directly to a local or hosted drop-in replacement endpoint.

Key Implementation Factors

• Similarity Threshold : Controls how close a new prompt's vector must be to a cached vector (e.g., cosine similarity > 0.92) to trigger a cache hit. Setting this too loose introduces stale or hallucinated answers; setting it too tight reduces your cache hit rate.
• Vector Backend : Most production proxies rely on Redis Stack (with RediSearch HNSW indexes) or Qdrant to perform sub-millisecond similarity lookups.
• Complementary Savings : Combine your semantic cache proxy with provider-level prompt caching (native to models like Anthropic Claude or OpenAI) to discount input tokens for large shared system prompts or RAG context blocks.

If you'd like, let me know:

• Your preferred deployment stack (Docker, Kubernetes, Python/Node.js service)
• Which LLM provider or model you primarily use
• Your anticipated query volume

I can help you select and configure the ideal proxy architecture.

Yes. If you want a drop-in proxy in front of OpenAI/Anthropic/etc. that can return semantically similar cached responses, I’d shortlist these:

My recommendation: LiteLLM + Redis

LiteLLM GitHub is probably the best starting point if you want a production gateway rather than just a caching library. It provides an OpenAI-compatible proxy, supports 100+ providers, and has caching functionality with Redis integration.

A typical architecture would be:

Your application
       │
       ▼
┌──────────────────────┐
│     LiteLLM Proxy    │
│                      │
│  normalize request   │
│        ↓             │
│  semantic similarity │
│        ↓             │
│   Redis/vector cache │─── HIT ──► cached response
│        │
│       MISS
│        ▼
│   LLM provider       │
└──────────────────────┘
       │
       ▼
 OpenAI / Anthropic / Gemini / etc.

The important distinction is that ordinary response caching only catches essentially identical requests. Semantic caching embeds the incoming query and searches for a sufficiently similar previous query. Research has shown this can substantially reduce repeated LLM calls, although the similarity threshold is critical because an overly aggressive cache can return an inappropriate answer.

If semantic caching is the main requirement

I'd also look at Khazad. It's a newer open-source approach that intercepts httpx requests and uses Redis vector search to match semantically similar conversations, with support for OpenAI, Anthropic, Gemini, Azure OpenAI and others.

That is particularly attractive if your Python application already uses an HTTP-based SDK and you don't want to replace every LLM client with a gateway endpoint.

One important design choice

I wouldn't blindly cache every LLM request. I'd use something like:

request
   │
   ├── exact-match cache ────────► hit
   │
   ├── semantic cache
   │       │
   │       ├── similarity > 0.95 ─► hit
   │       └── otherwise
   │
   └── LLM
          │
          └── store response + embedding

And make the cache key include at least:

• model
• system prompt/version
• relevant conversation context
• tools/function definitions
• important generation parameters
• tenant/user scope where appropriate

For RAG, customer support, FAQ, classification, and other repetitive workloads, semantic caching can be particularly effective. For highly personalized, real-time, or rapidly changing queries, I'd either disable it or use a very conservative threshold.

If you tell me your stack (Python/Node/Go), LLM providers, and approximate requests/day, I can recommend a concrete architecture—including Redis/vector DB choice and the cache similarity strategy.