We want to implement rate limiting and caching for our internal LLM API. What's the best open-source API gateway for this?

When implementing rate limiting and caching for an internal LLM API, the clear frontrunner in the open-source ecosystem is LiteLLM , though a few infrastructure-level or performance-focused alternatives are gaining traction depending on your stack.

The top open-source options break down as follows based on how they handle caching and rate limiting:

1. LiteLLM (The Ecosystem & Feature Standard)

LiteLLM Proxy is widely considered the standard for internal LLM proxies. It translates standard OpenAI-formatted API requests to 100+ LLM providers.

• Rate Limiting: Supports robust per-key budgets, Requests Per Minute (RPM), and Tokens Per Minute (TPM) tracking . It also handles automatic fallback and cooldown on provider 429 Too Many Requests errors.
• Caching: Integrates natively with Redis for distributed request caching and virtual-key authentication caching.
• The Catch: Written in Python/FastAPI, so its raw throughput ceiling under massive enterprise loads (>10k RPS) is lower than compiled languages, though more than enough for most internal company APIs.

2. Portkey (The Feature-Rich Alternative)

Portkey’s open-source core offers production-grade reliability tools.

• Rate Limiting & Caching: Provides advanced routing, automatic retries, and both semantic and exact-match caching via Redis to prevent redundant LLM generation.
• The Catch: While it has a very polished UI and robust logging, some advanced enterprise extensions lean on their managed tier, though the core gateway remains open source.

3. Envoy AI Gateway (The Kubernetes/Infrastructure Choice)

Built on top of the Envoy proxy, the Envoy AI Gateway is designed natively for cloud-native and Kubernetes service-mesh environments.

• Rate Limiting & Caching: Inherits Envoy’s blistering fast, distributed, low-latency C++ performance and robust global rate-limiting infrastructure.
• The Catch: It is newer to the LLM-specific feature set and lacks high-level LLM native niceties like semantic caching or complex virtual-key budget hierarchies out of the box.

4. Bifrost (The High-Throughput Go Option)

A newer open-source entry written in Go specifically to address the performance overhead of Python-based gateways.

• Rate Limiting & Caching: Offers token-aware rate limiting, multi-key pooling, and semantic caching with microsecond-level proxy overhead.
• The Catch: Smaller community and ecosystem compared to LiteLLM.

Recommendation Summary

• Go with LiteLLM if you need the fastest setup, support for dozens of diverse model providers, and easy budget/token tracking per internal team.
• Go with Envoy AI Gateway if you are an infrastructure/Platform Engineering team deeply embedded in Kubernetes and want maximum proxy performance with minimal latency.

To help narrow down the best fit, tell me:

• What is your estimated peak RPS (Requests Per Second)?
• Are you deploying on Kubernetes or standard Docker/VMs?
• Which LLM providers (OpenAI, Anthropic, local vLLM/Ollama) are you targeting?

For an internal LLM API, I’d shortlist Kong Gateway and LiteLLM, but they solve slightly different problems.

My pick: Kong Gateway

If you primarily need a general-purpose API gateway with solid rate limiting, caching, auth, and operational maturity, Kong is probably the best fit.

• Rate limiting: per consumer, credential, service, route, etc.; supports local, cluster, and Redis-backed strategies.
• Caching: built-in Proxy Cache with configurable TTLs, cache keys, response codes, methods, and content types.
• LLM-specific capabilities: current Kong also has specialized AI/LLM rate-limiting functionality.
• Deployment: works well in Kubernetes, Docker, traditional infrastructure, and hybrid/multi-cloud environments.
• Open source: the core Kong Gateway is open source, although some advanced functionality is commercial.

One caveat: generic HTTP response caching isn't necessarily the right cache for LLM completions. For LLMs, you'll often want a semantic/request-aware cache keyed on things like model, messages, temperature, tools, and relevant generation parameters.

When I'd choose LiteLLM instead

If your gateway is specifically an LLM gateway, rather than an API gateway that happens to front an LLM, I'd seriously consider LiteLLM Proxy.

It gives you LLM-native features such as:

• model/provider abstraction
• rate limits and budgets per key/team/user
• usage and cost tracking
• model routing/load balancing
• authentication hooks
• LLM-specific observability

Its proxy is explicitly designed as an LLM gateway and exposes rate-limiting and cost controls directly.

The architecture I'd use

For an internal platform, I'd lean toward:

                 ┌───────────────┐
Applications ───►│ Kong Gateway  │
                 │ auth          │
                 │ rate limits   │
                 │ quotas        │
                 └───────┬───────┘
                         │
                         ▼
                 ┌───────────────┐
                 │ LiteLLM Proxy │
                 │ model routing │
                 │ cost tracking │
                 │ provider API  │
                 └───────┬───────┘
                         │
                ┌────────┴────────┐
                ▼                 ▼
             OpenAI            Anthropic
             Azure             Local models

But don't deploy both by default. If your API is relatively simple—e.g. "our services call /v1/chat/completions, and we need authentication + quotas + caching + observability"—I'd start with LiteLLM alone.

If you already have a broader microservice/API-gateway platform and LLM is just one of many upstreams, use Kong.

Rule of thumb:

One important distinction: if by "caching" you mean avoid paying the LLM provider when the same/similar prompt comes in, tell me that. I'd recommend a somewhat different architecture than simply enabling Kong's HTTP proxy cache.

For an internal LLM API, my first choice would be Apache APISIX.

apisix.apache.org is particularly well suited because LLM-specific capabilities are built into the open-source gateway rather than requiring you to assemble generic API-gateway plugins.

My recommendation: Apache APISIX

It gives you:

• Token-based rate limiting — you can limit consumers by tokens rather than just requests, which is much more meaningful for LLM workloads.
• LLM response caching — its ai-cache plugin supports exact-prompt caching through Redis and optional semantic caching.
• Streaming cache support — SSE responses can be cached and replayed, with incomplete streams excluded from the cache.
• Multiple model/provider routing — useful if your internal API eventually fronts OpenAI, Anthropic, Bedrock, self-hosted models, etc.
• Retries, fallbacks, load balancing and observability at the gateway layer.
• Apache 2.0 licensing and an open plugin architecture.

A sensible architecture would be:

Internal applications
        │
        ▼
   ┌───────────┐
   │  APISIX   │
   │           │
   │ Auth      │
   │ Rate limit│─── Redis
   │ AI cache  │─── Redis
   │ Routing   │
   │ Metrics   │
   └─────┬─────┘
         │
    ┌────┴─────────────┐
    ▼                  ▼
OpenAI/Anthropic   vLLM/Ollama/etc.

For an internal service, I'd use two levels of limiting:

1. Request-level limits — e.g. 60 requests/minute per service/team.
2. Token-level limits — e.g. 500k input+output tokens/hour per team.

The second one is important: two requests aren't remotely equivalent in LLM cost. APISIX's AI rate limiter explicitly supports limiting total, prompt, or completion tokens.

What about Kong?

konghq.com is the strongest alternative I'd consider. Kong has an AI Gateway, token-aware AI rate limiting, and caching/semantic-cache capabilities.

I'd choose Kong if your organization already runs Kong or wants its broader enterprise ecosystem. For a new, open-source internal LLM gateway, though, I'd lean APISIX because its LLM-specific rate limiting and caching are particularly directly aligned with your requirements.

| APISIX | Kong --- | --- | --- Open source | Excellent | Good, but ecosystem has commercial components LLM/token rate limiting | Yes | Yes Exact response cache | Yes | Yes Semantic cache | Yes | Yes Multi-LLM routing | Yes | Yes Redis integration | Yes | Yes General API gateway | Excellent | Excellent My pick for a new internal LLM API | 🏆 APISIX | Runner-up

If your requirement is specifically "OpenAI-compatible internal endpoint + Redis + per-team token quotas + response caching", APISIX is a very good fit.

When implementing rate limiting and caching for an internal LLM API, the "best" open-source gateway depends on whether you need an AI-native proxy built specifically for LLMs or a traditional enterprise API gateway with AI extensions.

LLM traffic differs from standard REST traffic because you need to track tokens (not just requests), handle streaming responses (SSE), and potentially use semantic caching (matching prompt meaning, not just exact URLs).

Top Open-Source Options

1. LiteLLM (Best AI-Native Gateway)
   • Why it’s great: LiteLLM has become a standard open-source proxy explicitly built for LLM workflows. It translates standard OpenAI-format API requests to 100+ LLM providers (Anthropic, Bedrock, Vertex, local vLLM, etc.).
   • Rate Limiting: Supports robust budget tracking, per-user/per-team rate limits, and token-based rate limiting (RPM/TPM - requests/tokens per minute) using a Redis backend.
   • Caching: Built-in response caching using Redis to short-circuit identical prompts.
   • Best for: Fast setup, heavy LLM feature requirements, virtual API key management, and multi-model cost tracking. Check out the LiteLLM GitHub.
2. Kong AI Gateway (Best if you already use Kong)
   • Why it’s great: Kong is a heavy-hitting traditional API gateway. Its AI Gateway plugins allow you to route, transform, and log LLM traffic alongside your standard microservices.
   • Rate Limiting: Offers advanced AI rate-limiting plugins that analyze query costs and token counts.
   • Caching: Highly optimized proxy caching plugins, though true semantic caching features or advanced AI governance often require pushing into their enterprise tier.
   • Best for: Enterprises already running Kong infrastructure who want centralized, unified gateway management. Learn more via Kong AI Gateway Documentation.
3. Apache APISIX (Best Cloud-Native / High-Performance Option)
   • Why it’s great: Built on Nginx/Lua and etcd, APISIX offers dynamic routing and low latency. They have a dedicated APISIX AI Gateway initiative.
   • Rate Limiting & Caching: Fine-grained token rate limiting, multi-LLM load balancing, and upstream circuit breaking/fallbacks.
   • Best for: Teams looking for a high-performance, lightweight cloud-native gateway with native Kubernetes support and native AI proxy features.
4. Portkey / Helicone (Alternative Hosted/Self-Hosted Observability & Gateway layers)
   • Why they're great: While often used as managed cloud services, both offer open-source or self-hosted components focused strictly on LLM caching, semantic cache layers, guardrails, and granular rate-limiting.

Recommendation Summary

• Go with LiteLLM if your primary goal is rapid deployment for LLMs, easy provider-switching, virtual key distribution, and native token-budgeting.
• Go with Kong or Apache APISIX if you already have an existing API gateway footprint and need to enforce strict enterprise security, mTLS, or OIDC policies at the edge alongside your LLM traffic.

To help narrow down the ideal choice, tell me:

• Are you looking for exact-match caching or semantic/embedding-based caching?
• Do you need to track budgets and assign virtual keys to different internal teams/departments?

The "best" open-source API gateway for an internal LLM API depends heavily on your performance requirements, existing infrastructure , and whether you need basic exact-match caching or advanced semantic caching.

The top open-source choices tailored for LLM workloads are categorized below by their strengths:

• Bifrost : Best for high-throughput, ultra-low latency . Written in Go, it features an impressive 11 μs routing overhead at 5,000+ RPS. It natively supports token-aware rate limiting, hierarchical budgeting, and robust dual-layer exact and semantic caching (including streaming response caching).
• LiteLLM (Proxy) : Best for Python-centric teams and maximum model compatibility . It supports over 100 LLM providers out of the box and provides a polished proxy with virtual key management, spend tracking, and Redis/Qdrant-backed semantic caching. Keep in mind that heavy traffic loads (>1k RPS) can experience higher runtime overhead due to its Python foundation.
• Kong AI Gateway : Best for enterprises already running Kong . It extends Kong's battle-tested NGINX + Lua core with specialized AI plugins for provider routing, token rate-limiting, and semantic caching. Caveat : The most robust AI governance and advanced rate-limiting/caching plugins require a commercial license or Enterprise edition, meaning the free open-source tier can feel limited.
• Apache APISIX : Best for cloud-native, dynamic microservices . APISIX has expanded its hot-reloadable plugin ecosystem with native ai-proxy and token-aware rate-limiting capabilities, making it ideal if you manage your traffic via cloud-native ingress controllers.

If you'd like to narrow this down, please share:

• Your expected request volume (RPS) or token throughput
• Whether you prefer a Go vs. Python operational stack
• If you need semantic caching (matching similar prompts) or just exact-match caching

For an internal LLM API, I’d shortlist LiteLLM and Kong Gateway, with the choice depending on whether you want an LLM-native gateway or a general-purpose API gateway.

My recommendation: LiteLLM

LiteLLM GitHub is probably the best fit if the gateway is specifically for LLM traffic.

It gives you:

• Per-user/project/API-key rate limits
• Token-based budgets and limits, rather than only requests/sec
• Response caching, including Redis-backed caching
• Authentication, logging and spend/cost tracking
• Model routing, fallbacks and load balancing
• OpenAI-compatible API, making it easy to put in front of existing applications

The architecture is essentially:

                    ┌───────────────┐
Apps ──► LiteLLM ──►│ Redis         │
                    │ rate limits   │
                    │ cache         │
                    └───────────────┘
                           │
                           ▼
                    ┌───────────────┐
                    │ LLM providers │
                    │ OpenAI/Azure/ │
                    │ Anthropic/etc │
                    └───────────────┘

For an internal service, I'd start with LiteLLM + Redis.

When I'd choose Kong instead

Kong Gateway is the stronger choice if you already use Kong or need a broader API-management layer.

Kong has mature request/consumer rate limiting with local, cluster and Redis-backed strategies. developer.konghq.com Its AI Gateway additionally supports LLM-specific proxying and semantic caching.

One important caveat: Kong's particularly interesting AI token/cost-based rate limiting functionality is currently part of its AI Gateway Enterprise offering, so don't assume all of the LLM-specific features are in the open-source edition.

What about Envoy?

Envoy Gateway is excellent if you're already heavily invested in Kubernetes/Envoy. It provides local and distributed/global rate limiting, with Redis commonly used for the global rate-limit service.

But I'd not pick it solely for an LLM gateway: you'd likely end up building the LLM-aware caching, token accounting, routing, etc. yourself.

Quick decision

For your stated requirements, I'd start with LiteLLM + Redis. If your API is already behind Kong, however, I'd strongly consider putting the LLM functionality into Kong rather than introducing another gateway.

If you tell me whether you're running Kubernetes, Docker/VMs, or bare metal, and whether the cache should be exact-prompt or semantic, I can recommend a concrete architecture and configuration.

Choosing the best open-source API gateway for rate limiting and caching an internal LLM API depends heavily on your performance profile, existing infrastructure , and governance needs.

The top open-source options break down by their ideal use cases:

• Bifrost — Best for high-throughput, low-latency performance. Built in Go, it introduces minimal overhead (≈ 11 μ s at 5,000 RPS) and offers robust token-aware rate limiting, virtual-key cost attribution, and native semantic caching. It works well if you want a dedicated, high-performance standalone AI proxy.
• LiteLLM Proxy — Best for Python-heavy stacks and maximum provider breadth. It standardizes calls across over 100 LLM providers, featuring built-in virtual keys, per-team/per-key budget controls, and dual-tier caching (in-memory or Redis/vector-based semantic caching). The trade-off is that its Python runtime can face performance bottlenecks under extreme, high-concurrency enterprise traffic compared to Go or Rust proxies.
• Kong AI Gateway — Best for organizations already running Kong for traditional APIs. Kong extends its battle-tested, Nginx/Lua core via AI plugins (ai-proxy-advanced and ai-semantic-cache ). It allows you to manage traditional and LLM traffic on the same plane, though advanced semantic caching and complex governance features may require their enterprise tier.
• Envoy AI Gateway — Best for native Kubernetes and service-mesh environments. It extends the Envoy proxy and Kubernetes Gateway API directly, making it ideal if your infrastructure is deeply anchored around Istio/Envoy. Its open-source core has a steeper learning curve and currently lacks advanced semantic caching out of the box.
• Apache APISIX — Best for dynamic, cloud-native microservices. APISIX features an ai-proxy plugin suite that handles token-aware rate limiting and multi-provider load balancing alongside standard API traffic. Configuration can be complex if you aren't already using APISIX.

To help narrow down the ideal gateway for your team, tell me:

• What is your estimated peak requests per second (RPS) or daily token volume?
• Are you already running a Kubernetes/service mesh infrastructure (like Envoy or Kong) or looking for a standalone proxy?

For an internal LLM API, I’d pick Apache APISIX unless your main requirement is an OpenAI-compatible LLM proxy rather than a general API gateway.

My recommendation: Apache APISIX

Apache APISIX is a strong fit because it gives you both conventional API-gateway controls and LLM-specific functionality:

• Token-aware rate limiting — limit by prompt/total tokens rather than just requests. APISIX's ai-rate-limiting plugin supports limits by consumer, route, service, etc.
• Distributed rate limiting — Redis-backed rate limiting lets multiple gateway instances share quotas.
• LLM routing/failover — proxy to OpenAI, Anthropic, Bedrock, DeepSeek, Ollama, etc., with load balancing and fallback.
• Normal API-gateway features — auth, request limits, routing, observability, transformations, etc.
• Fully open source / Apache 2.0 and built around NGINX + etcd.
• The current 3.16 release also adds dynamic, variable-based AI rate limits, which is particularly useful for per-team/per-service quotas.

What about caching?

I'd separate HTTP/API caching from LLM response/prompt caching.

For ordinary cacheable API responses, APISIX can sit in front of your existing cache infrastructure. For LLMs, however, you'll probably want a dedicated semantic/prompt cache rather than blindly HTTP-caching POST responses. The cache key may need to incorporate model, system prompt, temperature, tools, relevant request parameters, etc.

The main alternative: LiteLLM

LiteLLM is arguably the better choice if your system is specifically an LLM gateway.

It provides an OpenAI-compatible interface across 100+ providers, plus routing/fallbacks, spend tracking, virtual keys, rate limiting, and configurable caching.

I'd choose:

For your stated use case, I'd start with APISIX + Redis, with APISIX handling authentication, per-team/service token quotas, routing, and gateway-level caching, and use a dedicated semantic cache only if repeated LLM prompts actually justify it.

If you tell me whether you're running Kubernetes, Docker/VMs, or bare metal, I can sketch a production architecture and config for APISIX + Redis + your LLM backend.

When choosing an open-source API gateway for rate limiting and caching internal LLM traffic, the landscape has converged around three top contenders. The "best" choice depends heavily on your team's existing technology stack, scale, and whether you need exact-match or semantic caching.

Top Open-Source LLM Gateways Compared

• Bifrost (Maxim AI) — Best for high-throughput production and microsecond-level performance.
  • Architecture: Built in Go (Apache 2.0 license), offering extreme performance with roughly 11μ s of proxy overhead at scale.
  • Rate Limiting & Caching: Features robust token-aware rate limiting, hierarchical budgeting, and advanced built-in semantic caching (which short-circuits semantically similar queries via vector math rather than just exact string matching).
  • Pros: Drop-in OpenAI-compatible API, zero-config rapid deployment, multi-provider failover, and native Model Context Protocol (MCP) support.
  • Cons: Newer ecosystem compared to traditional API management tools.
• LiteLLM Proxy — Best for Python-heavy environments and maximum model breadth.
  • Architecture: Open-source Python proxy supporting 100+ LLM providers under a unified OpenAI-compliant format.
  • Rate Limiting & Caching: Strong virtual key management, per-consumer/per-team budgets, and token limits (RPM/TPM). Supports semantic/exact caching via Redis or Qdrant backends.
  • Pros: Unmatched provider support (great if you use niche or self-hosted open models alongside commercial ones).
  • Cons: Python runtime overhead (5--8ms per request) can introduce tail latency bottlenecks under high concurrency (>1,000 RPS).
• Kong AI Gateway — Best for enterprises already running Kong in production.
  • Architecture: Extends the traditional Nginx/Lua-based Kong API Gateway with AI-specific plugins (ai-proxy, ai-semantic-cache, ai-rate-limiting-advanced).
  • Rate Limiting & Caching: Enables robust cost-based rate limiting and Redis-backed semantic caching.
  • Pros: Ideal if your platform team already standardizes on Kong for traditional REST APIs.
  • Cons: The open-source version is limited; advanced AI governance and semantic cache plugins often require paid enterprise tiers (Kong Konnect). Configuration complexity is high.

Recommendation Summary

• Go with Bifrost if your primary concerns are minimal latency overhead, heavy traffic volume, and native semantic caching.
• Go with LiteLLM if you need a quick Python-native setup with a massive array of obscure model integrations.
• Go with Kong only if your infrastructure team already runs standard Kong and you plan to manage AI traffic alongside regular microservices.

To help narrow down the right choice, could you share:

• Your expected request volume (RPS) or daily token load?
• Do you need semantic caching (catching queries that mean the same thing phrased differently) or is exact-match caching sufficient?
• What is your preferred infrastructure language/runtime (Go, Python, Kubernetes/Envoy)?

For an internal LLM API, my first choice would be Apache APISIX.

Apache APISIX is a particularly good fit because you need both conventional API-gateway controls and LLM-specific traffic management.

My shortlist

Why I'd pick APISIX

• Rate limiting: supports conventional request limits as well as LLM/token rate limiting. APISIX's AI gateway specifically supports token-based limits per consumer.
• Caching: its proxy-cache plugin supports POST requests, which matters for LLM APIs, and can use memory or disk with configurable cache keys and TTLs.
• LLM routing: it can proxy multiple providers/models, with load balancing, retries and fallbacks.
• Streaming: APISIX has explicit support for disabling proxy buffering for SSE/streaming APIs, important for streamed LLM responses.
• Extensibility: plugins can be written in Lua, Python, Go, Java, or Wasm if you eventually need custom LLM policies.

One important caveat: LLM caching

I wouldn't blindly cache every LLM POST. I'd make the cache key something like:

model + normalized_prompt + relevant_generation_parameters

and only cache requests where you know the response is safely reusable—for example, deterministic temperature-0 internal queries. Don't let an authenticated user's response accidentally become another user's cached response. APISIX's cache has consumer isolation specifically to help with this.

If your bigger requirement is "one OpenAI-compatible endpoint in front of OpenAI + Anthropic + Bedrock + local models, with semantic caching, spend tracking, fallbacks, etc.", I'd instead look hard at LiteLLM. Its proxy is specifically designed as an LLM gateway with provider abstraction and caching.

Rule of thumb:

• API gateway first, LLM second → APISIX
• LLM gateway first, API gateway second → LiteLLM
• Already standardized on Kong → Kong

For a new internal platform, I'd probably deploy APISIX + Redis and keep the LLM service itself behind it.