We're burning cash on GPU bills — how can we cut inference costs without hurting latency?

Cutting inference costs while keeping latency tight requires attacking the problem across the model, the serving stack, and the hardware utilization layer.

Here is a breakdown of the most effective strategies to lower your GPU bill without letting latency slip:

• Model Quantization : Move from FP16/BF16 to INT8, INT4, or advanced formats like FP4/NVFP4 using frameworks like vLLM or TensorRT-LLM . This slashes memory footprint and boosts memory bandwidth efficiency, often improving token generation latency while cutting VRAM usage in half or more.
• Continuous Batching and PagedAttention : Stop using static batch sizes. Implement continuous (dynamic) batching to pack incoming requests on the fly, maximizing GPU compute saturation. PagedAttention eliminates memory fragmentation in KV caching, allowing much higher concurrency without spiking time-to-first-token (TTFT).
• Speculative Decoding : Use a small, fast "draft" model to guess a few tokens ahead, and verify them in a single forward pass with your large primary model. This can yield a 1.5× to 2.5× speedup on generation latency while keeping output quality identical to the large model.
• Prefix Caching (Automatic KV Caching) : If your prompts share common system instructions, few-shot examples, or long document contexts, enable automatic prefix caching in serving engines like vLLM . It caches the key-value states of recurring prefixes so you never recompute them, drastically dropping TTFT and saving FLOPs.
• Kernel Fusion and Compilation : Run your models through compilers like TensorRT, ONNX Runtime , or torch.compile. Fusing operations reduces the overhead of launching separate GPU kernels and minimizes high-bandwidth memory (HBM) round-trips.
• Model Distillation / Right-Sizing : Benchmark if a smaller distilled model (e.g., an 8B or 14B parameter model) matches your fine-tuned 70B+ model for your specific constrained tasks. Often, a well-tailored smaller model running un-quantized or lightly quantized matches task accuracy at a fraction of the hardware cost.

To help narrow down the best play for your stack, tell me:

• What models are you currently running (size and architecture)?
• What serving framework or infrastructure (e.g., vLLM, TGI, Triton, pure PyTorch) are you using?
• Are your workloads bottlenecked by Time-to-First-Token (TTFT) or tokens-per-second generation?

Yes. The key is to reduce GPU-seconds per request while keeping your latency SLO fixed, rather than simply buying smaller GPUs.

Highest-ROI moves

1. Increase effective batching without adding queueing delay. Use continuous/in-flight batching so a request can join work already running instead of waiting for a conventional batch boundary. This is particularly valuable because decode is memory-bound and benefits from sharing GPU work.

• Keep a strict max queueing budget, e.g. 5–20 ms.
• Tune batch-token limits against TTFT and inter-token latency, not just tokens/sec.
• With vLLM, chunked prefill can improve GPU utilization while prioritizing active decodes; smaller max_num_batched_tokens generally favors inter-token latency, while larger values favor TTFT.

2. Turn on prefix/KV caching aggressively where prompts repeat. System prompts, tool definitions, RAG scaffolding, and conversation history often create huge shared prefixes. Prefix caching lets subsequent requests reuse the KV computation instead of recomputing it. GitHub This is one of the best optimizations because it can lower cost without changing model quality or generation latency.
3. Quantize weights—and potentially the KV cache. Move from FP16/BF16 toward FP8, INT8, or 4-bit where quality permits. The benefit isn't merely cheaper memory: smaller weights can let you fit more concurrent sequences, improving GPU utilization. Google specifically identifies quantization as a major inference-cost lever. Google Cloud For example, current vLLM supports FP8 KV-cache storage as well as several lower-precision formats. vLLM I'd test FP8 first if latency/quality are sacred; it's usually a much less disruptive step than jumping directly to 4-bit.
4. Right-size the model per request. Don't send every query to your largest model:

• small/clear requests → 3B–8B model
• difficult reasoning/tool-use → larger model
• fallback to the large model when the small model is uncertain

This can dwarf micro-optimizations because model FLOPs dominate GPU cost. 5. Reduce tokens, especially output tokens. Put a hard but sensible output-token budget on each endpoint. Streaming can make latency feel low, but you're still paying for every generated token. Also consider concise system prompts and structured outputs where applicable. 6. Fix GPU under-utilization before scaling out. Look for:

• GPU utilization with lots of idle gaps
• low tokens/sec/GPU
• KV-cache exhaustion and request preemption
• excessive CPU/tokenizer overhead
• synchronization caused by excessive tensor parallelism

In vLLM, frequent KV-cache preemption causes recomputation and hurts latency; increasing available KV capacity or reducing excessive concurrency can help.

A practical order of attack

I'd run the optimization program in this order:

Measure → prefix cache → continuous batching → FP8 → model routing → token reduction → GPU/parallelism right-sizing.

For every change, track $/1M input tokens, $/1M output tokens, p50/p95 TTFT, p95 inter-token latency, tokens/sec/GPU, and quality. Don't optimize solely for GPU utilization—95% utilization with a blown p95 is not an optimization.

A particularly useful metric is:

GPU-seconds per successful request at your latency SLO That gives you one number that captures both cost and performance.

If you tell me model(s), GPU type/count, average input/output tokens, requests/sec, and your p95 latency target, I can map this into a concrete cost-cutting plan and estimate where the biggest savings are likely to come from.

To cut inference bills without hurting latency, you must optimize memory bandwidth, eliminate redundant token processing, and right-size your models . Because generation speed is bound by memory traffic rather than raw compute, smart software configuration often saves up to 70% of costs while keeping response times fast.

Core Strategies to Cut Costs

• Semantic Caching: Store common queries and their responses in a fast data layer like Redis using vector embeddings. Similar incoming questions bypass the LLM entirely, cutting token volume instantly with zero added latency.
• Prompt Caching: Enable built-in provider features like Anthropic Prompt Caching or OpenAI Prompt Caching . This reuses processed system prompts and long context prefixes, cutting costs for repeated context by up to 50% to 90% without slowing down time-to-first-token.
• Prompt Compression: Use tools like LLMLingua to strip redundant words and tokens from large system instructions dynamically. Smaller inputs mean faster prefill phases and cheaper processing.
• Model Routing: Route simple tasks (classification, extraction, basic formatting) to smaller, faster, and cheaper models like GPT-4o-mini or smaller open-source alternatives. Reserve expensive frontier models strictly for complex reasoning steps.
• Inference Engine Optimization: Deploy high-performance serving frameworks such as vLLM or TensorRT-LLM . These engines use PagedAttention to manage the Key-Value (KV) cache efficiently, drastically increasing throughput and hardware utilization without impacting per-request latency.

If you share your current model stack (e.g., self-hosted Llama vs. API providers) and average traffic patterns , I can recommend the highest-impact optimization to start with.

If GPU spend is exploding while latency SLOs are already tight, the goal is usually higher GPU utilization and fewer memory movements, not simply “use smaller GPUs.” The biggest wins tend to come from the serving stack.

1. Fix underutilized GPUs first

A lot of inference fleets are expensive because GPUs sit idle between requests.

• Enable continuous/in-flight batching rather than static batching. - Static batching waits for a batch to fill or finish.
• Continuous batching keeps the GPU fed by inserting new requests as others complete.
• This often improves throughput without materially increasing user-visible latency when tuned correctly.

Measure:

• GPU utilization
• tokens/sec/GPU
• p50/p95 time-to-first-token (TTFT)
• p95 inter-token latency

If GPUs are below ~60–70% utilization during peak traffic, start here.

2. Quantize the model (carefully)

Move from FP16/BF16 where possible:

• FP8: often a good first step for production inference.
• INT8: common compromise.
• INT4 weight-only: bigger savings, but validate quality.

The reason this helps latency: LLM decoding is often memory-bandwidth bound. Smaller weights mean less data moved per generated token.

A practical rollout:

1. Benchmark FP16 baseline.
2. Try FP8.
3. Run eval suite + production shadow traffic.
4. Only then try INT8/INT4.

3. Attack KV-cache costs

For LLMs, memory pressure often comes from KV cache, especially with long contexts.

Options:

• Paged KV cache / paged attention
• KV cache reuse
• Prefix caching for repeated system prompts, tool definitions, or RAG prefixes

These can improve both throughput and latency by avoiding repeated work and reducing memory fragmentation.

High-impact candidates:

• Agents with large system prompts
• Chat apps where users share the same instructions
• RAG systems with repeated retrieved context

4. Use speculative decoding for latency-sensitive paths

If your bottleneck is decode latency:

• A small draft model proposes tokens.
• The larger model verifies them in fewer sequential steps.

This can reduce generation latency without changing the final model behavior, although it needs workload-specific tuning.

Good fit:

• Interactive chat
• Code completion
• Short-to-medium responses

Less useful:

• Very high batch throughput jobs where the GPU is already saturated.

5. Route requests intelligently

Avoid using your most expensive model for every request.

Common patterns:

• Small model → classification, extraction, simple Q&A
• Large model → reasoning-heavy tasks
• Escalation only when confidence is low

A 70B model answering 80% easy requests is usually a routing problem, not a GPU problem.

6. Separate latency classes

Don't make one cluster serve everything.

Example:

Interactive pool

• Low batch size
• Aggressive latency SLOs
• Speculative decoding
• Higher priority scheduling

Batch pool

• Larger batches
• Maximum throughput
• Lower cost/token

Mixing these workloads often causes expensive overprovisioning.

7. Tune the serving engine

If you are running custom inference, compare against optimized runtimes:

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

They implement many of the difficult pieces: batching, KV-cache management, optimized kernels, and scheduling.

A realistic optimization order

I would usually do:

1. Measure bottleneck (prefill vs decode, compute vs memory)
2. Enable continuous batching
3. Add KV-cache/prefix caching
4. Move FP16 → FP8
5. Right-size models with routing
6. Add speculative decoding
7. Consider architecture changes (prefill/decode separation, hardware changes)

The fastest savings usually come from making the same GPUs serve more tokens, not from buying cheaper GPUs.

If you share:

• model size (7B/70B/etc.)
• GPU type (A10/A100/H100/L40S/etc.)
• current QPS/concurrency
• p95 latency target
• serving stack (vLLM, TGI, TensorRT-LLM, custom)

…I can suggest the highest-ROI changes in order.

When you are burning cash on GPU bills, the culprit is almost always idle silicon, memory fragmentation , or redundant token processing . Naive serving stacks keep expensive GPUs underutilized while processing requests sequentially, meaning you pay for capacity you aren't using.

You can slash your infrastructure spend without degrading Time-to-First-Token (TTFT) or Inter-Token Latency (Throughput) by implementing these production-tested optimizations in order of highest ROI:

• Switch to High-Performance Serving Engines: Ditch naive Hugging Face pipelines or standard Flask/FastAPI wrappers. Deploy models using production engines like vLLM or SGLang . They feature PagedAttention (which eliminates KV-cache memory fragmentation) and continuous/in-flight batching (which packs incoming requests dynamically at the iteration level). This safely multiplies your aggregate throughput per GPU, dropping your cost per token drastically.
• Implement Prompt / Prefix Caching: If your application uses long system prompts, multi-turn chat history, or recurring RAG contexts, enable prefix caching . Engines like vLLM cache the computed Key-Value (KV) tensors of the static prompt portion. When a new request arrives with the same prefix, it skips the expensive prefill phase entirely, slashing your TTFT and compute overhead for input-heavy workloads by up to 8 0%.
• Apply Modern Quantization (FP8 / INT4): Move away from unquantized FP16/BF16 weights. Moving to modern formats like FP8 quantization (natively supported on modern NVIDIA architectures via TensorRT-LLM ) shrinks your VRAM footprint with virtually zero degradation in model intelligence. A smaller VRAM footprint allows you to pack larger batch sizes or run on smaller, cheaper GPU instances.
• Deploy Intelligent Model Routing (Cascading): Do not send every simple extraction, classification, or formatting task to your expensive 70B+ parameter model. Implement a gateway router that classifies incoming queries. Route straightforward tasks to a smaller, hyper-fast model (like an 8B model or specialized fast APIs) and only escalate to your heavy model when complex reasoning is required. This frequently yields a 3 0% to 5 0% aggregate drop in cost.
• Prune and Shape Inputs/Outputs: Output tokens cost significantly more in compute time and provider pricing than input tokens. Use aggressive response length caps, structured output constraints, and tools like LLMLingua for context compression on agentic traces to strip out low-signal tokens before they hit the model forward pass.

To help tailor this to your stack, tell me:

• Are you self-hosting on cloud GPUs (AWS/GCP/RunPod) or calling managed APIs (OpenAI, Anthropic)?
• What model size and inference framework (vLLM, TGI, Triton) are you currently running?

Yes. The key is to optimize cost per successful request at your latency SLO, not raw GPU utilization. The biggest wins usually come from reducing tokens/GPU-seconds while keeping the tail latency budget intact.

Where I’d attack first

1. Right-size the model per request

   • Route easy tasks to a smaller model and reserve the large model for requests that actually need it.
   • Consider distillation or a smaller fine-tuned model for high-volume workloads.
   • Put a hard ceiling on max_output_tokens; runaway generations are one of the easiest hidden cost multipliers.

2. Use continuous/in-flight batching

   • Don't wait to form conventional batches. Continuous batching lets new requests enter while others are decoding, substantially improving GPU utilization without imposing a full batch's waiting time.
   • vLLM's scheduler, for example, explicitly prioritizes decode work and uses chunked prefill to balance TTFT against inter-token latency.
   • If you're using NVIDIA's stack, TensorRT-LLM similarly provides in-flight batching and KV-cache optimizations.

3. Quantize aggressively—but measure quality

   • Start with FP8 for weights/activations where your hardware supports it; move to INT8/INT4/AWQ/GPTQ where the quality tradeoff is acceptable.
   • For long-context workloads, FP8 KV cache is particularly interesting: it halves KV-cache storage and memory traffic, potentially increasing concurrency and reducing decode latency.
   • Don't benchmark only tokens/sec. Compare p50/p95 TTFT, p95/p99 inter-token latency, quality, and $/request.

4. Attack your KV cache

   • Long prompts can make inference memory-bound rather than compute-bound.
   • Enable prefix caching when users repeatedly share system prompts, retrieved documents, tool definitions, etc.
   • Keep an eye on KV-cache exhaustion. vLLM notes that preemption causes recomputation and can hurt end-to-end latency; increasing available KV capacity or reducing excessive concurrency can help.
   • If GPU KV memory is your bottleneck, KV offloading to CPU memory can avoid recomputation during preemption.

5. Use speculative decoding selectively

   • This is particularly useful when you're latency-bound at low/medium QPS rather than saturating the GPU.
   • A small draft model proposes tokens and the large model verifies several at once, reducing sequential decode steps.
   • vLLM specifically positions speculation for medium/low-QPS, memory-bound workloads.
   • Don't blindly enable it at high batch sizes—the extra drafting work can actually increase cost.

6. Separate TTFT from decode optimization

   Treat these as two different problems:

   High TTFT: optimize prefill, prompt length, prefix caching, chunked prefill, and batching.

   High ITL: optimize KV bandwidth, quantization, speculative decoding, GPU choice, and decode scheduling.

   vLLM's tuning guidance makes this distinction explicit: smaller max_num_batched_tokens tends to favor inter-token latency, while larger values favor TTFT/throughput.

The cost metric I'd put on the dashboard

Instead of:

GPU utilization = 72%

track:

$/1M input tokens
$/1M output tokens
$/successful request
GPU-seconds/request
p95 TTFT
p95 ITL
p99 end-to-end latency
quality/pass rate

Then run an A/B matrix:

My practical order: instrument → cap wasteful output → continuous batching → prefix/KV caching → FP8 → model routing → speculative decoding → deeper quantization/distillation.

The important trick is to optimize each traffic class separately. A chat workload with short prompts and a long-context RAG workload can have completely different cost/latency bottlenecks, so one global GPU configuration often leaves a lot of money on the table.

Cutting inference costs without hurting latency requires shifting from naive model hosting to an optimized inference stack . Because GPU generation is memory-bandwidth bound rather than compute-bound, hardware often sits idle waiting to fetch weights and context . Implementing a combination of KV caching, continuous batching, model quantization, and smart routing can dramatically slash your token spend while maintaining or even improving response times.

1. Implement Semantic Caching and Prefix Caching

Users frequently submit overlapping queries or identical system prompts, forcing the GPU to redundantly re-process context tokens.

• Semantic Caching: Use Amazon ElastiCache (Valkey) or Redis with vector embeddings to intercept semantically similar requests before they ever touch your model, cutting redundant generation.
• Prefix/KV Caching: Switch to a serving engine that supports automatic KV cache reuse across shared system prompts or multi-turn chat prefixes. This dramatically accelerates Time-to-First-Token (TTFT) for recurring contexts.

2. Upgrade Your Serving Engine and Enable Continuous Batching

Running raw Hugging Face transformers in production leaves expensive silicon underutilized because requests are handled sequentially.

• Optimized Engines: Deploy high-performance engines like vLLM or SGLang.
• Continuous Batching: These engines use dynamic batching, packing new incoming requests into unused GPU memory slots on the fly rather than waiting for rigid static batches. This maximizes GPU throughput and lowers average queue wait times.

3. Apply FP8 or INT8 Quantization

Dropping precision from FP16 down to FP8 reduces the VRAM footprint of both model weights and the KV cache by half.

• Hardware-Native Support: Modern accelerators (such as NVIDIA Hopper/Blackwell or AMD Instinct MI325X ) natively process FP8 precision with virtually zero degradation in output quality or accuracy.
• The Payoff: Halving the memory footprint doubles your concurrent user capacity per GPU and speeds up memory-bandwidth-bound token generation.

4. Deploy Intelligent Model Cascading / Routing

Not every query requires a massive frontier model.

• Smart Routers: Route simple, short, or low-complexity prompts (e.g., basic classification, formatting, or straightforward lookups) to a smaller, fine-tuned 8B or 70B model, reserving your most expensive frontier endpoints strictly for complex reasoning tasks.
• The Frugal Approach: Studies demonstrate that cascading or routing simple queries to cheaper models can reduce total token expenditure by up to 85% while preserving ~95% cumulative aggregate quality.

The KV cache optimization section is gold. One extension that's extremely relevant for multi-agent workflows: when you have an orchestration pipeline where multiple agents share similar system prompts or context prefixes, you can amortize that KV cache across agents using prefix caching.

Most developers are burning GPU cycles or API budget on redundant computations. Every time an LLM generates a token, attention blocks re-processes the entire conversation history. This means your system prompt needs to be processed on each new user query. KV caching solves this.

If you share your current model size, average input/output token lengths , and whether you're self-hosting or using commercial APIs , I can pinpoint which of these levers will yield the highest cost reduction for your setup.

Yes. The key is to optimize $/token while holding your latency SLO fixed, rather than simply trying to make each request cheaper.

Highest-impact moves

1. Quantize first

   • Try FP8 before more aggressive INT4/FP4.
   • Quantization reduces memory traffic and lets you fit more work per GPU. On H100-class hardware, NVIDIA reports substantial throughput and latency improvements from FP8, though you need to validate quality on your workload.
   • Quantize the KV cache too if long prompts/conversations are driving memory pressure; NVIDIA's example saw throughput rise from ~3,390 to ~5,300 tokens/sec with FP8 KV cache while TTFT stayed roughly flat.

2. Use continuous/in-flight batching

   • Don't wait for a conventional batch to fill. Admit requests at the token/iteration level so generation from one request can overlap with prefill from another.
   • This generally improves GPU utilization without deliberately adding batching delay, making it one of the best cost/latency trades.

3. Attack prompt and output tokens

   • Cache repeated prefixes/system prompts.
   • Remove unnecessary context and retrieved documents.
   • Put hard output-token limits on endpoints where long answers aren't useful.
   • Measure cost per useful response, not just GPU utilization. A 20% reduction in generated tokens can beat a lot of kernel tuning.

4. Add speculative decoding for low-batch traffic

   • A small draft model proposes several tokens and the expensive model verifies them together.
   • It's particularly useful when requests are latency-sensitive and batches are small; it can reduce sequential decode steps without changing the target model's output distribution.
   • Don't blindly enable it at high concurrency—its benefit depends heavily on draft-token acceptance and workload characteristics.

5. Stop over-provisioning for peak

   • Separate traffic by latency class:
     • interactive: aggressively optimized, dedicated capacity
     • normal: continuous batching
     • offline/bulk: larger batches, cheaper GPUs/instances
   • If your GPU is <~50–60% utilized during significant portions of the day, you're probably paying for latency headroom rather than inference.

6. Tune the serving runtime

   • Use an optimized engine such as TensorRT-LLM/vLLM rather than raw framework inference.
   • CUDA graphs, kernel fusion, paged KV management, scheduling and CPU/GPU overlap can eliminate surprisingly large amounts of overhead. TensorRT-LLM, for example, documents CUDA-graph and overlap-scheduler optimizations specifically for reducing execution overhead and GPU idle time.

What I'd do in order

The metric I'd put on the dashboard

Track this by endpoint and model:

GPU $ / 1M output tokens
alongside p50/p95 TTFT, p50/p95 inter-token latency, tokens/sec/GPU, GPU utilization, and quality/error rate.

Then run changes against a fixed latency budget—for example, p95 TTFT ≤ 300 ms and p95 end-to-end ≤ 2 s. If an optimization cuts GPU cost 35% while staying inside those bounds, ship it.

If you tell me which models/GPUs you're running (e.g. Llama 70B on H100s), your requests/sec, average input/output tokens, and current p95 latency, I can sketch a concrete cost-cutting architecture and estimate where the biggest savings are likely to come from.

Cutting your GPU inference bill while protecting (or even improving) latency requires moving away from naive model serving and optimizing the entire stack—from how requests are routed to how memory is managed on the silicon.

The most effective, production-proven strategies to slash costs without sacrificing speed include:

• Implement Intelligent Model Routing
  The single biggest waste of compute is sending every simple classification, extraction, or basic chat query to a heavy, expensive frontier model. Set up a lightweight router to send routine traffic to smaller, highly distilled models (like smaller open-weights models or efficient endpoints), reserving your heavy infrastructure strictly for complex reasoning tasks. This routinely cuts aggregate costs by 4 0%--5 0% while speeding up average response times.
• Turn on Prompt and Prefix Caching
  If your application uses heavy system prompts, multi-turn conversation histories, or long tool/RAG definitions, you are repeatedly paying to compute the exact same Key-Value (KV) states on the GPU. Leverage native prompt caching features (offered by major providers and modern serving engines) to skip the prefill phase entirely for repeated prefixes. This drops Time-to-First-Token (TTFT) dramatically and slashes costs for repetitive context.
• Upgrade to Production-Grade Serving Engines
  If you are running standard Hugging Face pipelines or unoptimized wrappers, your GPUs are sitting idle waiting for sequential tokens. Swap your runtime to high-performance inference engines like vLLM or SGLang . Features like continuous batching and paged attention maximize hardware utilization, allowing a single GPU cluster to handle significantly more concurrent throughput.
• Apply Aggressive Quantization (INT8 / INT4 / FP8)
  Moving 16-bit weights (B F 16 cap B cap F 16𝐵𝐹1 6 or F P 16 cap F cap P 16𝐹𝑃1 6 ) through memory for every token creates a massive memory-bandwidth bottleneck. Compressing models down to 8-bit or 4-bit precision—or utilizing native F P 8 cap F cap P 8𝐹𝑃8 quantization if you are on modern enterprise architectures—cuts your VRAM footprint in half or quarters. Smaller memory footprints mean data moves faster through the chip, improving inter-token latency while letting you pack larger or multiple models into cheaper GPU instances.
• Deploy Speculative Decoding
  To accelerate generation latency on self-hosted setups, use a small, fast "draft" model to guess a few upcoming tokens, and let your primary "verifier" model check them in a single parallel forward pass. This yields a 1.5×--3× speedup on generation latency without compromising output accuracy, effectively lowering your compute time per useful token.
• Layer in Semantic Caching
  For high-volume, repetitive user bases (like support bots or common queries), implement a semantic cache (using vector search databases or gateways). Instead of an exact string match, it checks if a incoming query's meaning matches a past response. If it's a match, it returns the result in milliseconds, completely bypassing the LLM inference call.

To help narrow down the fastest wins for your stack, tell me:

• Are you self-hosting on raw GPUs or calling third-party APIs?
• What serving framework or engine are you currently using?

Yes. The key is to optimize GPU-seconds per successful request, not just tokens/sec. You can usually cut inference spend substantially while holding p95/p99 latency roughly flat by attacking the biggest sources of wasted GPU capacity in this order:

1. Quantize first

If you're running BF16/FP16, benchmark FP8 before changing architecture. TensorRT-LLM supports FP8/FP4 plus quantized KV caches, and lower precision can reduce both memory pressure and compute cost.

For latency-sensitive production, I'd test:

BF16 → FP8 weights/activations → FP8 KV cache

rather than jumping immediately to aggressive INT4. The goal is to fit more concurrent sequences on each GPU without introducing a quality or latency regression.

2. Get much better utilization with continuous batching

If you're doing one request → one GPU execution, you're probably leaving enormous capacity unused.

Use continuous/in-flight batching so new requests enter as others finish instead of waiting for a fixed batch boundary. TensorRT-LLM explicitly uses this approach to improve GPU utilization while maintaining low latency.

The important tuning variables are:

• maximum concurrent sequences
• maximum batched tokens
• prefill/decode scheduling
• queueing timeout
• GPU memory utilization

Don't maximize batch size blindly: once queueing delay starts contributing materially to p95, you've gone too far.

3. Attack repeated prompts with prefix/KV caching

If your workload has common system prompts, RAG instructions, tool definitions, or conversation prefixes, prefix caching can eliminate redundant prompt computation. vLLM's implementation reuses KV-cache blocks for identical prefixes.

This is particularly attractive because it can reduce cost without changing the model or output quality.

Measure:

prefix_cache_hit_rate × cached_tokens/request

If that number is high, prioritize this ahead of more exotic optimizations.

4. Use speculative decoding selectively

Speculative decoding can reduce inter-token latency by having a cheaper model/proposer predict tokens and the expensive model verify them. Current vLLM documentation specifically positions it for medium/low-QPS, memory-bound workloads.

It's not automatically a cost win: you're adding draft-model computation. But if your expensive model is the bottleneck and acceptance rates are high, you can get more useful output tokens per unit of GPU time.

A good experiment:

baseline → add speculative decoding → compare GPU-ms / generated token and p95 TTFT/TPOT

rather than looking only at tokens/sec.

5. Reduce context before reducing model size

Long prompts are a silent GPU bill multiplier.

Look at:

• duplicated conversation history
• unnecessarily large RAG top-k
• verbose tool schemas
• repeated system instructions
• retrieved chunks that never affect the answer
• maximum context configured much higher than actual need

A 30% reduction in input tokens can be more valuable than squeezing another 5% out of kernels—especially for workloads dominated by prefill.

6. Right-size the model per request

If you're serving everything with your largest model, introduce a model router:

easy request → small/cheap model
ambiguous/high-value request → large model

You can make routing progressively conservative so the latency-sensitive path stays essentially unchanged.

For many applications, this is the biggest cost lever because it reduces actual compute rather than merely making the same compute more efficient.

7. Don't over-provision GPUs

Once the above is working, measure utilization against your actual latency SLO.

You want to find the point where:

GPU utilization ↑ → cost/request ↓

until:

queueing ↑ → p95/p99 latency violates SLO

Then scale horizontally only when that knee is reached.

---

What I'd do in the next 2 weeks

Week 1 — measurement + easy wins

1. Instrument TTFT, inter-token latency, output tokens, input tokens, queue time, GPU utilization and GPU-hours/request.
2. Separate prefill and decode cost.
3. Enable continuous batching.
4. Add prefix caching.
5. Benchmark FP8.
6. Establish a cost/latency baseline for each traffic class.

Week 2 — bigger levers

7. Test speculative decoding on decode-heavy traffic.
8. Test smaller models/router fallbacks.
9. Tune max concurrency and batching against your p95/p99 SLO.
10. Load-test at 50%, 75%, 100% and 125% of expected peak.

The metric I'd put on the dashboard is $/1M successful output tokens at a fixed p95 latency, not simply GPU utilization.

One especially useful rule: if you're memory-bound, prioritize quantization/KV-cache efficiency; if you're compute-bound, prioritize batching/kernel optimization; if you're queue-bound, stop increasing batching and add capacity. KV cache in particular grows with both concurrent requests and context length, so it frequently becomes the hidden constraint.

If you tell me your model, GPU type, average input/output tokens, QPS, and current p50/p95 latency, I can help identify which 2–3 changes are most likely to cut your bill without moving the latency SLO.