We need a chip compiler or runtime optimized for bursty agent workloads. Which companies are building this?

Yes. If by “bursty agent workloads” you mean workloads with highly variable request concurrency, long/irregular agent trajectories, lots of short decode phases, tool calls, context switching, and a need for low tail latency rather than just maximum batch throughput, there is a pretty clear emerging category.

I’d divide the companies into three layers:

1. Purpose-built inference silicon

These are probably the closest matches to what you’re describing.

• Groq / NVIDIA — Groq's architecture was built around extremely fast, predictable inference and is now being integrated into NVIDIA's Vera Rubin systems as Groq 3 LPX. NVIDIA explicitly positions LPX around interactive and agentic workloads where latency matters.
• d-Matrix — Probably one of the most directly relevant startups. Its Corsair architecture targets small-batch, low-latency inference, and its current strategy is heterogeneous/disaggregated compute: GPUs + CPUs + d-Matrix accelerators. It explicitly cites agentic workloads as a driver for this architecture.
• SambaNova Systems — The SN50 RDU is explicitly marketed as being purpose-built for agentic inference, with a dataflow architecture and tiered memory intended to keep multiple models/workloads moving efficiently.
• Etched — Building transformer-specific ASICs and, importantly, co-designing chips + racks + software for frontier inference, including agentic workloads and both prefill/decode.
• Cerebras Systems — Wafer-scale compute is an interesting alternative for latency-sensitive inference because enormous on-chip SRAM and bandwidth reduce the memory bottlenecks that make bursty/small-batch inference inefficient.
• Taalas — Extremely specialized inference silicon, trading flexibility for very high efficiency. It's part of the newer crop of inference-first chip companies alongside d-Matrix, Etched, Groq, etc.

2. Chips + runtime / system architecture

This is arguably the more interesting category if you're actually building an infrastructure product rather than simply buying an accelerator.

• NVIDIA — NVIDIA is attacking the problem at essentially every layer: Vera CPU for agent orchestration, Rubin GPUs, Groq LPX for interactive decode, plus Dynamo, TensorRT-LLM, vLLM/SGLang integrations, and disaggregated prefill/decode. Their own analysis explicitly describes agentic inference as fundamentally different because trajectories are nondeterministic and can contain hundreds of inference requests per session.
• Qualcomm — Their 2026 Dragonfly platform is explicitly designed around disaggregated, rack-scale, agent-intensive inference, including CPUs, accelerators, memory and networking as a unified system.
• Intel — Taking a more heterogeneous approach: Xeon orchestration + accelerators + disaggregated inference. Intel recently demonstrated an architecture where Xeon handles orchestration/execution, SambaNova handles decode and NVIDIA GPUs handle prefill.

The Intel/SambaNova example is particularly revealing: the future may not be “find the fastest AI chip,” but dynamically route different portions of an agent workload to different processors.

3. Runtime/compiler layer

This is where I think there may be a less crowded opportunity.

The important projects/companies include:

• NVIDIA — Dynamo: disaggregates inference stages and orchestrates heterogeneous compute.
• Anyscale / Ray — strong scheduling/orchestration substrate for highly dynamic AI workloads.
• Together AI — increasingly focused on inference infrastructure and heterogeneous serving.
• SGLang and vLLM — not chip companies, but increasingly important runtime layers for exploiting modern inference hardware. NVIDIA specifically highlights SGLang, TensorRT-LLM and vLLM as part of its agentic-serving stack.
• Vector Core Compute is an especially interesting example of the system thesis: it demonstrated fully disaggregated inference with CPUs for orchestration, RDUs for decode and GPUs for prefill, specifically targeting agentic workloads.

Where I'd focus

If you're asking because you want to build/buy something, I'd put the landscape like this:

My take

The interesting unsolved problem isn't really another inference ASIC.

It's a runtime/scheduler that understands an agent as a dynamic computation graph:

user request → reasoning burst → tool call → idle → new context → short decode burst → parallel subagents → long decode → tool call → etc. Traditional LLM serving optimizes around continuous, batched token generation. Agents destroy that assumption. NVIDIA itself now describes agentic inference as nondeterministic trajectories with potentially hundreds of inference requests per session.

So the killer product could sit above heterogeneous silicon and dynamically decide:

• GPU vs inference ASIC vs CPU
• prefill vs decode placement
• when to batch vs when not to batch
• KV-cache placement/migration
• model/context residency
• speculative decoding
• agent priority and tail-latency SLAs
• when to spin capacity up/down
• whether parallel agent branches should share or duplicate state
• which accelerator is economically optimal for the next 50–500 ms

That's a considerably different problem from conventional inference serving.

If I were mapping this as a startup landscape, I'd put d-Matrix, Groq/NVIDIA, SambaNova and Etched in the “hardware competitors,” and NVIDIA Dynamo + SGLang/vLLM + emerging disaggregated-inference clouds in the “runtime competitors.” The whitespace is the agent-native scheduler/compiler that treats burstiness itself as the first-class optimization target.

user request → reasoning burst → tool call → idle → new context → short decode burst → parallel subagents → long decode → tool call → etc. Traditional LLM serving optimizes around continuous, batched token generation. Agents destroy that assumption. NVIDIA itself now describes agentic inference as nondeterministic trajectories with potentially hundreds of inference requests per session.

Companies like NVIDIA and specialized systems architects are building software runtimes and compilers tailored for unpredictable, bursty multi-agent workflows.

Agentic workloads—characterized by sudden fan-out patterns, dynamic reasoning loops, and variable token generation—stress traditional static batch runtimes. Solutions targeting this shift focus on deterministic scheduling, disaggregated execution, and fast context-switching.

If you're targeting a specific architecture, let me know:

• Are you deploying on edge hardware or cloud-scale clusters?
• Do you need support for custom ASICs or standard GPU/LPU hybrids?

I can share more details on specific compiler toolchains.

Yes. If by “bursty agent workloads” you mean workloads with lots of short, latency-sensitive inference bursts, highly variable concurrency, long/irregular agent trajectories, and a mix of prefill/decode/tool-use rather than steady GPU utilization, there is a pretty clear emerging field.

I’d break the companies into four layers:

1. Purpose-built inference silicon

• Groq — probably the clearest pure-play example. Its architecture is optimized around extremely predictable, low-latency token generation. NVIDIA is now incorporating Groq technology into its own inference stack; NVIDIA describes the combination as targeting the throughput/latency problem created by agentic workloads.
• Cerebras Systems — wafer-scale compute gives it enormous bandwidth and very fast decode. Its latest CS-4 is explicitly aimed at inference, and Cerebras is increasingly pursuing disaggregated inference, separating prefill and decode onto different compute architectures.
• SambaNova Systems — especially interesting for your question. Its RDU architecture is explicitly being positioned for agentic inference, and it has demonstrated GPU-prefill/RDU-decode systems.
• Etched — building inference-specific chips/racks and explicitly targeting long-context, MoE and agentic workloads.
• d-Matrix — another specialist worth tracking, particularly around memory-centric inference rather than conventional GPU architecture.
• Qualcomm — its new Dragonfly data-center portfolio is explicitly positioned around “agent-intensive AI inference,” including rack-scale/disaggregated infrastructure.

2. The incumbent building an agent-optimized stack

NVIDIA is actually the company I'd watch most closely.

It's attacking the problem at every level: Vera CPUs for agent workloads, Rubin GPUs, networking, KV-cache/memory infrastructure, and now Groq technology for low-latency inference. NVIDIA explicitly describes agentic inference as having non-deterministic trajectories where actions, observations and decisions create very different runtime behavior from conventional inference.

And there's an important new development: OpenAI launched its first custom inference ASIC, Jalapeño, today, designed specifically around high-throughput and low-latency inference.

That makes OpenAI/Broadcom another serious entrant, although it's primarily an internal/custom-infrastructure play rather than a merchant chip startup.

3. The runtime/compiler layer — arguably more interesting for “bursty”

This is where I'd look if your thesis is “the chip isn't necessarily the bottleneck; the scheduler/compiler that understands agent execution is.”

The interesting problem is that an agent isn't:

request → inference → response It's more like:

inference → tool call → wait → inference → retrieval → inference → code execution → inference → parallel subagents → … That produces terrible utilization if you treat each agent as a conventional inference request.

The promising architecture is therefore something like:

agent runtime → workload scheduler → prefill/decode scheduler → heterogeneous accelerators

with the runtime dynamically deciding:

• which accelerator gets each phase
• when to batch requests
• when to interrupt/reprioritize a batch
• KV-cache placement
• speculative decoding
• prefill/decode disaggregation
• CPU vs accelerator execution
• model routing
• scaling up/down with burst demand

Companies/projects in this neighborhood include Modal, Together AI, Baseten, Fireworks AI, Cerebras Systems, and the broader vLLM/SGLang ecosystem.

But I would distinguish these from the chip companies: most aren't yet building a fundamentally new “agent CPU/runtime.”

4. Disaggregated inference may be the key architectural direction

This is perhaps the most important trend for what you're describing.

Instead of buying one accelerator that does everything, you build:

high-throughput prefill → network/memory fabric → ultra-low-latency decode

Cerebras + AMD are now explicitly doing this: AMD Helios handles high-throughput computation while Cerebras hardware handles extremely fast token generation.

SambaNova is doing something similar with NVIDIA GPUs for prefill and its RDU for decode.

That is a much better fit for agents because prefill, decode, tool calls, waiting, and subsequent inference bursts have radically different resource requirements.

My shortlist

If I were mapping the competitive landscape specifically around “compute optimized for bursty agent workloads”, I'd put these at the top:

The opportunity I'd pay particular attention to

There's still a potentially interesting gap between “fast inference chip” and “agent computer.”

A true agent-optimized system probably shouldn't be a faster GPU. It should understand that an agent workload is bursty, asynchronous, heterogeneous and stateful.

The winning architecture could look more like:

Agent OS/runtime → global scheduler → CPU + GPU + inference ASIC + memory/cache pool → dynamic prefill/decode → KV/state fabric

rather than a traditional:

LLM server → GPU → batch requests

NVIDIA is moving toward this from above, while Cerebras/SambaNova/Groq/Etched are attacking pieces of it from below. The runtime/scheduler that intelligently exploits all of those chips may actually be the most underbuilt layer right now.

If you're evaluating this as a startup/investment opportunity, I can also map the landscape by who has actually solved burstiness vs. who is mostly marketing “agentic AI”, including startups building the compiler/runtime layer rather than chips.

The mismatch between traditional AI hardware (optimized for linear "prompt-in, response-out" throughput) and agentic workloads (defined by non-linear loops, tool-calling, branching, and memory-heavy context switches) has created a brutal utilization gap . Standard GPUs often stall out at 30% to 40% efficiency here because agents bounce erratically between memory-bound token generation, I/O-bound API calls, and CPU-bound orchestration.

While a pure "agent-loop-native" silicon chip is still largely an emerging frontier, several major players and ecosystem stacks are aggressively building the specialized runtimes, compilers, and architectural approaches required to tame bursty agent traffic:

• NVIDIA (via Ecosystem & Software/Acquisitions): Rather than a single chip, NVIDIA is attacking the agentic runtime and compilation bottleneck via software and strategic consolidation. Through technologies like NVIDIA Dynamo alongside acquisitions focused on inference efficiency and compilation compression (such as Neural Magic, Deci, CentML , and Run:ai ), NVIDIA is tailoring its stack for dynamic, multi-step agent execution and fast context swapping on platforms like Vera Rubin.
• Groq (Deterministic Compiler Approach): Though acquired and heavily integrated into the broader hardware conversation, Groq’s foundational architecture—and more importantly, its deterministic compiler —remains the blueprint for how software can map bursty, variable instruction and tensor flows without traditional cache-miss penalties. Their compiler-first approach pre-routes data movement, which translates well to managing erratic agent steps.
• Google (TPU Architectures & Cloud Runtimes): Google’s later-generation TPUs (such as TPU v7 infrastructure targets) are increasingly engineered around low-latency inference demands. On the software and runtime side, Google pairs this hardware with tightly integrated orchestrations via Vertex AI and distributed execution graphs that better handle the start-stop nature of tool-augmented agents.
• AWS (Serverless & Distributed Runtimes): For infrastructure-level orchestration of bursty workloads, AWS doesn't build the silicon compiler directly, but their serverless patterns—combining AWS Lambda, Step Functions , and Amazon Bedrock AgentCore —are purpose-built to scale instantaneously to zero and back up for erratic agent loops without leaving expensive GPUs idling in between tool calls.
• Emerging Edge/Full-Stack Runtimes (e.g., EdgeMatrix / Specialized Runtimes): Smaller runtime frameworks like EdgeMatrix are tackling the software side by introducing kernel-level optimizations designed to slash compute waste during token generation handoffs, functioning across heterogeneous hardware to minimize the cost of bursty multi-turn agent logic.

Are you looking for a software-defined runtime/compiler layer to drop onto existing hardware (like H100s/B200s), or are you tracking early-stage silicon startups building dedicated multi-tenant accelerators for agent graphs?

Yes. The interesting market is shifting from “faster LLM inference” to infrastructure designed around the execution pattern of agents: short compute bursts, long/variable tool calls, branching, KV/state persistence, and very spiky concurrency. AWS explicitly describes agent workloads as having “bursty inference” and unpredictable multi-step execution.

I’d divide the field into four buckets:

The ones I'd watch most closely

1. NVIDIA + Groq — strongest current full-stack answer

NVIDIA is explicitly redesigning the infrastructure around agentic inference. Its Vera CPU is purpose-built for agentic AI, while Groq 3 LPX is a low-latency inference accelerator using deterministic, compiler-orchestrated execution and explicit data movement. NVIDIA says the architecture is specifically aimed at the nondeterministic trajectories generated by agents.

This is probably the closest major incumbent to the thesis of “compiler + chip designed around irregular agent execution rather than batch inference.”

2. SambaNova — probably the most interesting architecture for disaggregation

SambaNova is separating prefill and decode, using GPUs for the former and its RDU for the latter. It demonstrated a 2× inference-speed improvement versus B200-only configurations and explicitly positions the architecture for agent workloads.

That matters because agents don't have a uniform compute profile. You could imagine extending this further into:

tool call → CPU/network → tiny inference burst → memory retrieval → another inference burst

rather than keeping an expensive accelerator occupied throughout.

3. Silmir — closest startup to the actual “bursty-agent chip” thesis

This one is unusually close to what you're describing. Silmir says current accelerators can run agent workloads at under 30% utilization and is building self-adapting silicon that dynamically reorganizes compute, memory and interconnect at runtime. Its stated thesis is that the bottleneck is the loop, not raw matrix compute.

It's early — FPGA prototype / pre-tape-out — so I'd classify it as high thesis fit, low maturity.

4. Unikernel.ai — if you mean runtime/compiler rather than silicon

This may actually be the more interesting answer if you're looking for a software company. It is building an agent compiler, minimal runtime and fast boot loader, explicitly targeting bursty agents, event-driven workloads and serverless inference.

The key insight is that if an agent is active for 200 ms and then waiting on a tool for 10 seconds, keeping a heavyweight VM/container/runtime around is economically stupid. The runtime should essentially disappear during the idle periods.

5. Arm — attacking the overlooked CPU/orchestration layer

Arm's new AGI CPU is explicitly designed for agentic AI infrastructure. Arm argues that agents move continuously between CPUs, memory, storage and accelerators, making the CPU an orchestration engine, rather than merely the thing feeding the GPU.

That's an important architectural direction: agentic AI may be more of a heterogeneous-computing problem than an accelerator problem.

The second tier

Cerebras is more about eliminating latency and maximizing token generation than specifically solving burstiness. Its new CS-4 is explicitly positioned for chatbot/inference workloads and is shipping in 2026.

Etched is extremely interesting if the workload can be constrained to Transformer inference. Its Sohu architecture is specialized for inference and the company now has >$1B of customer contracts; Jane Street is its first customer. www.reuters.com But specialization makes it less naturally suited to the heterogeneous CPU/network/tool portions of an agent loop.

d-Matrix is worth tracking because its compute-in-memory architecture attacks the memory movement problem directly. The broader 2026 inference-chip landscape now includes d-Matrix, Etched, Groq, Cerebras, SambaNova, Tenstorrent, MatX, Taalas, Furiosa, etc.

One important distinction

I wouldn't actually frame the opportunity as “an agent inference chip.”

The recent agent-serving research suggests the harder problem is the scheduler/runtime. Real agent sessions can be stateful for minutes or hours, have heterogeneous CPU/GPU/memory requirements, and shift bottlenecks throughout execution. One recent benchmark found non-LLM components dominate latency in half of its tested agent applications, while sandbox memory can reach 28 GB/session.

So the really interesting architecture is:

agent runtime → scheduler → CPU / accelerator / memory / network → compiler

rather than:

LLM → faster ASIC

That opens a potentially much bigger opportunity around “Kubernetes/vLLM for bursty agents, but co-designed with silicon.”

And there is already evidence that the compiler is becoming a first-class differentiator: Meta's MTIA-2i work uses a Triton backend + TorchInductor integration to make custom accelerator programming substantially easier.

If I were mapping this market for an investment/startup thesis, my shortlist would be:

1. NVIDIA/Groq — incumbent with the deepest hardware/software integration
2. Silmir — most directly aligned with bursty, irregular agent execution
3. SambaNova — strongest disaggregated-inference architecture
4. Unikernel.ai — intriguing runtime/compiler wedge
5. Arm — CPU/orchestration layer
6. d-Matrix — memory-centric accelerator
7. Cerebras — latency/token-generation specialist
8. Etched — extreme specialization/economics

There is also a new benchmark specifically for this market, AA-AgentPerf, which measures agents served per megawatt using real coding-agent trajectories rather than conventional tokens/sec. That is probably a much better lens for comparing these companies than standard LLM benchmarks.

If you're evaluating this as a startup opportunity, the white space I see is not another Groq/Etched-style ASIC. It's a runtime/compiler + scheduling layer that understands the entire agent execution graph and dynamically moves work between CPU, accelerator, memory and network—with silicon co-design as the eventual moat.

Traditional GPUs often drop to 30%–40% utilization on autonomous agent loops . Because agents bounce unpredictably between memory-bound token generation, IO-bound tool calls (APIs, web searches), and CPU-bound orchestration, standard static batching and prompt-in/response-out runtimes create massive idle gaps.

A new tier of silicon, compiler, and runtime companies are tackling the "agent loop" hardware problem—focusing on persistent KV-cache memory fabrics, fast multi-model context switching, and agentic software layers that auto-tune for dynamic workloads.

1. Silicon & Hardware Architecture Players

• d-Matrix: Using Digital Computing-in-Memory (DIMC) architecture via their d-Matrix Corsair platform, they feature SRAM designed to keep weights resident across prefill and decode phases while recycling transient activations. This drastically cuts the memory-movement penalty typical in bursty multi-step agent calls.
• Groq: While initially famous for deterministic tensor streaming processors (TSPs), their real breakthrough has always been their compiler, which eliminates runtime jitter by statically scheduling data movement. Their architecture is increasingly relevant as the industry moves toward ultra-low-latency, predictable multi-turn agent response pipelines.
• Etched: With their Etched Sohu ASIC, they completely hardwire transformer operations directly into silicon, removing the general-purpose instruction fetch overhead that slows down heavy orchestration loops.
• Taalas: Taking a more extreme approach with hard-wired architectures like the Taalas HC1 , they burn entire transformer models and logic directly into silicon masks, eliminating the traditional software runtime stack entirely for dedicated execution.

2. Compiler & Runtime Optimization Layers

• Infinity: They provide an autonomous AI agent layer named Ignition alongside MemoryScope . Instead of human engineers spending months hand-tuning low-level compute kernels for new inference silicon, Infinity’s agent stack autonomously writes, tests, and optimizes kernels and memory locality for dynamic architectures (such as partnering with d-Matrix) in days.
• vLLM & Disaggregated Runtimes (Open Source/Community): At the runtime infrastructure level, projects are moving past collocated serving. Using technologies like RDMA-based KV transfer connectors (NixlConnector ), runtimes are beginning to disaggregate the prefill and decode phases, allowing persistent context to follow an agent across distributed nodes without re-computing or dropping state during pauses.
• Academic & Specialized Compilers (e.g., Stanford's A1, Compiler-next): Emerging frameworks like the Stanford A1 Compiler compile agent workflows ahead-of-time (AOT) or just-in-time (JIT) into optimized execution blocks uniquely tailored to irregular branching input patterns.

Are you looking to integrate these runtimes into on-premise hardware clusters , or are you evaluating cloud-accessible inference providers for an agentic application?

For bursty agent workloads (many short-lived reasoning loops, tool calls, retries, variable latency, idle periods, heterogeneous models), the winning stack probably looks different from today’s “serve a chatbot with continuous batching” stack. The companies attacking this space fall into a few layers:

1. Agent-native inference runtimes / orchestration

These are closest to the problem you describe: scheduling workflows, not just tokens.

• Valar Labs — building an execution layer for agentic inference that treats agents as long-running workloads with scheduling, placement, QoS, and cost optimization across GPUs, TPUs, and other accelerators.
• Gimlet — focused on running multi-stage agent workloads with orchestration, heterogeneous hardware support, and compiler/runtime optimization.
• Together AI — more general inference infrastructure, but increasingly relevant as agent workloads drive demand for flexible inference clusters.

The key innovation here is likely task-level scheduling:

• predict agent execution graphs
• move workloads between accelerators
• prewarm only when needed
• optimize cost per completed task rather than tokens/sec

---

2. AI compiler companies

These attack the “compile the model once, run it everywhere” problem.

• Modular — building compiler infrastructure (including Mojo) aimed at portable high-performance inference across hardware targets.
• Luminal — focused on ahead-of-time compilation, graph optimization, fusion, scheduling, and generating optimized code for GPUs and ASICs.
• Kernelize — working on hardware-agnostic kernel/compiler tooling to reduce the effort of bringing models to new accelerators.

For agents specifically, the compiler challenge changes:

• dynamic graphs instead of fixed graphs
• speculative execution
• KV-cache lifetime management
• memory movement optimization
• compiling tool-use paths

---

3. Custom inference silicon + compiler stacks

These companies are betting that agentic AI needs chips designed around latency and irregularity, not just FLOPS.

• Groq — deterministic execution and compiler-managed scheduling are aimed at predictable low latency inference.
• Etched — building inference-specific silicon and racks optimized for frontier models and agentic workloads.
• Cerebras Systems — large-scale wafer-scale AI systems with a compiler-centric architecture.
• SambaNova Systems — dataflow architecture plus compiler/runtime stack.

The interesting design question: agents have lots of small bursts, so chips may need:

• fast context switching
• massive SRAM/cache
• low startup latency
• efficient sparse execution
• good performance at low utilization

---

4. Hyperscaler internal stacks

The big clouds are also moving down the stack:

• NVIDIA — investing heavily in agentic inference platforms, including compiler-driven inference systems and low-latency accelerator architectures.
• Google — TPU + XLA stack.
• Amazon Web Services — Trainium/Inferentia plus compiler/runtime tooling.
• Microsoft — Maia accelerators and Azure inference stack.
• Meta Platforms — MTIA silicon and automated kernel optimization systems. Meta has described systems that automatically search and optimize inference kernels across heterogeneous hardware.

---

5. Agent runtimes / sandbox infrastructure

A less obvious but important layer: agents are closer to serverless functions than traditional inference servers.

• Unikernel.ai — explicitly targeting bursty agents with lightweight runtimes, fast boot, and reduced idle infrastructure overhead.

---

The companies I would watch most closely for your exact thesis

If the thesis is:

“The next AI computer needs a compiler/runtime designed around bursty autonomous workers, not chat requests.”

I would watch:

1. Valar Labs — task-level agent inference scheduler
2. Gimlet — agent-native inference cloud/runtime
3. Modular — portable compiler layer
4. Groq / Etched — hardware + compiler co-design
5. NVIDIA — likely incumbent winner if CUDA evolves toward agents
6. Unikernel.ai — runtime model for ephemeral agents
7. Kernelize — compiler/kernel automation layer

The missing “killer abstraction” today is probably something like an operating system scheduler for AI workers: a runtime that understands intent, deadlines, tool calls, model choice, and hardware availability rather than simply batching tokens.

NVIDIA (with its Vera single-threaded CPU architecture and Groq 3 LPX co-designed low-latency accelerators), SGLang (via RadixAttention for state-sharing across agent steps), and specialized infrastructure stacks like AWS (with AgentCore ) are currently leading the charge in optimizing silicon, runtimes, and compilers for the bursty, loop-heavy demands of agentic workloads.

Traditional GPUs and serving stacks drop to 30% to 40% utilization during agent execution because agents constantly bounce between memory-bound inference, I/O tool use, and CPU-heavy sequential reasoning loops. The industry is shifting toward specific hardware-software co-designs to fix these bottlenecks:

Key Players and Approaches

• NVIDIA : With the Vera Rubin platform, NVIDIA introduced the Vera CPU —purpose-built with high single-threaded performance rather than massive core density. Because agent loops are sequential (reason -> execute -> repeat), a faster single core reduces micro-jitter and speeds up the loop. They also leverage Groq 3 LPX co-design chips to handle the small-batch FFN decode loops characteristic of agent steps.
• SGLang : Emerged as a primary runtime choice for agentic serving. Unlike traditional runtimes, its RadixAttention engine automatically caches and shares the KV cache across multi-step agent branches and tool-use iterations, avoiding redundant prefill computation when an agent backtracks or queries context iteratively.
• AWS (Amazon Web Services) : Through AgentCore , AWS is building runtime architectures specifically engineered to manage long-running asynchronous agent loops (spanning hours) alongside native state and context memory tracking.
• vLLM Project : Continues to adapt its serving engine with fine-grained batching and scheduling algorithms to smooth out the erratic request arrival rates typical of multi-agent tool execution.

If you can share whether you are deploying on-premises or targeting a specific cloud provider (like AWS, GCP, or Azure) , I can help narrow down which of these runtimes or hardware stacks best fits your latency and throughput constraints.

Yes. If by “bursty agent workloads” you mean workloads with lots of short, heterogeneous inference calls, tool calls, idle gaps, fan-out/fan-in, and tight tail-latency SLOs, there’s a pretty interesting stack forming around this.

The key architectural insight is that agents are not ordinary batch inference: demand arrives in spikes, utilization is uneven, and the workload repeatedly crosses CPU ↔ accelerator boundaries. Recent production/academic work is explicitly finding this mismatch in conventional homogeneous GPU servers.

The companies I'd watch

1. NVIDIA Dynamo — probably the most direct answer

NVIDIA Dynamo is arguably the closest thing today to an agent-workload runtime rather than simply an inference engine.

It provides scheduling, request routing, KV-cache management, dynamic GPU allocation and, importantly, disaggregated serving—separating prefill and decode into independently scalable pools. NVIDIA explicitly lists deploying AI agents as a Dynamo use case.

That matters for burstiness because you don't want to reserve an entire GPU configuration for every agent. Dynamo can move resources around as workload pressure changes.

NVIDIA is also pushing this below the software layer: Vera Rubin combines GPUs with Groq LPUs, and the architecture is explicitly aimed at agentic inference.

My take: if you're asking “who is building the reference architecture for this?”, NVIDIA is the strongest incumbent.

---

2. Modular — perhaps the most interesting compiler play

Modular is particularly relevant if your question is really about the compiler/runtime layer.

Its MAX stack is positioned as compiled inference for agentic workflows, including tool calling, across NVIDIA, AMD and Apple hardware. Modular specifically talks about workflows where dozens of LLM calls are chained together and says it optimizes the entire pipeline, not just individual model inference.

That's conceptually important: the opportunity isn't merely a faster GEMM. It's a compiler/runtime that understands:

agent graph → model calls → tool calls → memory state → scheduling → accelerator selection → batching

rather than treating every inference request independently.

My take: if you're looking for the company closest to an “agent compiler”, I'd put Modular at the top of the list.

---

3. SambaNova — unusually explicit about agents

SambaNova has gone unusually far in positioning its hardware around agentic inference. Its SN50 RDU is explicitly described as purpose-built for agentic inference, with the company emphasizing data movement and memory rather than simply FLOPS.

More interestingly, SambaNova is demonstrating heterogeneous/disaggregated inference: GPU for prefill and RDU for decode. Its production demo reported 2× inference speed versus B200-only configurations.

That's almost exactly the architecture I'd expect for agents: different silicon for different phases of a highly irregular workload.

---

4. Groq — the low-latency extreme

Groq is more narrowly focused: make inference extremely fast and predictable.

That becomes valuable for agents because an agent may make 10–50 sequential model calls. A modest reduction in per-call latency compounds across the workflow.

And this is no longer an isolated startup architecture: NVIDIA has incorporated Groq 3 LPX inference accelerator racks into the Vera Rubin platform.

So I'd view Groq less as the complete “agent runtime” and more as an important specialized execution engine inside a heterogeneous agent-inference system.

---

5. Cerebras — brute-force decode acceleration

Cerebras Systems is another major answer if the bottleneck is sequential token generation.

Its wafer-scale architecture is particularly attractive for low-latency inference, and its cloud inference business is expanding rapidly; its Q2 2026 results showed the inference/cloud business growing sharply.

Cerebras is also moving toward disaggregated inference, pairing its decode-oriented hardware with AMD's systems.

So:

NVIDIA/Modular = orchestration/compiler

Groq/Cerebras = extremely fast execution

That's an increasingly important distinction.

---

6. Etched — interesting if you believe the workload becomes stable

Etched is taking the ASIC route. Its approach is essentially: if Transformer inference is important enough, specialize the silicon aggressively.

The company is now building rack-scale inference systems and explicitly co-designing chips + packages + PCBs + cooling + interconnect + software. Its first hardware is intended to optimize both prefill and decode.

The tradeoff is obvious: specialization gives you excellent economics when the workload is predictable, but agents are intrinsically more heterogeneous than conventional LLM serving.

That makes Etched more interesting for large-scale, known agent workloads than for arbitrary agent execution.

---

7. d-Matrix — memory-centric inference

d-Matrix is attacking the memory bottleneck with compute-in-memory architectures.

That's relevant because agent workloads often aren't compute-saturated; they spend a lot of time moving model/KV state around. The broader inference-chip landscape increasingly includes d-Matrix alongside Groq, Cerebras, Etched, SambaNova, etc.

---

8. AMD + Taalas — worth watching

AMD acquired Taalas this month specifically to strengthen its inference silicon. Taalas' technology targets compute/memory bottlenecks in inference and is being integrated into AMD's accelerator roadmap.

AMD is therefore becoming a more credible second ecosystem for this architecture, particularly when combined with Modular and SambaNova-style heterogeneous serving.

---

But there's a gap

The interesting thing to me is that nobody completely owns the “bursty agent machine” yet.

Today's stack roughly looks like:

Agent framework
↓
agent scheduler/orchestrator
↓
inference runtime — Dynamo / MAX / vLLM / SGLang
↓
compiler — CUDA / Triton / MAX / vendor compiler
↓
heterogeneous accelerators — GPU / LPU / RDU / ASIC
↓
memory + network

The missing layer is something like an agent-native runtime/OS that understands the whole workflow.

It would dynamically decide:

“Agent A is waiting on a browser tool, so don't reserve accelerator capacity. Agent B just returned a 40K-token context, so send it to prefill hardware. Agent C is in sequential reasoning, so move it to a low-latency decode accelerator. Agent D is idle but has a hot KV cache—keep that state resident. Five agents just fanned out simultaneously—temporarily batch them.”

That is materially different from conventional continuous batching.

Recent research is pointing in exactly this direction: agent workloads have low baseline utilization punctuated by sudden spikes, heterogeneous model/tool composition, and CPU/GPU fragmentation.

If I were mapping the opportunity

I'd divide the competitive landscape into four bets:

Compiler: Modular
Runtime/orchestrator: NVIDIA Dynamo
Specialized low-latency silicon: Groq / Cerebras
Agent-specific heterogeneous system: SambaNova

And I'd keep a close eye on Etched and d-Matrix as the more aggressive ASIC bets.

The white space, in my view, is an agent-native runtime that makes heterogeneous silicon look like one elastic computer—with burst-aware scheduling, persistent KV/state, tool-call-aware preemption, and automatic placement of each stage of an agent graph onto the cheapest accelerator that can hit the tail-latency target.

That feels like a more interesting company category than “yet another LLM inference chip.”

Traditional AI accelerators (like standard enterprise GPUs) often experience low utilization (3 0% to 4 0%) on agentic workloads. Because agents constantly loop, call tools, branch, backtrack, and maintain long-running KV context across variable steps, the bottleneck isn't just raw compute—it’s the chaotic mix of memory-bound model calls, I/O-bound tool execution, and CPU-bound orchestration.

A new wave of infrastructure players are targeting this exact friction point through specialized compilers, heterogeneous runtimes, and architectural orchestration:

• Modular : Building the Modular MAX Engine and the Mojo language ecosystem, they focus heavily on hardware-agnostic, whole-graph compilation and unified kernel execution. Their runtime abstracts heterogeneous hardware (GPUs, CPUs, and custom accelerators) to optimize serving paths and drastically slash memory/latency overhead typical in dynamic execution graphs.
• Groq : Though widely known for their LPU (Language Processing Unit) architecture and deterministic streaming, their core breakthrough has always been their software compiler . By moving scheduling logic entirely to compile time rather than runtime, their approach eliminates standard dynamic memory queuing bottlenecks—making it inherently suited for predictable, rapid-fire token bursts.
• Tenstorrent : Led by Jim Keller, their RISC-V processor architecture and software stack are explicitly built for scalable, flexible, and on-premise AI inference. Their hardware-software co-design targets distributed, non-traditional workloads where orchestration and localized memory proximity matter as much as raw FLOPs.
• Etched : Operating on custom ASIC design (Sohu), they bypass traditional programmable core overhead by hardcoding transformer architectures directly into silicon. While optimized for continuous massive inference, companies confronting extreme throughput demands look to hardwired ASICs to bypass the compiler/scheduler tax.
• ARM (with specialized ecosystem partners) : Through high-core-count CPU architectures tailored for agentic orchestration (e.g., handling rapid parallel API fan-outs, search, and sandboxed code compilation alongside accelerators), ARM-based server nodes are increasingly targeted by compiler toolchains aiming to optimize the CPU-bound orchestration side of the agent loop.

If you're designing a stack for a specific bottleneck, tell me:

• Are you constrained mostly by KV cache memory persistence, I/O tool latency , or model-to-model context switching?
• Are you targeting edge devices or data center scale?

I can give you a more tailored breakdown of how these runtimes handle your exact pipeline.