The modern AI stack has expanded far beyond foundational frontier labs (like OpenAI, Anthropic, and Google DeepMind) into a dense, highly competitive ecosystem of AI infrastructure startups . These companies build the plumbing, guardrails, compute engines, and data loops that make large-scale models reliable and deployable.

1. Evals & Observability (Testing, Scoring, and Tracing)

As frontier models and autonomous agents grow more complex, deterministic testing is insufficient. Startups in this layer provide LLM-as-a-judge frameworks, regression tracking, and CI/CD quality gates.

• Braintrust: An enterprise-grade evaluation and experimentation platform widely used for tracking model performance and managing prompt engineering workflows.
• Confident AI (DeepEval): Known for its open-source framework (DeepEval ) and enterprise platform providing automated unit testing, multi-turn conversation simulation, and adversarial red-teaming.
• LangChain (LangSmith): Provides comprehensive tracing, debugging, and evaluation tools tailored for complex LLM chaining and agent architectures.
• Arize AI: Combines traditional ML observability with LLM evaluation, tracing, and experimentation pipelines.
• Langfuse: An open-source LLM engineering platform handling tracing, analytics, and prompt management.

2. Data Infrastructure (Curation, Sourcing, and Synthetic Data)

Frontier labs no longer just need raw web scrapes; they need high-density reasoning data, expert human verification, and synthetic data generation.

• Scale AI: The dominant incumbent-turned-infrastructure titan providing RLHF (Reinforcement Learning from Human Feedback) and data labeling.
• Mercor: Focuses on matching elite human expert knowledge and specialized domain data rather than generic crowd-sourced labeling.
• Surge AI: Specializes in high-complexity data labeling, RLHF, and semantic data curation for frontier reasoning models.
• Distill/Synthetic Data Startups: A rising cohort of specialized seed-to-series startups focused on generating synthetic reasoning traces (distilling larger model outputs to train smaller, efficient models).

3. Inference (Serving, Optimization, and Routing)

Running frontier models efficiently requires aggressive memory-bandwidth optimization, speculative decoding, and dynamic request routing.

• Together AI: Provides decentralized cloud compute and blazing-fast inference APIs optimized for open and custom weights.
• Fireworks AI: Known for ultra-low latency production inference engines optimized for open-source LLMs and fine-tunes.
• Groq: Pioneers the LPU (Language Processing Unit) architecture, bypassing traditional GPU memory bottlenecks to deliver extreme deterministic token-per-second throughput.
• OpenRouter: Acts as a model-agnostic routing and aggregation layer, letting developers dynamically failover or route traffic across dozens of distinct AI providers.
• Cerebras / Etched: Hardware-level inference infrastructure building wafer-scale engines and application-specific integrated circuits (ASICs) designed exclusively to make transformer inference orders of magnitude faster.

4. Memory (Persistent State and Agentic Recall)

While raw context windows are expanding, production agents require standalone long-term memory systems to recall past user interactions, preferences, and enterprise context across sessions.

• Mem0: Specializes in a universal, persistent memory layer for AI assistants and multi-agent systems.
• Letta (formerly MemGPT): Treats LLM memory like an operating system, utilizing virtual memory paging to allow agents to manage long-term state across infinite lifecycles.
• Zep: Builds memory and temporal knowledge graphs tailored to LLM applications to drastically improve retrieval accuracy and personalization.
• Pinecone / Qdrant / Milvus: Vector database foundations powering the semantic indexing and retrieval architecture beneath these memory layers.

5. Safety, Security, and Guardrails (Alignment and Compliance)

With enterprise adoption tied directly to compliance, data privacy, and jailbreak prevention, safety infrastructure has shifted from theoretical alignment to active runtime governance.

• Noma Security: Focuses heavily on application-layer AI security, protecting enterprise agentic workflows from prompt injection and data exfiltration.
• Llama Guard / NeMo Guardrails (NVIDIA ecosystem) / Lakera: Real-time firewall tools deployed in front of inference endpoints to screen inputs and outputs for toxicity, PII leaks, and malicious exploits.
• Robust Intelligence (acquired by Cisco, but pioneered the space): Automated security testing for AI model pipelines against adversarial attacks and data drift.

Would you like to explore how the memory layer integrates with vector databases , or dive deeper into the economics of production inference providers?

Yes. If you mean the picks-and-shovels startups that become more valuable as the number of frontier/model labs increases, I’d map the market roughly like this.

The important shift is that the stack is moving above raw GPUs: new labs can rent compute, but they still need proprietary data, post-training loops, evals, inference optimization, persistent agent state, and safety infrastructure.

My shortlist

1. Data is probably the biggest opportunity

The interesting part isn't traditional labeling anymore. It's post-training data and environments.

Mercor is particularly interesting because it has moved toward expert-generated datasets, benchmarks and RL environments rather than simply selling annotation labor. Its own description explicitly positions it around benchmarks, evaluation environments and expert datasets for frontier labs.

I'd put these into two buckets:

• Human expertise: Mercor, Scale AI, Surge AI, Turing, Deccan AI
• Synthetic / environment / RL data: startups building simulated worlds, verifiers, coding environments and agent tasks

Deccan AI is a useful signal: it raised a $25M Series A specifically around post-training data and evaluation work, including coding and agent capabilities.

Thesis: the scarce resource is increasingly not tokens of data but high-information training experiences.

---

2. Evals become the control plane

This may be the most strategically important layer.

The frontier labs can build their own internal benchmarks, but every company building on those models needs to answer:

"Did the new model/prompt/agent actually get better?" That's why I'd watch:

• Braintrust — evaluation + experimentation + production traces
• Langfuse — open-source observability/evaluation
• Patronus AI — model evaluation and safety
• Arize — ML/LLM observability and evaluation
• LangSmith — evaluation/observability around LangChain
• Humanloop — evaluation and human feedback

Braintrust and Langfuse are an especially interesting pair: one is a polished commercial platform, the other an open-source/self-hostable alternative.

The deeper opportunity: evals become the equivalent of CI/CD for intelligence. Once models and agents are changing every day, you need automated tests before you can safely ship those changes.

---

3. Inference is becoming its own infrastructure category

This is probably the most mature startup market on the list.

My current tier-one group:

• Fireworks AI
• Baseten
• Together AI
• Modal
• Replicate
• fal.ai
• RunPod

Fireworks, Baseten and Together are particularly notable because they're attacking the economics of running open-weight/custom models rather than simply reselling frontier APIs. Current industry comparisons put those three at the center of the production-inference market.

Then there's a second layer:

Inference routers

• OpenRouter
• Vercel AI Gateway

These don't necessarily want to own the GPU. They want to own the decision about which model gets each request.

That could become extremely valuable if model choice becomes dynamic:

request → classify → route → infer → evaluate → retry/escalate In that world, the router starts looking more like a model operating system.

---

4. Memory is the newer, more speculative category

This is the one I'd watch most closely if you're looking for new companies rather than established infrastructure.

The core players:

• Mem0
• Zep / Graphiti
• Letta
• Supermemory
• Cognee
• Hindsight

But "memory" actually contains several different businesses:

Personal memory → "What does this user like/know?"

Episodic memory → "What happened in previous sessions?"

Semantic memory → "What facts has this agent learned?"

Temporal/relational memory → "What was true about X at a particular point in time?"

Agent working memory → "What should remain in context while this autonomous process runs?"

Mem0, Zep and Letta are making notably different architectural bets: Mem0 is a general memory layer, Zep/Graphiti emphasizes temporal knowledge graphs, while Letta treats memory as part of the agent runtime itself.

My bet: memory eventually gets absorbed into agent runtimes/models, but there's a very interesting window before that happens.

---

5. Safety is splitting into two markets

This is important.

Model safety

• Gray Swan AI
• Haize Labs
• Giskard
• Patronus AI

These companies are closer to:

"Can we discover what this model can do wrong?"

Red teaming, adversarial testing, automated jailbreak discovery, capability evaluation, etc.

Runtime / application security

• Lakera
• HiddenLayer
• Protect AI

These are closer to:

"Can we keep an AI application from being compromised?"

HiddenLayer, for example, now positions its platform across AI discovery, supply-chain security, attack simulation and runtime security.

That distinction matters because agent security is becoming cybersecurity: prompt injection, malicious tools, data exfiltration, compromised MCP servers, model supply-chain attacks, excessive permissions, etc.

---

If I were building an investment/watch list

I'd divide the opportunities into three tiers.

🟢 Already becoming infrastructure

• Fireworks AI
• Baseten
• Together AI
• Modal
• Braintrust
• Langfuse
• Mercor
• Scale AI
• Arize
• Lakera
• HiddenLayer

🟡 Early category leaders with potentially huge upside

• Mem0
• Zep
• Letta
• Supermemory
• Patronus AI
• Gray Swan AI
• Haize Labs
• Deccan AI

🔵 The categories I'd hunt for new startups

This is where I think it gets particularly interesting:

• RL environments as a service
• Automated frontier-model eval generation
• Agent trajectory/data infrastructure
• Synthetic expert data with verification
• Long-horizon agent memory
• Agent identity + permissions
• Tool/MCP security
• Continuous adversarial testing
• Inference routing/optimization
• Model behavior observability
• Data provenance/licensing for training
• Post-training infrastructure
• Verifiable AI outputs / proof systems

The common thread is that the new AI labs are increasingly becoming learning systems, not merely model-training organizations. They continuously collect experience → generate data → train/post-train → evaluate → deploy → observe failures → generate more data.

So the really interesting infrastructure company may sit across several of these boxes, rather than cleanly inside one.

The stack I'd draw

                 FRONTIER / NEW AI LAB
                         │
        ┌────────────────┼────────────────┐
        │                │                │
      DATA             MODELS          COMPUTE
        │                │                │
   Mercor/Scale     OpenAI/etc.      GPU clouds
   Surge/Deccan                        │
        │                           Inference
        ▼                        Fireworks/Baseten
   POST-TRAINING                       │
        │                              ▼
        ├──────────────► EVALS ◄─── Observability
        │                  │
        │                  ▼
        │              SAFETY
        │          Red team / guardrails
        │                  │
        ▼                  ▼
                 AGENT RUNTIME
                       │
                ┌──────┴──────┐
                ▼             ▼
             MEMORY        TOOLS/MCP
                │             │
                └──────┬──────┘
                       ▼
                 PRODUCTION
                       │
                       └────► new data
                                │
                                └──────► training loop

The biggest strategic insight: inference is becoming commoditized infrastructure, while data + evals + feedback + memory + safety form the closed-loop infrastructure around intelligence. That's the part I'd pay the most attention to over the next 2–3 years.

And the market is already moving in that direction: inference platforms are proliferating rapidly, while training-data companies are shifting toward expert data, evaluations and RL environments.

Yes. If you mean the picks-and-shovels companies that become more valuable as frontier-model labs multiply, rather than the labs themselves, I’d map the market roughly like this:

The ones I think are especially interesting

1. Scale AI — data + evals + post-training

Scale is arguably the clearest example of an AI-lab infrastructure company. Its Data Engine spans data generation, RLHF, evaluation and red teaming, while Scale Labs now explicitly works on agents, post-training, safety, evaluation and alignment.

The interesting evolution is from “labeling company” → training-data operating system for frontier labs. Scale says its data business now supplies training/test/improvement infrastructure to leading AI labs, and its newer frontier-data products include agent and complex-reasoning datasets.

2. Fireworks / Baseten / Together — the independent inference layer

This may be the biggest infrastructure opportunity after GPUs.

The core thesis is that increasingly many labs will produce models, but relatively few application companies will want to operate the serving stack themselves. Fireworks, Baseten and Together are competing to own that layer: GPU orchestration, optimized kernels, quantization, autoscaling, fine-tuning and model APIs. A recent 2026 industry survey identifies these three, along with Modal, as the leading independent inference platforms.

I'd distinguish them:

• Fireworks — performance/optimization-heavy model serving.
• Baseten — production deployment for your own/custom models.
• Together — broad open-model ecosystem + inference + training.
• Modal — more developer/cloud infrastructure abstraction.
• OpenRouter — potentially the routing/control plane above all of them.

The OpenRouter thesis is particularly interesting: instead of owning compute, own the traffic and routing layer across model providers.

3. Letta + Mem0 — memory becomes a primitive

This category is earlier but potentially enormous.

letta.com is going after persistent, stateful agents rather than simply adding a vector database. Its current architecture treats memory, skills, computer use and agent execution as one runtime, with memory decoupled from the underlying model provider.

mem0.ai is taking a more infrastructure-like approach: a persistent memory API that sits between the agent and storage, automatically extracting, updating and retrieving memories. It says it has more than 160,000 developers and has raised $24M.

My bet here is that “memory” eventually looks less like a feature and more like a database primitive—just as caching, identity and search became infrastructure.

4. Patronus — evals evolving into simulated worlds

Patronus is one of the more interesting companies because it's moving beyond static benchmark/eval dashboards. Its 2026 strategy includes simulated digital environments where agents can be stress-tested on realistic, multi-step tasks. It raised $50M in June 2026 for that direction.

That's important because frontier-agent evaluation increasingly becomes:

model → environment → trajectory → outcome rather than:

prompt → answer → score. That creates a much larger infrastructure market.

5. Lakera / Giskard — security becomes runtime infrastructure

The security problem changes radically once models can use tools, access databases and take actions.

lakera.ai is positioned around runtime protection, prompt-injection prevention, data-leakage protection and AI red teaming.

giskard.ai is pushing toward continuous, agent-specific red teaming: dynamically generating attacks based on the agent's behavior and business context rather than relying on static test cases.

I think agent security is substantially bigger than “LLM guardrails.” The eventual product is closer to a combination of Cloudflare + application security + runtime authorization for autonomous software.

My mental model of the emerging stack

I'd draw the new AI-lab infrastructure stack like this:

Data → training data, preference data, synthetic data, agent trajectories ↓

Training / post-training → RL environments, reward models, fine-tuning ↓

Models → frontier labs / open-weight labs ↓

Inference → serving, routing, optimization, GPU orchestration ↓

Context → retrieval, memory, state, identity ↓

Agents → tools, computer use, multi-agent execution ↓

Evals → benchmarks, simulations, observability, outcome measurement ↓

Safety → red teaming, runtime security, policy enforcement, monitoring

The really interesting thing is that the boundaries are collapsing. Scale is already spanning data + evals + safety; Patronus spans evals + safety + simulated environments; Letta spans memory + agent runtime; inference companies increasingly incorporate fine-tuning and evaluation.

If I were making a startup-investor watchlist

My highest-conviction categories would be:

1. Inference control plane — especially routing/orchestration rather than commodity GPU hosting.
2. Agent trajectory/data infrastructure — collecting, curating and replaying real agent interactions.
3. Evaluation-as-infrastructure — evals integrated into training, deployment and CI rather than a benchmark dashboard.
4. Agent memory/context — persistent state that survives model/provider changes.
5. Agent security — runtime authorization, prompt injection, data exfiltration and tool-use security.
6. Synthetic + expert data — particularly reasoning, coding, robotics and domain-specific post-training data.

And I'd watch Surge AI particularly closely on the data side: it is now selling frontier training data, evals and RL environments as off-the-shelf products, including coding-agent and enterprise-agent datasets.

The deeper investment thesis is: the number of frontier labs may grow 10×, but the infrastructure required to train, evaluate, serve, remember and secure their models may grow even faster. The winners don't necessarily need to build the best model—they need to become unavoidable pieces of the model-development loop.

My bet here is that “memory” eventually looks less like a feature and more like a database primitive—just as caching, identity and search became infrastructure.

4. Patronus — evals evolving into simulated worlds

Patronus is one of the more interesting companies because it's moving beyond static benchmark/eval dashboards. Its 2026 strategy includes simulated digital environments where agents can be stress-tested on realistic, multi-step tasks. It raised $50M in June 2026 for that direction.

That's important because frontier-agent evaluation increasingly becomes:

model → environment → trajectory → outcome rather than:

prompt → answer → score. That creates a much larger infrastructure market.

5. Lakera / Giskard — security becomes runtime infrastructure

The security problem changes radically once models can use tools, access databases and take actions.

lakera.ai is positioned around runtime protection, prompt-injection prevention, data-leakage protection and AI red teaming.

giskard.ai is pushing toward continuous, agent-specific red teaming: dynamically generating attacks based on the agent's behavior and business context rather than relying on static test cases.

I think agent security is substantially bigger than “LLM guardrails.” The eventual product is closer to a combination of Cloudflare + application security + runtime authorization for autonomous software.

My mental model of the emerging stack

I'd draw the new AI-lab infrastructure stack like this:

Data → training data, preference data, synthetic data, agent trajectories ↓

Training / post-training → RL environments, reward models, fine-tuning ↓

Models → frontier labs / open-weight labs ↓

Inference → serving, routing, optimization, GPU orchestration ↓

Context → retrieval, memory, state, identity ↓

Agents → tools, computer use, multi-agent execution ↓

Evals → benchmarks, simulations, observability, outcome measurement ↓

Safety → red teaming, runtime security, policy enforcement, monitoring

The really interesting thing is that the boundaries are collapsing. Scale is already spanning data + evals + safety; Patronus spans evals + safety + simulated environments; Letta spans memory + agent runtime; inference companies increasingly incorporate fine-tuning and evaluation.

If I were making a startup-investor watchlist

My highest-conviction categories would be:

1. Inference control plane — especially routing/orchestration rather than commodity GPU hosting.
2. Agent trajectory/data infrastructure — collecting, curating and replaying real agent interactions.
3. Evaluation-as-infrastructure — evals integrated into training, deployment and CI rather than a benchmark dashboard.
4. Agent memory/context — persistent state that survives model/provider changes.
5. Agent security — runtime authorization, prompt injection, data exfiltration and tool-use security.
6. Synthetic + expert data — particularly reasoning, coding, robotics and domain-specific post-training data.

And I'd watch Surge AI particularly closely on the data side: it is now selling frontier training data, evals and RL environments as off-the-shelf products, including coding-agent and enterprise-agent datasets.

The AI ecosystem has shifted from a pure "frontier model training" race to an intense focus on the supporting infrastructure stack. As frontier labs (OpenAI, Anthropic, Google, Meta, DeepSeek) push capabilities further, a robust ecosystem of specialized startups is building the plumbing for evals, data, inference, memory, and safety.

1. Inference (Speed, Scale & Cost-Efficiency)

Inference is where ongoing compute demand lives. Startups here focus on maximizing GPU efficiency, reducing latency, and serving open-source or custom models at scale.

• Fireworks AI : Recognized as a leading high-throughput, low-latency pure inference engine optimized for production-grade open-source models.
• Together AI : Provides a decentralized cloud platform specialized for running, fine-tuning, and scaling generative AI models.
• Baseten : Highly favored by enterprises for deploying and serving custom models with high performance and control.
• Modal : A fast-rising serverless developer infrastructure platform that lets engineers spin up compute for inference and workloads instantly.

2. Evals & Tracing (Testing, Monitoring & Quality)

Because LLMs and agents are non-deterministic, debugging them requires entirely new evaluation frameworks to test accuracy, regression, and behavior.

• Braintrust : Leading enterprise evaluation platform that handles data collection, evals, and prompt-testing pipelines at scale.
• LangChain (LangSmith) : Expanded heavily from an orchestration tool into a dominant ecosystem provider offering robust tracing, debugging, and evaluation for LLM applications.
• LMArena (Chatbot Arena) : Has transitioned from a crowdsourced benchmarking project into critical market infrastructure for model evaluation.

3. Data & Synthetic Data (Curation & Training Prep)

With the world running low on fresh human-generated text for pre-training, data startups focus on cleaning unstructured data and generating high-fidelity synthetic data.

• Gretel AI (integrated into NVIDIA's ecosystem ecosystem following acquisition): A pioneer in generating privacy-preserving synthetic tabular, text, and time-series data to train robust ML/AI pipelines.
• Unstructured Data Cleaners : A wave of specialized platforms focused on parsing complex PDFs, audio, and video into clean, machine-readable tokens for Retrieval-Augmented Generation (RAG) and fine-tuning.

4. Memory & Context Management (Vector & State Layer)

Expanding context windows is hardware-heavy, so memory infrastructure relies on efficient vector databases and state orchestration layers to give models long-term memory.

• Pinecone : A fully managed vector database built specifically for high-scale, low-latency semantic search and long-term agent memory.
• Milvus / Chroma : Open-source and enterprise vector databases handling the massive scale of embedding spaces required for advanced RAG.
• LlamaIndex : Focuses heavily on the data-to-agent connective tissue, indexing enterprise data sources so models can query them logically.

5. Safety, Governance & Guardrails (Risk & Compliance)

As enterprises deploy agents that can take actions, safety has morphed from theoretical alignment into enterprise compliance firewalls, risk platforms, and runtime guardrails.

• [Llama Guard / NeMo Guardrails (NVIDIA ecosystem) / Independent startups]: Focused on real-time input/output filtering to stop prompt injections, toxicity, and hallucinations.
• AI Risk & Governance Platforms : A rapidly growing category capturing a massive share of venture capital deal counts, providing continuous auditability, access control, and alignment monitoring for enterprise agent rollouts.

If you want to zoom in on one of these layers, let me know which area (Inference, Evals, Data, Memory , or Safety ) you want to explore, and I can break down their technical architectures or market positioning.

The AI infrastructure stack has evolved past raw GPU compute into a specialized ecosystem of software and hardware control points. Startups are tackling major bottlenecks across evals, data, inference, memory, and safety:

• Evaluations (Evals) & Observability
  • Braintrust provides production trace-to-eval workflows, AI-assisted scoring, and release gating for enterprise deployments.
  • Other prominent evaluation and testing platforms like Coval specialize heavily in automated agent reliability and regression testing.
• Data Infrastructure
  • Scale AI remains a dominant force for high-end human and synthetic data labeling for frontier labs.
  • Surge AI provides advanced data labeling and reinforcement learning from human feedback (RLHF) pipelines, while platforms like lakeFS bring code-like version control and reproducibility to massive AI data lakes.
• Inference & Routing
  • Together AI and Fireworks AI lead the independent charge in high-throughput, low-latency model serving and fine-tuning.
  • Baseten and Modal offer serverless and containerized infrastructure engineered specifically to scale production inference workloads.
  • OpenRouter acts as a unified gateway/router across dozens of proprietary and open-source models.
• Memory & Storage Hierarchy
  • VAST Data and WEKA build high-performance, flash-based disaggregated storage platforms required to feed massive token volumes and training checkpoints.
  • At the hardware-adjacent memory layer, companies are optimizing pooled Compute Express Link (CXL) architectures and persistent KV cache layers to alleviate the heavy dynamic memory bottlenecks facing agentic inference.
• Safety, Security & Guardrails
  • Goodfire has pioneered mechanistic interpretability, allowing developers to look inside model weights and understand operational internals.
  • Noma Security specializes in end-to-end AI supply chain and runtime security.
  • Patronus AI and Mindgard focus on automated adversarial red-teaming, prompt injection defense, and vulnerability assessments for enterprise deployments.

Would you like to explore the economic and pricing models of inference vs. training infrastructure , or dive deeper into one specific layer like agent memory architecture?

If you mean the startups that become indispensable suppliers to frontier-model labs, rather than generic “AI infrastructure,” I’d map the market roughly like this as of August 2026:

1. Evals / model quality

These are building the measurement and feedback loop around models and agents.

• Braintrust — evals, datasets, experiments, tracing and CI-style regression testing. Probably one of the clearest “developer infrastructure” plays.
• Patronus AI — automated evaluation, safety testing and benchmarking of foundation models.
• Arize AI — observability/evaluation, particularly strong around production monitoring and its open-source Phoenix stack.
• Confident AI — developer-centric LLM/agent testing.
• Galileo — evaluation, monitoring and quality management for GenAI.
• Langfuse — open-source tracing, datasets and evaluation infrastructure.

The interesting shift is that evals are becoming part of the training loop, rather than simply a QA product. The emerging stack is: generate data → train/post-train → evaluate → identify failures → generate more data → repeat. Current market surveys put Braintrust, Arize, Langfuse, Patronus and others squarely in this layer.

2. Data / RL environments

This may be the most strategically important layer for frontier labs.

• Mercor — expert-generated data, human evaluation, benchmarks and increasingly RL environments. Its pitch is essentially “the labor/data layer for frontier AI.”
• Surge AI — high-end human data, RLHF and frontier-model post-training.
• Scale AI — still enormous, spanning labeling, evaluation, RLHF and government/enterprise data.
• Handshake AI — expert data and post-training.
• Turing — expert data, coding data and model-training services.
• Invisible Technologies — human-in-the-loop data and post-training operations.
• Deccan AI — newer entrant focused on post-training data/evaluation, particularly expert work.
• Snorkel AI — programmatic/synthetic data generation and data curation.

The really interesting subcategory here is RL environments. Mercor, for example, explicitly describes building realistic environments, tools, tasks and verifiers for frontier models.

That suggests the next generation isn't merely “label 10 million examples.” It's “construct an environment in which an agent can attempt millions of difficult tasks and automatically receive a meaningful reward.”

3. Inference / serving

This is arguably the hottest infrastructure market right now because inference is becoming the economic bottleneck.

• Fireworks AI — high-performance inference, open-weight models, fine-tuning and optimized serving.
• Baseten — production deployment and inference for custom models.
• Together AI — open-model hosting, inference, fine-tuning and training.
• Modal — programmable GPU infrastructure and model serving.
• Groq — specialized inference hardware/cloud.
• DeepInfra — low-cost model inference.
• fal — especially strong in image/video/audio generation.
• OpenRouter — routing across model providers rather than owning the underlying compute.

There's a useful structural distinction:

GPU/cloud layer → inference engine → inference platform → model router.

Fireworks/Baseten/Together/Modal are fighting around the middle; OpenRouter is moving above them. Recent market research identifies those companies as the core independent inference platforms, while inference spending itself is becoming a larger share of AI infrastructure economics.

And there is a new hardware layer worth watching: Etched. It just raised $700M at a reported $21B valuation, explicitly betting that inference economics justify purpose-built silicon.

4. Memory / context

This is the layer I think is most underappreciated.

• Mem0 — general-purpose persistent memory API.
• Zep — temporal/knowledge-graph memory via Graphiti.
• Letta — descended from MemGPT; treats memory as a fundamental part of the agent runtime.
• Cognee — graph-oriented memory/context infrastructure.
• Supermemory — memory/context layer, particularly interesting for coding agents.
• Hindsight — persistent agent memory/retrieval.

The architectural battle is fascinating:

vector memory (Mem0) vs. temporal graphs (Zep) vs. agent-managed/paged memory (Letta) vs. knowledge graphs (Cognee).

The common thesis is that a 1M-token context window isn't the same thing as memory. Agents need to store, retrieve, update, reconcile and forget information over months of interaction. Current comparisons consistently put Mem0, Zep and Letta among the leading dedicated systems.

5. Safety / security

I'd split this into runtime safety, model security, and red teaming.

• Lakera — prompt-injection/jailbreak defense, data leakage and agent runtime security. It says it now protects AI agents and MCP workloads as well as conventional GenAI apps.
• Protect AI — model/supply-chain security and AI security tooling.
• HiddenLayer — model security and adversarial protection.
• CalypsoAI — enterprise AI security, governance and protection.
• Adversa AI — red teaming and adversarial AI security.
• Haize Labs — automated adversarial testing/red teaming.

The category is moving beyond traditional “content moderation.” The emerging problem is agent security: prompt injection, tool abuse, malicious MCP servers, exfiltration, model supply-chain attacks and agents escaping their intended sandbox. Recent industry analysis groups Protect AI, HiddenLayer, Lakera, CalypsoAI and Adversa around this broader AI-security stack.

My shortlist of the most interesting companies

If I were trying to understand where the next infrastructure layer around frontier labs is forming, rather than compiling every vendor, I'd watch:

But there's a bigger pattern here.

The stack is converging

The most valuable companies may not remain cleanly in one category.

The frontier-lab infrastructure loop increasingly looks like:

Data → post-training/RL → evals → inference → telemetry → memory → safety → more data

And the companies with the strongest strategic position are the ones that sit on multiple edges of that loop.

For example, Mercor is moving from expert labor → data → evaluations → RL environments. Fireworks is moving from inference → fine-tuning → evaluation. Memory companies are moving toward agent runtimes. Safety vendors are moving from static guardrails toward runtime agent security.

That's why I'd pay particular attention to Mercor, Fireworks, Baseten, Braintrust, Mem0, Letta and Lakera. They aren't merely selling another developer tool; they're trying to own a piece of the production feedback loop around increasingly autonomous models.

If your underlying question is “which of these could become the next $10B+ infrastructure companies?”, I'd rank the categories quite differently: inference and frontier data are the obvious current winners, while memory and evals may have the more interesting new-platform upside.

The AI infrastructure stack has evolved past raw compute into a sophisticated ecosystem of control points . As inference workflows, agentic systems, and frontier models scale, specialized startups have emerged around key bottlenecks: evaluations, data, inference, memory, and safety/security.

1. Evaluations & Observability (Evals)

Evaluating non-deterministic model outputs, scoring agent behaviors, and regression testing require dedicated evaluation infrastructure:

• Braintrust : Combines production traces, datasets, automated scorers, and release gating, scaling heavily as enterprise eval infrastructure.
• Arize AI : A leading ML observability and LLM evaluation platform (utilizing tools like Phoenix) for debugging and monitoring model behavior.
• Galileo : Focuses on agent reliability with fast evaluators designed to run inline or across production requests.
• DeepEval (Confident AI) : A pytest-style framework for CI/CD-native LLM testing and synthetic dataset generation.

2. Data Infrastructure

Managing unstructured data pipelines, high-value domain data, and reinforcement learning feedback loops remains strategic:

• Scale AI : Continues to anchor the data-centric stack, supplying crucial RLHF (Reinforced Learning with Human Feedback) and custom data pipelines for frontier builders.
• Qdrant : A high-performance vector search engine built in Rust to handle production-grade retrieval-augmented generation (RAG) and memory indexing.
• SuperAnnotate : Provides advanced data curation, annotation, and pipeline tooling tailored for complex multi-modal and vision AI models.

3. Inference & Routing

With inference accounting for the vast majority of active AI compute, serverless execution and intelligent routing are dominant:

• Together AI : A full-stack decentralized cloud platform delivering fast serverless inference and fine-tuning for open-weights models.
• Fireworks AI : A lightning-fast inference microservice provider optimized for cost-effective, low-latency enterprise token generation.
• Baseten : Offers high-performance model serving infrastructure that lets engineering teams deploy and scale custom models with full code control.
• OpenRouter : Operates as an independent model routing gateway, aggregating multi-provider access under a single unified API endpoint.

4. Memory & Hardware-Adjacent Infrastructure

Overcoming the "memory wall" and bandwidth limitations for larger context windows and real-time agents has sparked specialized architecture startups:

• Etched : Building application-specific integrated circuits (ASICs) designed explicitly to run Transformer models without traditional GPU bottlenecks, bypassing general-purpose hardware limits.
• Ayar Labs : Developing optical I/O chip-to-chip interconnect solutions using light instead of electricity to radically scale data movement and memory bandwidth.
• Positron AI : Developing purpose-built hardware and software systems (like Atlas and Asimov silicon) focused strictly on tackling memory bandwidth walls during transformer inference.

5. Safety, Guardrails & Security

As models act autonomously via agents, security has expanded to combat prompt injection, data exfiltration, and model abuse:

• Goodfire : Pioneers model interpretability, allowing developers to peer into the inner workings of black-box neural networks.
• Gray Swan AI : Focuses on advanced frontier-lab red-teaming and safety guardrails engineered specifically for the agent era.
• Noma Security : Provides comprehensive posture management and security guardrails protecting enterprise AI pipelines and supply chains.
• WitnessAI : Secures enterprise interactions with generative AI by inserting policy controls and governance at the application and gateway boundary.

If you'd like to dive deeper, let me know which of these layers (Evals, Data, Inference, Memory, or Safety ) you want to explore further, or if you need help mapping a specific startup's funding or technical architecture.

Yes. If you think of the frontier-model lab as the new “application,” there’s a pretty interesting picks-and-shovels layer forming around it. The important shift in 2026 is that the stack is moving beyond “LLM APIs + vector DBs” toward post-training data, agent environments, inference economics, persistent state, and continuous safety/evaluation.

Here’s how I’d map the startups I’d be watching.

The ones I find most strategically interesting

1. Mercor — data is becoming an interface to the real world.
This is probably the most interesting evolution in the data category. Mercor isn't just supplying annotators; it describes its business as building benchmarks, human datasets and RL environments for frontier labs. Its environments increasingly let agents interact with realistic tools and workflows rather than merely answering static questions.

That matters because the bottleneck for the next generation of models may be less “more internet tokens” and more “more high-quality experience of doing difficult things.”

2. Braintrust — the “GitHub Actions for AI quality” thesis.
Braintrust is moving toward a full development loop: traces → datasets → evals → regression tests → production monitoring. Its pitch is essentially that evals become infrastructure rather than a research exercise.

I think this category becomes particularly important for frontier labs selling models to developers: every model release creates an enormous downstream evaluation problem.

3. Patronus AI — potentially more interesting than a conventional eval SaaS.
Patronus has expanded from LLM evaluation into agent evaluation, synthetic environments and world models. Its June 2026 Series B announcement explicitly positioned the company around simulating intelligence and training/evaluating agents.

That makes it a good example of where the stack is going: evals → environments → training data → oversight.

4. Baseten / Fireworks / Together — inference is becoming its own cloud.
This is one of the clearest infrastructure races. Baseten, Fireworks and Together are effectively competing to become the AWS/GCP layer specifically optimized for model inference. Recent industry comparisons put them at the center of the production-inference market, alongside Modal, Replicate, OpenRouter and others.

The interesting part isn't simply GPU access. It's optimization of batching, quantization, scheduling, caching, model compilation, cold starts and routing—all of which can dramatically change inference gross margins.

5. Letta / Zep / Mem0 — memory is trying to become a primitive.
The emerging memory companies are making different architectural bets:

• Letta: explicit, agent-controlled tiered memory.
• Zep/Graphiti: temporal knowledge graphs.
• Mem0: managed persistent memory that extracts and retrieves useful facts.
• Supermemory: managed cross-session context.
• Cognee: graph/data-pipeline-oriented memory.

A 2026 comparison identifies essentially this group as the core independent memory market.

The bigger question is whether memory becomes an independent infrastructure layer—or gets swallowed by the model providers' agent runtimes.

6. Lakera / HiddenLayer / Protect AI — safety is converging with security.
This is another category I'd watch closely. Lakera is focused heavily on prompt injection, malicious context and runtime protection; HiddenLayer covers AI discovery, supply-chain security, attack simulation and runtime defense; Protect AI spans model selection/testing through runtime.

That's important because agentic systems blur the boundary between “AI safety” and “cybersecurity.” An agent that can browse, retrieve documents, call tools and execute code has an attack surface much closer to an operating system than a chatbot.

One company that spans almost the whole stack

Scale AI is unusual because it touches data + human evaluation + benchmarks + safety + post-training. Its Scale Labs initiative now explicitly covers agents, post-training, reasoning, safety, evaluation and data science.

That makes Scale an interesting bellwether for where the infrastructure market is going: the boundaries between data provider, evaluator and training environment are disappearing.

My rough “watch list”

If I were trying to understand where the next $100B of AI infrastructure value might accrue, I'd start with:

Data / experience

1. Mercor
2. Scale
3. Surge
4. Gretel

Evals / oversight

1. Braintrust
2. Patronus
3. Arize
4. Galileo

Inference

1. Baseten
2. Fireworks
3. Together AI
4. Modal
5. OpenRouter

Memory

1. Letta
2. Zep
3. Mem0
4. Supermemory

Safety / security

1. Protect AI
2. HiddenLayer
3. Lakera
4. CalypsoAI

And there's a sixth category I'd add that you didn't mention: RL environments / agent training infrastructure. That's potentially the most important emerging layer. The thesis is that once frontier labs exhaust the easy gains from static web data, they need environments where models can act, fail, receive rewards, use tools, and learn from trajectories. Current activity around Mercor, Patronus and startups such as Mechanize points strongly in that direction.

The meta-thesis: the winning infrastructure companies may not be the ones selling compute to the labs. They may be the ones selling the labs the feedback loop: generate experience → collect data → train → evaluate → deploy → observe failures → generate better data → retrain.

That's a much bigger market than “LLM observability” as the category was defined two years ago.

The AI infrastructure ecosystem supporting frontier labs and enterprise deployments has shifted dramatically. With inference spending overtaking training in the cloud and the rise of autonomous agent architectures, specialized startups have emerged across the critical layers of the stack.

1. Evaluations & Observability (Evals)

Evaluating probabilistic models, tracking behavioral drift, and running automated red-teaming have become vital platform layers.

• Braintrust – Provides enterprise-grade evaluation and data infrastructure for tracking LLM performance, logging, and continuous experimentation.
• LMArena (Large Model Systems Organization) – Scaled significantly as the go-to benchmark infrastructure for comparative model evaluation.
• Mindgard – Specializes in automated security testing and continuous red-teaming specifically targeted at production AI systems.

2. Data Infrastructure & Labeling

High-cost reasoning models require clean, specialized, and synthetic data to push past the limits of public web scraping.

• Scale AI – The dominant incumbent and data-centric infrastructure platform powering RLHF (Reinforcement Learning from Human Feedback) for top frontier labs.
• Mercor – AI-native platform specialized in automated global data collection, vetting, and labeling.
• Surge AI – Provides high-end data labeling, RLHF data services, and semantic data curation for complex reasoning tasks.

3. Inference & Compute Orchestration

Maximizing throughput, lowering time-to-first-token, and dealing with specialized silicon constraints are the core focus here.

• Together AI – Provides decentralized and dedicated GPU cloud infrastructure optimized for massive model training and scalable fast inference.
• Fireworks AI – Highly performant production inference engine optimized for low-latency serverless model execution.
• Baseten – High-throughput model serving platform that gives engineering teams absolute control over inference infrastructure.
• Infinity – Focuses on agentic and automated low-level compute generation to make alternative AI accelerator chips inference-ready rapidly.
• Crusoe – Builds energy-first "AI factories," combining power infrastructure with streamlined serverless fine-tuning and fast inference stacks.

4. Memory & Context Management

Stateless foundation models face a hard wall with multi-turn conversations and long-running workflows, prompting a dedicated memory layer.

• Mem0 – Building persistent, cross-session smart memory infrastructure designed specifically for AI agents.
• Letta (formerly MemGPT) – Operates on an operating-system paradigm for LLMs, giving agents virtualized long-term memory and self-editing context management.
• Zep – Offers high-performance temporal memory and knowledge graphs engineered for conversational AI and autonomous agents.

5. Safety, Security & Governance

As agents gain tool-use capabilities and interact with internal corporate networks, runtime security and guardrails have become crucial.

• Noma Security – Provides a unified platform for continuous discovery, governance, and posture management for AI and autonomous agents.
• HiddenLayer – Delivers real-time runtime security and protection against adversarial manipulation directly inside the inference loop.
• Pillar Security – Focuses on end-to-end AI Security Posture Management (AI-SPM) across the development and production lifecycle.
• Aurascape – Specializes in securing emerging agentic attack vectors, notably addressing risks around the Model Context Protocol (MCP).

If you're exploring a specific layer, let me know if you would like to dive deeper into the economics of inference vs. training or look closer at the security stack for multi-agent systems.

If you mean the picks-and-shovels companies that sit directly around frontier-model labs—supplying the data, evals, inference stack, memory, and safety machinery—I’d map the 2026 landscape roughly like this.

The important shift is that the stack is moving upward from GPUs toward the “model production system”: proprietary data, post-training/RL environments, inference optimization, persistent context, and adversarial testing. Inference in particular is becoming a major economic control point; Gartner estimates inference will account for 55% of AI-optimized IaaS spending in 2026.

1. Evals / model testing

Most interesting:

• Braintrust — evals + tracing + datasets + production observability. Increasingly looks like the CI/CD layer for AI. Its customers describe eval-driven development as the AI analogue of test-driven development.
• Patronus AI — especially interesting for frontier/agent evals. Its newer approach is to create simulated “digital worlds” where agents have to perform realistic jobs rather than merely answer benchmark questions.
• Arize AI — observability/evaluation, with Phoenix as an important open-source component.
• Confident AI — developer-oriented evaluation infrastructure.
• Langfuse — open-source tracing/evals/LLMOps.

Thesis: the interesting company isn't necessarily another benchmark provider. It's the continuous evaluation system of record: every model version, prompt, agent trajectory, human judgment, regression and production failure flows through it.

---

2. Data / post-training / RL environments

This may be the most strategically important category.

• Mercor — expert data, human evaluation, benchmarks and RL environments. Mercor says it is already used by the top five AI labs and is explicitly positioning itself around frontier data rather than commodity labeling.
• Surge AI — high-end human feedback and training data; particularly important because frontier labs increasingly need expert-quality post-training data.
• Scale AI — still enormous, spanning labeling, RLHF, evaluation and government/enterprise work.
• Handshake AI — another increasingly important data supplier.
• Turing — expert data and model-training talent.
• Deccan AI — newer entrant focused on post-training/evaluation work, including coding and tool-use data. It raised $25M in 2026.

There's a fascinating structural point here: the training-data market appears to be concentrating around Scale, Surge, Mercor and Handshake, with estimates putting their combined share above 75%.

My bet: the next really valuable layer isn't generic labeling. It's proprietary environments that generate training signal—simulated workplaces, software environments, scientific environments, games, robots, browsers, etc. Whoever owns the environment can generate essentially unlimited fresh post-training data.

---

3. Inference

This is probably the hottest infrastructure layer right now.

• Baseten — production inference/post-training infrastructure. It raised $1.5B in June 2026 at a $13B valuation.
• Fireworks AI — optimized inference for open/custom models; raised $1.5B at a reported $17.5B valuation in 2026.
• Together AI — open-model training/inference; its new IBM deal is a $240M infrastructure commitment involving ~2,000 Blackwell GPUs.
• Modal — developer-friendly compute/inference abstraction.
• Replicate — model deployment/API layer.
• OpenRouter — increasingly important routing rather than compute ownership.
• Inferact — commercialized around the vLLM ecosystem.
• RadixArk — similar story around SGLang.

The emerging architecture looks like:

GPU → inference runtime → model host → routing/gateway → application

rather than every AI company simply calling OpenAI/Anthropic APIs. A 2026 industry survey already identifies Baseten, Together, Fireworks, Modal, Replicate, OpenRouter and others as distinct layers of this market.

The really interesting startups here are the ones improving tokens/$ and tokens/sec, not merely reselling GPU capacity.

---

4. Memory / context

This is earlier, but potentially enormous.

• Letta — descended from MemGPT; treats memory almost like an operating system resource that agents explicitly manage.
• Mem0 — managed memory layer for persistent user/agent context.
• Zep — particularly interesting for temporal memory; its Graphiti technology represents changing facts over time.
• Supermemory — managed persistent context, increasingly aimed at coding/agent workflows.
• Cognee — graph-oriented memory/data processing.

The architectural split is already becoming clear: Mem0 is relatively lightweight fact memory, Letta is agent-managed/tiered memory, and Zep/Graphiti is a temporal knowledge graph.

My contrarian take: vector databases aren't necessarily the winner here. The valuable primitive may become “state”—what an agent knows about a user, task, organization, environment and previous actions, including when each fact was true.

If agents become persistent workers rather than stateless chat sessions, this layer could become as important as databases are to conventional software.

---

5. Safety / adversarial infrastructure

I'd split this into model safety and AI security.

• Gray Swan — probably one of the most interesting frontier-lab-facing companies here. It performs adversarial evaluation/red-teaming and turns that intelligence into production defenses. It raised a $40M Series A in May 2026 and says its platform is used by Anthropic, OpenAI and Meta.
• Lakera — prompt injection, jailbreak and AI application security.
• HiddenLayer — model security, AI threat detection, red teaming and protection against model/data attacks.
• Protect AI — model/pipeline security.
• WitnessAI — governance/security around enterprise AI use.
• CalypsoAI — model/application security and guardrails.

The market is converging on continuous adversarial testing → vulnerability discovery → guardrails → monitoring, rather than a one-time “AI safety review.” Gray Swan is a particularly clean example of this loop.

---

The companies I'd watch most closely

If I had to reduce the whole thing to a shortlist of companies with potentially fundamental infrastructure positions, I'd start here:

The deeper pattern

I think there are three particularly attractive primitives emerging:

1. Data → feedback loops

The winners don't just sell datasets. They create an environment → collect agent behavior → evaluate it → generate better training data → improve the model → repeat.

2. Inference → economics

Inference is becoming the equivalent of the cloud-compute bill for AI. Whoever can make a model 2× cheaper/faster has a very tangible economic moat. The recent Baseten/Fireworks/Together fundraising and infrastructure commitments are evidence of how aggressively capital is moving into this layer.

3. Evals + safety → trust

As models become agents, “does this answer look good?” becomes inadequate. Labs need to know what the system can do, where it fails, whether it can be attacked, and whether a new model release regresses on anything important.

That's why I find Braintrust + Patronus + Gray Swan especially interesting: they sit on the measurement/verification side of the stack, which could become the equivalent of testing, observability and security for the AI-native software era.

And I'd put Mercor in a category of its own: if frontier models increasingly run out of useful internet-scale training data, the company that can manufacture new, expert, high-signal training environments could become one of the most strategically important suppliers to the labs.