My goal is to analyze the latency of each part of our LLM chain. What's the best tracing tool with support for detailed LLM and embedding spans?

The best tracing tools for breaking down chain latency into granular LLM and embedding spans are Langfuse for production-ready open source/cloud observability, Arize Phoenix for open-source RAG and embedding-heavy evaluation, and LangSmith if your stack is deeply integrated with LangChain.

These options leverage standard semantic conventions (like OpenInference or OpenTelemetry) to isolate the exact latency overhead of vector searches, embedding generation calls, and distinct LLM prompt/completion tokens.

Top LLM Tracing Tools Comparison

Recommended Selection Guide

• Choose Langfuse if you need a scalable, self-hostable or cloud solution with robust dashboards tracking latency distributions, token counts, and embedding/LLM pricing side-by-side.
• Choose Arize Phoenix if your chain relies heavily on vector embeddings and retrievers where you need immediate notebook-to-production visibility into semantic search performance and vector retrieval latency.
• Choose LangSmith if you are already building with LangChain or require a tightly integrated environment out-of-the-box.

Are you using a specific framework like LangChain, LlamaIndex , or custom Python/Asyncio code , and do you prefer a self-hosted or managed cloud setup?

For your specific goal—breaking down end-to-end latency across an LLM chain, including LLM calls, embeddings, retrieval, and application logic—I’d pick langfuse.com.

Why Langfuse

• Fine-grained spans: It models individual operations as observations and supports span, generation, embedding, retriever, tool, chain, etc.
• LLM-specific detail: Generation spans capture model information, token usage, cost, input/output, and latency.
• Embedding spans: Embedding calls have their own observation type, so you can distinguish embedding latency from vector DB/retrieval latency.
• Excellent latency visualization: Its trace timeline lets you see nested operations and identify which components dominate total latency.
• TTFT: You can capture time-to-first-token separately from total generation latency, which is particularly useful for streaming LLM applications.
• OpenTelemetry: Langfuse is built around OpenTelemetry, so you aren't locked into its instrumentation model.
• Self-hosting: It's open source and self-hostable if keeping traces inside your infrastructure matters.
• Analytics: Its APIs expose individual observations as well as aggregate latency/usage metrics, making it practical to build your own latency dashboards.

For example, I'd structure a RAG trace roughly like:

Request                         4.8s
├── preprocess                   40ms
├── query embedding             180ms
├── vector search               420ms
├── reranking                   650ms
│   └── reranker LLM            590ms
├── prompt construction          25ms
└── answer generation          3.4s
    ├── TTFT                    720ms
    └── token generation       2.7s

That makes questions like "Why did p95 increase?", "Is the embedding model slow or is the vector DB slow?", and "Are we waiting on TTFT or generation throughput?" much easier to answer.

Alternatives

One particularly important distinction: don't only instrument the LLM calls. For latency analysis, instrument the boundaries between components too—embedding, retrieval, reranking, prompt construction, serialization, network calls, and application code. Langfuse's hierarchical observations are well suited to this.

My recommendation: use OpenTelemetry as the instrumentation layer + Langfuse as the LLM-aware tracing UI/backend. That gives you detailed LLM/embedding spans today without giving up portability later.

When analyzing fine-grained latency across an LLM chain—including specific text generations, embedding lookups, vector database queries, and custom retrieval steps—a few top-tier tracing tools stand out. They explicitly break down execution via nested spans, track token counts, and measure time-to-first-token (TTFT) or embedding vector generation latency.

Here are the best tools for detailed span-level latency tracing, depending on your infrastructure and licensing preferences:

• Langfuse : An open-source (MIT licensed) platform that excels at granular trace visualization. It provides a flexible SDK allowing you to nest generations, embeddings, and custom processing steps into hierarchical spans. You can easily see exact execution times, token counts, and cost per span. It's available as a managed cloud service or self-hosted via Docker for complete data privacy. Check out the Langfuse Documentation to see how decorators and low-level SDKs handle nested spans.
• LangSmith : Built by the creators of LangChain, but entirely framework-agnostic (works fine with native OpenAI/Anthropic calls, LlamaIndex, or custom Python code). It offers deep, out-of-the-box latency breakdowns for every step of a chain or graph execution. Using the @traceable decorator or client SDK, it visualizes timelines, input/output payloads, and embedding latencies cleanly. Explore the LangSmith Overview for details on latency analysis and debugging queues.
• Arize Phoenix : An open-source, local-first evaluation and tracing library geared heavily toward data scientists and engineers working with RAG and embedding chains. Phoenix integrates smoothly with OpenInference to auto-instrument frameworks, giving you deep visibility into embedding model latency and retrieval steps without heavy code refactoring. Visit Arize Phoenix to review its local UI and notebook-friendly tracing.
• Phoenix / MLflow : If you are already tracking metrics via MLflow, its native LLM tracing capabilities capture multi-step chains, token usage, and tool latency natively in a single open-source ecosystem. Review the MLflow Tracing Guide for details on logging multi-step AI interactions.

To help narrow down the ideal choice, let me know:

• Are you looking for a self-hosted/open-source tool or a managed cloud service?
• What framework or libraries (e.g., LlamaIndex, LangChain, raw API calls) is your chain built with?

For your specific goal—breaking down end-to-end latency across an LLM chain, including individual LLM and embedding calls—my first choice would be Arize Phoenix.

Why Phoenix

Arize AI’s arize.com is particularly well suited because it uses OpenTelemetry + OpenInference, with explicit span types for:

• LLM — individual model calls, including latency and token usage
• EMBEDDING — embedding-generation calls
• RETRIEVER — vector/database retrieval
• RERANKER
• CHAIN — your application-level steps
• TOOL / AGENT — tool and agent execution

That means you can get a trace tree like:

request                         1,840 ms
├── query_embedding               82 ms
├── vector_search                 145 ms
├── reranker                      210 ms
├── prompt_construction            12 ms
├── LLM_call_1                    640 ms
├── tool_call                     310 ms
└── LLM_call_2                    441 ms

Phoenix explicitly supports tracing embeddings as their own spans and lets you nest the operations under a parent trace, making latency bottlenecks easy to identify.

How I'd compare the main options

Phoenix wins for your stated requirement because the OpenInference span model is explicitly designed around LLM applications, rather than treating LLM calls as generic application telemetry. It is also open-source and vendor/framework agnostic.

When I'd pick Langfuse instead

langfuse.com would be my second choice, especially if you want a more polished production observability product with prompt management, evaluations, usage/cost analysis, and tracing tightly integrated. Its current SDK is also OpenTelemetry-based and supports accurate latency tracking and nested observations.

Bottom line: if your primary question is “Where exactly are the milliseconds going in my LLM/RAG/agent pipeline?”, start with Phoenix. If the broader goal is “observability + prompt management + evaluations + production analytics,” I'd seriously consider Langfuse.

If you tell me whether your stack is Python or TypeScript and whether you're using LangChain/LangGraph, OpenAI Agents, LlamaIndex, or custom code, I can recommend the instrumentation setup I'd use.

For your specific goal—breaking down end-to-end latency across an LLM chain, including LLM calls, embeddings, retrieval, and custom application steps—I’d pick langfuse.com as the default.

Why Langfuse fits

• Fine-grained hierarchical traces: You can represent each operation as a span/observation and nest them, so you can see exactly where the time goes.
• LLM-specific spans: Model calls are first-class generation observations with model, token, prompt/output, cost, and latency information.
• Embedding/retrieval visibility: Their tracing model explicitly supports retrieval pipelines; their own example shows an embedding operation followed by vector DB search and generation.
• Custom spans: You can instrument preprocessing, chunking, reranking, database calls, network calls, etc., and compare their latency against the actual LLM time.
• OpenTelemetry-based: This is particularly useful if you already have conventional application tracing—you can combine LLM/GenAI spans with normal application spans rather than creating a completely separate observability system.
• Low instrumentation overhead: Langfuse's SDK sends trace data asynchronously and batches it, so tracing isn't intended to add meaningful request latency.
• Latency analytics: You can aggregate latency by model, prompt/version, component, etc., which is more useful for finding systematic bottlenecks than looking at individual traces.

A trace for a RAG request could look roughly like:

Request                         4,230 ms
├── preprocess                    35 ms
├── embed query                  150 ms
├── vector DB retrieval          510 ms
├── reranking                    180 ms
├── prompt construction           40 ms
└── LLM generation             3,160 ms

That makes it very easy to answer questions like "Is our latency actually the LLM, or are retrieval/embedding/network steps responsible?"

Other strong choices

Current 2026 comparisons generally put Langfuse among the strongest choices for open-source tracing, while Braintrust stands out more for evaluation workflows.

My recommendation: instrument your application with OpenTelemetry + Langfuse, rather than relying exclusively on framework-generated traces. That gives you a consistent span tree containing both conventional application latency and LLM-specific information, and keeps you less locked into one observability backend.

If you tell me your stack (e.g. Python/TypeScript + OpenAI/Anthropic + LangChain/LlamaIndex + Pinecone/pgvector), I can recommend the exact instrumentation setup and span hierarchy.

Here are top web results for exploring this topic:

LangChain·https://www.langchain.com 9 LLM Observability Tools for Production AI Agents - LangChain What to look for in an LLM observability tool. Trace capture vs. trace actionability. Most tools can show the basic run record. You can usually see the prompt and response, latency, token counts, and MLflow·https://mlflow.org**Best LLM Tracing Tools** for Multi-Agent Systems in 2026 - MLflow What features make LLM tracing tools effective for multi-agent systems?​. Multi-agent systems create debugging challenges that single-model applications never face. When five agents pass context betwe Galileo AI·https://galileo.ai 7 Best LLM Observability Tools for Debugging and Tracing - Galileo AI What is an LLM observability tool for debugging and tracing. LLM observability tools capture, structure, and visualize the full execution path of LLM applications. They enable engineers to inspect, de Truefoundry·https://www.truefoundry.com 10 Best LLM Observability Tools in 2026 - Truefoundry Best LLM Observability Tools in 2026. Here's a quick overview of the different LLM observability platforms: Tool, Best For, Key Features, Deployment, Open Source ... Reddit·https://www.reddit.com**What** do you use for observability & tracing for LLm powered apps?... apps we've found two layers are needed: normal infra logs/traces for latency, errors, and retries, and a separate quality layer that runs evaluators on the actual model outputs. Only evals + traci www.braintrust.dev·https://www.braintrust.dev/articles/best-llm-tracing-tools-2026**Best LLM tracing tools** for multi-agent systems (2026 review) - Articles Every LLM call, tool invocation, and retrieval step gets logged with full context. The platform is built for teams who need to move fast without breaking things. The tracing interface shows complete e Confident AI·https://www.confident-ai.com Top 5 Tools for Monitoring LLM Applications in 2026 - Confident AI That's the gap LLM monitoring tools fill. They trace prompts, completions, tool calls, and retrieval steps across your AI pipeline — then evaluate whether your application is actually performing well, Coralogix·https://coralogix.com 10 LLM Observability Tools to Know in 2025 - Coralogix Large language model (LLM) observability tools help developers and organizations monitor, analyze, and troubleshoot LLM deployments. Arize AI·https://arize.com Traces and Spans in LLM Orchestration Frameworks: A Deep Dive Traces, Traces represent a single invocation of an LLM application. For example, when a chain is run or a query engine is queried, that is a trace. Another way to think of traces is as a sequence of s DeepEval·https://deepeval.com**LLM Tracing** | DeepEval - The LLM Evaluation Framework Debug with precision: See exactly where and why things fail — whether it's tool calls, intermediate outputs, or context retrieval steps. Run targeted metrics on specific components: Attach LLMTestCase

When you need to break down latency across an LLM chain—specifically isolating time-to-first-token (TTFT), generation speed, vector search/retrieval steps, and embedding generation—the modern landscape comes down to a few standout tracing tools.

Because you are looking for detailed LLM and embedding spans , the best choices are built on OpenTelemetry (OTel) or use specialized GenAI semantic conventions so that embedding models, chunking logic, and prompt calls are cleanly broken down into separate, measurable tree nodes.

Top Recommended Tracing Tools

• Langfuse : The leading open-source (MIT licensed) platform for LLM engineering . It offers native SDK and OpenTelemetry integration that automatically captures nested spans, generations (LLM calls), and custom execution steps . It tracks latency, exact token usage, and cost per span out-of-the-box , and can be easily self-hosted if you have strict data privacy requirements.
• Arize Phoenix : An open-source, notebook-first and OTel-native tool built heavily around vector embeddings and RAG evaluation . If your chain involves heavy embedding steps (e.g., embedding lookups, semantic similarity searches, or reranking), Phoenix provides specialized visualization and evaluation primitives for embedding spaces and retrieval latencies that other general-purpose dashboards approximate.
• LangSmith : The gold standard if your chain or agent graph is built using LangChain or LangGraph . It captures high-fidelity execution trees down to exact parameter values, prompt templates, tool selections, and vector retrievals . It also accepts generic OpenTelemetry input if parts of your chain run outside the LangChain ecosystem.
• Traceloop (OpenLLMetry) : An open-source, pure OpenTelemetry implementation . If your organization already routes infrastructure metrics to Datadog , Honeycomb, or New Relic , Traceloop auto-instruments popular LLM and embedding libraries (OpenAI, Anthropic, Cohere, LlamaIndex) and pipes the spans directly into your existing APM stack without needing an entirely new dedicated LLM dashboard.

Comparison for Latency Breakdown

If you can share which framework or libraries you are using (e.g., raw OpenAI SDK, LangChain, LlamaIndex) and whether you prefer a managed SaaS or self-hosted setup, I can recommend the exact integration snippet or approach for your stack.

For analyzing latency breakdowns across an LLM chain, especially with LLM calls, embedding generation, retrieval, reranking, tool calls, and custom chain steps, I would prioritize tools that expose a true span tree rather than just request logs. The best choice depends on your stack:

My default recommendation

For a new system where your main goal is "where is my latency going?", I would start with Arize Phoenix if you have embeddings/RAG, or Langfuse if you have a broader agent workflow.

Choose Phoenix when:

• You need to see:
  • embedding generation latency
  • vector search latency
  • retrieved chunks
  • reranking steps
  • LLM generation time
• You want OpenTelemetry-compatible traces and AI-specific span semantics.
• You expect to debug retrieval quality as well as latency.

A typical trace you want should look like:

Request (2.8s)
 ├── Query embedding (80ms)
 ├── Vector DB search (120ms)
 ├── Reranker (300ms)
 ├── Prompt assembly (20ms)
 ├── GPT-5 generation (2.1s)
 │     ├── TTFT: 600ms
 │     └── Decode: 1.5s
 └── Post-processing (50ms)

Choose Langfuse when:

• You want a polished production tracing UI.
• You need prompt versions, datasets, evaluations, and cost tracking alongside traces.
• You have a mixed stack (OpenAI SDK + custom code + multiple frameworks).

Choose LangSmith when:

• Your chain is already built with LangChain/LangGraph.
• You want automatic visibility into LangChain nodes with minimal instrumentation.

What I would instrument regardless of tool

Make sure every span records:

LLM spans

• model name/version
• prompt template ID
• input/output token counts
• time to first token (TTFT)
• total generation latency
• streaming duration

Embedding spans

• embedding model
• batch size
• number of documents
• vector dimensions
• latency per batch

Retrieval spans

• vector DB query latency
• top-k
• filters
• number of candidates
• reranker latency

Application spans

• orchestration time
• tool calls
• retries
• queue time

A good long-term architecture is to emit OpenTelemetry/OpenInference spans and choose the backend later. That keeps you from rewriting instrumentation if you switch observability platforms.

If your stack is LangChain/LangGraph, LlamaIndex, OpenAI Agents SDK, or a custom Python/TypeScript chain, the recommendation changes slightly.

For analyzing latency breakdowns across an LLM chain with detailed LLM + embedding spans, I’d shortlist these:

My recommendation for your specific goal: Arize Phoenix

For "where is the latency coming from?" in an LLM chain, Phoenix is probably the strongest default choice because it models the chain as nested spans and has first-class visibility into:

• LLM calls (model, prompts, tokens, parameters, latency)
• embedding generation
• retrieval calls
• retrieved documents
• tool calls
• end-to-end trace timing

It is built around OpenTelemetry/OpenInference, so you can instrument individual operations instead of being locked into one framework.

A typical trace might look like:

request (2.8s)
 ├── query preprocessing (20ms)
 ├── embedding generation (120ms)
 ├── vector DB search (180ms)
 ├── reranker (250ms)
 ├── prompt construction (15ms)
 ├── LLM call (2.1s)
 │    ├── time to first token
 │    └── generation tokens
 └── post-processing (40ms)

That makes it easy to answer questions like:

• "Are embeddings slow?"
• "Is retrieval dominating latency?"
• "Which model call is expensive?"
• "Did a prompt change increase generation time?"

When I’d choose something else

Choose LangSmith if your stack is mostly LangChain/LangGraph. Its trace views are very convenient for agent graphs and LangChain-specific workflows.

Choose Langfuse if you want an open-source, self-hosted product with strong LLM tracing and product analytics workflows.

Choose plain OpenTelemetry + your existing observability stack if your company already runs Datadog, Grafana, Honeycomb, etc. The important part is emitting meaningful LLM spans (model calls, embeddings, retrieval, tools), not just HTTP latency.

Span fields I would make sure you capture

For each LLM span:

• model name/version
• input/output token counts
• prompt template ID
• latency
• time-to-first-token (streaming)
• temperature/max tokens
• retries/errors

For embeddings:

• embedding model
• batch size
• number of vectors
• input length
• embedding latency

For retrieval:

• vector DB latency
• top-k
• reranking latency
• retrieved document IDs/scores

If your chain is RAG + agents, I’d start with Phoenix. If it is mostly LangGraph agents, I’d compare Phoenix and LangSmith side by side.

When analyzing the latency of an LLM chain—especially when you need explicit, nested span support for vector database queries, text chunking, embedding generation, and final generation—the "best" tool depends heavily on your stack and data privacy needs.

The top tracing tools for handling granular LLM and embedding spans feature distinct advantages:

• Langfuse: Best open-source, all-in-one tracing platform. It offers a brilliant hierarchical waterfall view for traces and spans. It explicitly logs embedding calls, retrieval/vector DB steps, and prompt tokens with precise latency attribution. Because it is MIT-licensed and fully self-hostable (powered by ClickHouse and Postgres), you maintain complete data ownership without worrying about per-span cloud cost inflation.
• Arize Phoenix: Best for RAG-heavy pipelines and embedding visualization. Built natively on OpenTelemetry and OpenInference semantic conventions, Phoenix excels at capturing every individual retrieval step, vector search duration, and embedding vector. It is stellar if you want to inspect embeddings, evaluate retrieval relevance locally in a notebook, or push open-standard telemetry to an enterprise backend.
• LangSmith: Best if you are deeply entrenched in LangChain/LangGraph. It auto-instruments chains out of the box with zero configuration, mapping out gorgeous execution trees that show you the exact timing, inputs, and outputs of your embedders and prompts. However, it is closed-source, and its trace-based pricing can scale up quickly under heavy agentic/embedding workloads.
• Braintrust: Best for combining latency tracking with rigorous evaluation loops. It visualizes complex nested span trees (retrieval→right arrow→ embedding→right arrow→ generation) cleanly and maps latency bottlenecks directly against quality scores. It features a very developer-friendly free tier.

If you tell me:

• What framework you are using (LangChain, LlamaIndex, pure OpenAI/Anthropic SDK, etc.)
• Whether you prefer self-hosting or a managed cloud SaaS

I can give you the exact initialization snippet for your preferred tool to capture your embedding spans.