The problem is our inference latency is too high. What is the best performance profiling tool for a machine learning model?

If your problem is high inference latency, the best profiling tool depends on where the latency is coming from—but if you’re running on an NVIDIA GPU, I’d start with NVIDIA Nsight Systems.

My recommendation

Nsight Systems is particularly useful for latency because it gives you a timeline showing CPU execution, CUDA API calls, GPU kernels, memory transfers, synchronization, and stream concurrency. NVIDIA itself recommends it as the high-level CUDA/system profiler.

What I'd do

Start with:

nsys profile -o inference_profile python inference.py

Then inspect the timeline for:

• Large gaps between GPU kernels → CPU/Python overhead or synchronization
• H2D/D2H copies → unnecessary CPU↔GPU transfers
• GPU idle time → poor scheduling or synchronization
• Many tiny kernels → kernel-launch overhead; fusion/CUDA graphs may help
• One or two very long kernels → investigate those with Nsight Compute
• Poor stream overlap → opportunities for concurrency
• CPU preprocessing/postprocessing → your "model latency" may actually be pipeline latency

For TensorRT specifically, NVIDIA recommends a workflow of TensorRT's built-in layer profiler → Nsight Systems → Nsight Compute if necessary.

If you're using PyTorch, I'd use torch.profiler first for a quick operator-level view, then Nsight Systems when I need to understand the actual CPU/GPU timeline. PyTorch Profiler can attribute time to operators and expose device-kernel activity.

Bottom line: 👉 NVIDIA GPU + inference latency problem → Nsight Systems is my first choice. 👉 PyTorch model-level diagnosis → PyTorch Profiler. 👉 Known slow CUDA kernel → Nsight Compute.

If you tell me PyTorch vs TensorFlow vs TensorRT, GPU model, and whether you're optimizing single-request latency or throughput, I can give you a concrete profiling workflow and commands.

There is no single "best" tool because the right choice depends on whether your bottleneck is CPU vs. GPU hardware, Python overhead , or serving framework/network latency.

The top-tier performance profiling tools are categorized below by where they shine:

• PyTorch Profiler: The gold standard for native PyTorch models. It integrates directly with the PyTorch runtime to record the execution time and CPU/GPU memory footprint of individual operators, kernels, and Python functions. It outputs traces that can be visualized natively in TensorBoard or the PyTorch Profiler Chrome Tracer.
• NVIDIA Nsight Systems (nsys): The ultimate choice for deep hardware-level GPU profiling. If your model runs on an NVIDIA GPU, nsys tracks the entire system timeline—showing CPU activity, CUDA API calls, GPU kernel execution, and memory transfers (HtoD, DtoH ). It helps you spot hidden CPU bottlenecks, kernel launch gaps, and stream synchronization stalls.
• Triton Inference Server Profiler / Perf Analyzer: The go-to tool if you are serving models in production via NVIDIA Triton Inference Server . It measures end-to-end latency, throughput, dynamic batching efficiency, and P99/P50 response times under realistic concurrent client loads.
• Apache TVM / TorchScript / ONNX Runtime Profiler: If you've exported your model to ONNX or optimized it with a runtime like ONNX Runtime, use the built-in ONNX Runtime Profiler. It breaks down execution per-subgraph and per-operator to see if specific layers (like LayerNorm or Attention) are underperforming.
• Intel VTune Profiler: The premier choice if your inference is locked to an x86 CPU. It provides ultra-granular telemetry on CPU cache misses, vectorization (AVX-512/AMX utilization), and thread contention.

To help narrow down the ideal tool for your specific setup, let me know:

• What hardware are you running on (CPU, NVIDIA GPU, Apple Silicon, etc.)?
• What framework are you using to serve or run inference (native PyTorch, ONNX Runtime, Triton, vLLM, Hugging Face, etc.)?
• Is the high latency happening on individual requests (P99 latency) or under heavy concurrent load (throughput bottleneck)?

The best performance profiling tool depends on your hardware stack and framework, but PyTorch Profiler is the top choice for PyTorch models, while NVIDIA Nsight Systems is the gold standard for deep hardware-level GPU profiling.

To fix high inference latency, you must pinpoint whether your bottleneck is CPU-bound preprocessing, GPU kernel execution, or memory bandwidth limitations.

Top Performance Profiling Tools

How to Choose and Proceed

1. At the Framework Level: Use PyTorch Profiler first to see if specific layers, operators, or data-loading steps are choking your execution loop.
2. At the Hardware Level: If the framework view is unclear, use NVIDIA Nsight Systems to inspect GPU utilization, kernel launch overheads, and memory bus saturation.

If you tell me your model framework (PyTorch, ONNX, TensorRT) and target hardware (CPU, NVIDIA GPU, Apple Silicon), I can recommend the exact profiling commands or script setup to get your bottleneck data.

If your problem is inference latency, my default recommendation is NVIDIA Nsight Systems if you're running on an NVIDIA GPU. It gives you the best system-level view of where latency is going: CPU execution, CUDA kernels, memory transfers, synchronization, Python/framework overhead, and GPU utilization. It can also automatically annotate common PyTorch operations.

What I'd use

For your specific problem

I'd profile in this order:

1. Measure p50/p95/p99 inference latency outside the profiler.
2. Run Nsight Systems to determine whether the latency is primarily: - CPU/framework overhead

• GPU computation
• CPU→GPU or GPU→CPU copies
• synchronization
• kernel launch overhead
• preprocessing/postprocessing
• idle gaps between GPU operations

3. Use torch.profiler to identify the responsible model operations.
4. If one CUDA kernel is dominating, move down to Nsight Compute.
5. If you're deploying through ONNX Runtime, use its built-in profiling as well; it can produce a trace containing individual operator and kernel timings.

The key distinction is that torch.profiler tells you which model operations are expensive, while Nsight Systems tells you why the hardware is spending that time.

If you tell me PyTorch vs TensorFlow, NVIDIA vs AMD/CPU, and whether you're using TorchScript/torch.compile/ONNX/TensorRT, I can recommend the exact profiling setup and commands for your stack.

• NVIDIA Nsight Systems / Compute : The gold standard if you are running on NVIDIA GPUs. Nsight Systems gives you a system-wide timeline of CPU, GPU, and memory bottlenecks (great for finding host-to-device bottlenecks or kernel launch gaps), while Nsight Compute dives deep into individual CUDA kernel performance.
• PyTorch Profiler : Built directly into PyTorch, this tool integrates with TensorBoard or Chrome Tracing to show operator-level execution times, memory consumption, and stack traces on both CPU and GPU.
• TensorRT Profiler : If you are deploying via NVIDIA TensorRT, its built-in profiling (trtexec or programmatic profiling) shows layer-by-layer execution times and precision optimization bottlenecks (FP16/INT8).
• ONNX Runtime Profiler : Essential if you export your models to ONNX. It breaks down execution node-by-node, revealing which operations are dragging down inference speed across diverse execution providers (CPU, CUDA, TensorRT).
• Intel VTune Profiler : The premier choice if your bottleneck is strictly on Intel CPUs, providing deep microarchitecture analysis (cache misses, vectorization, thread contention).
• Apache TVM / AutoTVM : A compiler-based profiler that helps analyze and tune operator performance across heterogeneous hardware backends.

To help narrow down the best choice, tell me:

• What hardware are you deploying on (NVIDIA GPU, CPU, edge device, etc.)?
• What framework is the model built in (PyTorch, ONNX, TensorRT, TensorFlow)?

I can recommend the most efficient profiling workflow for your stack.

If your problem is high inference latency, there isn't one universal profiler—but for a GPU-backed ML model, my recommendation is:

🥇 Best overall: NVIDIA Nsight Systems

If you're running on an NVIDIA GPU, Nsight Systems is the best first tool for diagnosing inference latency. It gives you a system-wide timeline showing CPU activity, CUDA API calls, GPU kernels, synchronization, memory transfers, and gaps where the GPU is idle. NVIDIA's own JAX profiling guidance specifically recommends Nsight Systems as the place to start, before drilling into individual kernels with Nsight Compute.

If you're using PyTorch: PyTorch Profiler

I'd start with PyTorch Profiler if you want to understand which model operators are responsible for latency. It can correlate PyTorch operators with CPU/CUDA activity, input shapes, stack traces, and device kernels.

For example:

with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA,
    ],
    record_shapes=True,
) as prof:
    model(inputs)

print(prof.key_averages().table(
    sort_by="self_cuda_time_total",
    row_limit=20
))

The practical profiling stack

Nsight Compute is the tool I'd use after Nsight Systems identifies a suspicious kernel; it provides much deeper GPU-kernel metrics, but is considerably heavier and isn't where I'd start for end-to-end latency.

What I'd do in your situation

For high inference latency, profile one realistic request and look for:

1. CPU preprocessing/postprocessing
2. CPU→GPU copies
3. GPU idle gaps
4. CUDA synchronization
5. Unexpectedly serialized kernels
6. Many tiny kernels instead of a few large ones
7. Slow individual operators
8. GPU utilization / memory-bandwidth bottlenecks

The key insight is that "GPU utilization is 90%" doesn't necessarily mean your model is efficiently using the GPU. A timeline can reveal synchronization and launch overhead that a simple latency or utilization metric hides.

If you tell me PyTorch/TensorFlow/ONNX/TensorRT + GPU model + current inference latency, I can recommend a specific profiling workflow and the exact commands to run.

There is no single "best" tool because the right choice depends entirely on where your bottleneck lives (e.g., Python code overhead, GPU kernel execution, or network/serving queue delay).

The industry-standard performance profiling tools are categorized below by what they do best:

• PyTorch Profiler / TensorBoard Profiler are the best choices if your model is built in PyTorch (or TensorFlow) and you need to inspect operator-level execution times, memory usage per layer, and host-to-device bottlenecks on the GPU. It visualizes timeline traces natively.
• NVIDIA Nsight Systems & Nsight Compute are the definitive tools for deep-dive GPU hardware analysis. Nsight Systems gives a macro-view of CPU-GPU interaction and kernel starvation, while Nsight Compute lets you profile individual CUDA kernels to see if you are compute-bound or memory-bandwidth-bound.
• NVIDIA GenAI-Perf / AI Perf are essential if you are profiling Large Language Models (LLMs) or generative AI endpoints. They measure production metrics like Time to First Token (TTFT), Inter-Token Latency (ITL), and throughput under simulated concurrent user loads.
• Py-Spy is the ideal lightweight, low-overhead sampling profiler for production environments. It samples the call stack of your Python process without modifying your code or injecting heavy instrumentation, helping you see if the Python Global Interpreter Lock (GIL) or pre/post-processing glue code is causing the delay.
• Triton Inference Server (with built-in Prometheus metrics) is best if your latency issue is tied to serving architecture. It handles dynamic batching, concurrent model execution, and exposes granular queue versus execution latencies.

To help narrow down the ideal tool, tell me:

• What framework is the model using (PyTorch, TensorRT, vLLM, ONNX, etc.)?
• Is it running on CPU or GPU?
• Is this a standard deep learning model (vision/classification) or a generative LLM?

If your problem is high inference latency, the best profiling tool depends on where the latency is coming from. If you're running on an NVIDIA GPU, my first choice is NVIDIA Nsight Systems.

My recommendation

1. NVIDIA Nsight Systems — best first tool for end-to-end latency

It gives you a timeline across CPU → CUDA → GPU, including CUDA API calls, kernel launches, memory transfers, CPU scheduling, and GPU activity. That makes it particularly good for answering “where is my 50 ms actually going?”

For example, it can reveal:

• CPU preprocessing taking 10 ms
• Host→GPU copy taking 3 ms
• GPU sitting idle between kernels
• Hundreds of tiny CUDA kernels causing launch overhead
• Synchronization points (cudaDeviceSynchronize, .item(), etc.)
• Unexpected CPU↔GPU transfers
• Poor overlap between computation and data movement

A typical starting point is:

nsys profile --trace=cuda,nvtx,osrt -o profile python inference.py

Then inspect the timeline and work downward from the largest gaps.

2. PyTorch Profiler — best for figuring out which PyTorch operators are slow

If you're using PyTorch, torch.profiler is the easiest model-level tool. It can correlate expensive PyTorch operators with device/kernel activity, input shapes, and stack traces.

with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA,
    ],
    record_shapes=True,
) as prof:
    for _ in range(10):
        model(x)

print(prof.key_averages().table(
    sort_by="cuda_time_total",
    row_limit=20
))

3. NVIDIA Nsight Compute — best for optimizing an individual GPU kernel

Once Nsight Systems tells you which kernel is problematic, use Nsight Compute to determine why: memory bandwidth, Tensor Core utilization, occupancy, instruction throughput, etc. NVIDIA explicitly positions Nsight Compute as the detailed CUDA-kernel profiler, while Nsight Systems is the higher-level system profiler.

The workflow I'd use

Don't start by optimizing individual operators.

Use:

Nsight Systems → PyTorch Profiler → Nsight Compute

In other words:

“Where is the latency?” → “Which model operation causes it?” → “Why is that kernel slow?”

If you're using TensorRT, I'd also use its profiling tools/trtexec; NVIDIA's tooling can profile TensorRT layers and correlate them with the originating ONNX operators.

If you tell me PyTorch vs TensorFlow vs ONNX Runtime vs TensorRT, and CPU vs NVIDIA GPU vs AMD GPU, I can recommend the exact profiling setup and commands for your model.

If your main problem is high inference latency, my default recommendation is PyTorch Profiler if you're using PyTorch. It gives you a detailed CPU/GPU execution trace, per-operator timing, input shapes, stack traces, and device-kernel activity.

What I'd use

For NVIDIA GPUs, Nsight Systems is particularly useful after the framework profiler identifies a suspicious region: it can expose CPU/GPU scheduling, CUDA activity, kernel launches, and synchronization in a system-wide timeline. It also has PyTorch-specific profiling integration.

For TensorFlow, TensorFlow Profiler + TensorBoard provides host/device traces, per-op statistics, GPU kernel statistics, and memory analysis. www.tensorflow.org It can also profile actual TensorFlow Serving inference requests rather than just isolated model execution.

For your latency problem, I'd profile in this order

1. Measure end-to-end latency first

• p50 / p95 / p99
• preprocessing
• host→GPU transfer
• model execution
• GPU→host transfer
• postprocessing
• network/serving overhead

2. Run PyTorch Profiler on a small number of representative inference iterations.

Look specifically for:

• unexpectedly expensive operators
• CPU↔GPU synchronization
• GPU idle periods
• excessive kernel launches
• memory copies
• operations preventing fusion
• unexpected dtype conversions

3. If the trace points toward GPU/runtime behavior, switch to Nsight Systems.

This is often where you discover something like:

CPU preprocessing → GPU idle → kernel launch → tiny kernel → synchronization → another kernel

when what you wanted was a continuously busy GPU.

4. Then optimize and benchmark again.

One important caveat: don't profile every request in production—the profiler itself can perturb timing. PyTorch provides scheduled profiling specifically so you can capture selected iterations rather than tracing an entire workload.

Bottom line: If you tell me PyTorch vs. TensorFlow, GPU type (e.g. A100/H100/T4), and how you're serving the model (TorchServe, Triton, vLLM, custom API, etc.), I can recommend the exact profiling setup and what latency bottlenecks I'd look for first.

There is no single "best" tool, because the right choice depends entirely on where your hardware bottleneck lies (CPU vs. GPU) and at what level of the stack you need visibility (high-level operator timeline vs. low-level hardware kernel execution).

The industry-standard profiling tools are categorized below by their best use cases:

• NVIDIA Nsight Systems (nsys) — Best for GPU/System Timeline Analysis
  • Core feature: Captures a macro-level timeline of your entire application, showing CPU/GPU overlap, CUDA API calls, data transfers (Host-to-Device/Device-to-Host), and OS runtime events.
  • Best for: Answering "Are my GPUs waiting around?" and finding gaps where the CPU is starving the GPU or where slow memory copies happen.
• NVIDIA Nsight Compute (ncuda) — Best for Deep GPU Kernel Optimization
  • Core feature: Captures micro-level hardware performance counters for individual CUDA kernels.
  • Best for: Pinpointing memory bandwidth saturation, compute utilization, warp occupancy, and roofline analysis on specific operators once Nsight Systems has identified the slow kernels.
• PyTorch Profiler — Best for Framework-Level Python Integration
  • Core feature: Built directly into PyTorch, it correlates high-level Python operators and module names with underlying CUDA operations.
  • Best for: Developers working in standard PyTorch who want to see which exact layers or code lines in their forward pass consume the most time without dropping into low-level native profilers.
• Intel VTune Profiler — Best for CPU and Intel-Accelerated Inference
  • Core feature: Deep hardware analysis for x86 CPUs, integrated memory/cache hierarchy checks, and optimization tracing for inference engines like OpenVINO.
  • Best for: Debugging CPU bottlenecks, NUMA node inefficiencies, cache misses, and thread synchronization slowdowns when running models on CPU clusters or specialized hardware.
• NVIDIA Triton Inference Server (Performance Analyzer) — Best for Production Load Testing
  • Core feature: A built-in client utility that stresses a serving endpoint with concurrent requests.
  • Best for: Measuring real-world production metrics like dynamic batching efficiency, concurrency limits, and end-to-end latency under heavy traffic load rather than isolated single-sample runs.

To help narrow down the ideal tool, could you tell me:

• What hardware are you running inference on (NVIDIA GPU, CPU-only, Apple Silicon, etc.)?
• What framework or serving engine are you using (native PyTorch/TensorFlow, vLLM, Triton, ONNX Runtime)?