Our on-call team wants an AI agent that can investigate incidents and propose fixes. What tools are best?

The best tools for on-call AI incident investigation and remediation combine observability platforms, automated root-cause analysis, and repository-connected coding agents.

Top AI Incident Investigation & Remediation Tools

• Grafana Assistant uses Grafana Labs agentic capabilities to query metrics, investigate anomalies, and create collaborative triage workflows.
• Datadog provides built-in LLM-powered assistant capabilities to summarize logs, analyze traces, and surface context directly to engineering teams.
• PagerDuty Advance applies automated noise reduction, event correlation, and generative AI summaries to accelerate incident triage.
• Augment Code or specialized coding agents connect production telemetry to codebases to suggest rollbacks, config tweaks, or pull requests.
• Arize AI and LangSmith offer managed debugging and tracing agents that trace failures in production systems and open direct fixes.

If you share your current monitoring stack (e.g., Datadog, Prometheus, PagerDuty) and code hosting platform (GitHub, GitLab), I can recommend the exact integration path for your team.

Yes. If the goal is “AI investigates the incident, finds likely root cause, and proposes a safe fix”, I’d evaluate tools in this order:

Best options

• datadoghq.com — my first choice if you're already heavily invested in Datadog. Bits Investigation can autonomously investigate alerts across metrics, logs, traces, incident context, and related signals, while Bits Code can generate production-oriented fixes.
• pagerduty.com — strongest fit if PagerDuty is the center of your on-call workflow. It uses incident history, runbooks, diagnostics and logs to investigate, identify likely causes, and recommend or perform approved remediation.
• resolve.ai, incident.io, rootly.com, and similar AI-SRE/incident-management products are worth a bake-off if you want a more specialized agent rather than extending your existing observability platform. A recent 2026 comparison puts these alongside Datadog and PagerDuty as leading AI-SRE approaches.

What I'd actually build

Don't give the agent unrestricted production access. Give it a toolbelt roughly like:

1. Observability: query logs, metrics, traces, profiles and service dependencies.
2. Change intelligence: inspect recent deploys, commits, feature flags and infrastructure changes.
3. Knowledge: search runbooks, architecture docs, previous incidents and postmortems.
4. Incident control: read/update PagerDuty or incident.io and communicate in Slack.
5. Code: inspect the relevant repository and generate a proposed patch.
6. Remediation: execute pre-approved actions such as rollback, restart, scaling or flag changes.
7. Verification: check telemetry after the action and automatically determine whether the incident improved.

The important distinction is propose → approve → execute → verify, rather than simply letting an LLM run shell commands. Recent autonomous-agent incidents are a good reminder that unrestricted agents can create serious security and operational risks.

My recommendation

If you're already on Datadog, start with Bits Investigation + Bits Code. It gives you the shortest path from “we got paged” → “here's the probable cause” → “here's a candidate fix.” Datadog explicitly supports investigation from an incident/Slack context and can surface root-cause findings and recommended next steps.

If you're already on PagerDuty, start with SRE Agent instead; its advantage is the deep integration with the actual on-call lifecycle and incident history.

If you tell me your current stack (e.g. AWS/Kubernetes + Datadog + PagerDuty + GitHub + Slack), I can sketch the agent architecture, permissions, tools/APIs, and guardrails I'd use.

For an on-call team, I’d build around three layers: an incident-management system, an observability/tool-access layer, and a tightly permissioned remediation layer.

Best options

• pagerduty.com — probably the strongest fit if you already use PagerDuty. Its agent can ingest incident data, runbooks, logs and historical incidents, identify likely root causes, and recommend diagnostic/remediation steps. It integrates with tools including Datadog, Grafana, New Relic, CloudWatch, GitHub and Confluence.
• datadoghq.com Bits Investigation — particularly attractive if Datadog is your observability backbone. Its agent autonomously investigates incidents and monitor alerts across telemetry and produces root-cause analyses. Datadog says Bits Investigation can reduce time to resolution by up to 95%.
• pagerduty.com — use this underneath the agent for actual fixes. It turns approved procedures into executable workflows—for example, gather diagnostics, restart a service, roll back a deployment, or perform a failover.

The architecture I'd recommend

Alert → AI investigator → evidence gathering → hypothesis → proposed fix → human approval → runbook execution → verification

Give the agent read access first to:

• Logs and traces
• Metrics/dashboards
• Recent deployments and configuration changes
• Kubernetes/cloud state
• Incident history
• Git repositories and runbooks
• Service ownership/topology

Then give it write access only through predefined, audited tools, rather than arbitrary shell/cloud credentials.

For example:

"Checkout API latency increased at 14:32. Investigate." The agent should automatically correlate the alert with traces, logs, deployment history and infrastructure metrics, conclude something like "95% confidence: the latency increase began immediately after deployment X; database connection-pool exhaustion is the likely mechanism," and propose:

"Roll back deployment X. Expected effect: restore previous connection-pool behavior. Risk: loses changes in X. Verification: p95 latency < 500 ms for 5 minutes." Only after approval should it invoke the rollback tool.

One important design choice

I would not start with an unrestricted autonomous agent. Current agentic systems create a real blast-radius problem: recent industry experience has reinforced the importance of isolation, permissions, monitoring and human oversight for agents that can interact with external systems.

A good maturity path is:

1. Observe: investigate and explain.
2. Recommend: propose specific fixes with evidence.
3. Approve: engineer clicks to execute.
4. Automate: allow low-risk, reversible fixes automatically.
5. Self-heal: only for highly predictable failure modes.

PagerDuty itself describes essentially this "crawl, walk, run" progression, starting with diagnostics and moving toward automated remediation.

My default stack: PagerDuty + Datadog/Grafana + GitHub/Confluence runbooks + Kubernetes/cloud APIs exposed through narrowly scoped tools + an AI agent with strong audit logs and approval gates.

If you're building this in-house rather than buying an SRE-agent product, I can also lay out the specific MCP/tools/API surface I'd give the agent (e.g. search_logs, query_metrics, get_recent_deploys, kubectl_describe, rollback_deployment, verify_recovery).

One important design choice

I would not start with an unrestricted autonomous agent. Current agentic systems create a real blast-radius problem: recent industry experience has reinforced the importance of isolation, permissions, monitoring and human oversight for agents that can interact with external systems.

A good maturity path is:

Building an AI agent to investigate incidents and propose fixes requires a mix of observability data connectors, LLM orchestration frameworks , and secure execution sandboxes . Because on-call environments demand high reliability and strict safety guardrails, tools are typically categorized by where they sit in your stack.

1. Native Observability AI (Best for Out-of-the-Box Triage)

If your telemetry is already centralized in a major observability platform, leveraging their native AI investigators is the fastest route because they already have secure access to your logs, metrics, and traces.

• Datadog Bits Investigation: Best if you use Datadog. It automatically investigates production alerts by pulling in operational context, correlating traces, metrics, logs, and deployment events to pinpoint the root cause.
• Dynatrace: Best for deterministic AI root-cause analysis (Davis AI). It doesn’t just guess; it isolates the exact component or code change causing a degradation across complex cloud environments.
• LogicMonitor Edwin AI: Best for infrastructure-heavy environments. It processes massive alert streams, reduces noise, and assists SREs by mapping cross-functional dependencies during outages.

2. Incident Response & Pager Platforms (Best for Workflow Orchestration)

These platforms don't just find the bug—they coordinate the response, pull in historical postmortems, and trigger remediation runbooks.

• PagerDuty Automation Actions: Excellent for combining AI-driven event intelligence with automated diagnostic scripts. It can surface relevant past incidents and run safe diagnostic commands the moment an alert fires.
• ServiceNow ITSM / ITOM: Best for enterprise governance. It uses embedded AI to map incidents to change requests (e.g., "Did this bad deploy cause the alert?" ) and proposes standardized fixes or knowledge articles.

3. Frameworks for Custom-Built AI Agents (Best for Tailored Fixes)

If your team wants to build a bespoke agent that can read your private runbooks, query internal APIs, and draft pull requests, you’ll want an agent framework rather than a closed product:

• LangChain / LangGraph: Ideal for building multi-step diagnostic loops (e.g., Step 1: Query Prometheus → Step 2: Fetch last 50 error logs from Loki → Step 3: Cross-reference internal runbook vector database → Step 4: Propose a fix).
• LlamaIndex: Best if your primary bottleneck is context retrieval—connecting the agent to your internal documentation, past postmortems, and architecture decision records (ADRs).
• GitHub Copilot Workspace / Custom GitHub Actions Agents: Great for the "propose a fix" portion. An agent can package its findings into a structured draft Pull Request with the corrected code or configuration file ready for a human on-call engineer to review.

To help narrow down the best architecture, tell me:

• What is your primary observability stack (e.g., Datadog, Grafana/Prometheus, AWS CloudWatch)?
• Do you want the agent to only suggest fixes in chat/tickets, or actively execute changes (like rolling back a deployment or scaling a pod)?

Here are top web results for exploring this topic:

Reddit·https://www.reddit.com Thinking about building an AI on-call investigation tool - talk me out ...Thinking about building an AI on-call investigation tool - talk me out of it / tell me what you'd actually want. Vendor / market research. Hi, Solo dev here. I keep getting annoyed during on-call at h Arvo AI·https://www.aurorasre.ai 10 Best AI -Powered Incident Investigation Tools 2026 | Arvo Datadog-only shops should shortlist Bits AI; multi-cloud, regulated, or air-gapped teams need a self-hosted agent; Kubernetes-only teams can start with HolmesGPT. Every tool on this list claims to inv Glean·https://www.glean.com**Best AI tools** for incident response and agent orchestration in 2026 Can the tool trace a null pointer exception across three microservices to the schema change that caused it? Can it generate a backward-compatible patch that won't break the five other services consumi Augment Code·https://www.augmentcode.com**AI Agent Incident** Response: From Alert to Fix - Augment Code Alert triage through AI agents turns raw monitoring events into incidents that engineers can act on. The process normalizes, deduplicates, suppresses, correlates, enriches, and routes signals before t Nurix AI·https://us.nurix.ai**Best AI Agents** for Incident Response Automation in 2026 - Nurix AI Your submission has been received! Oops! Something went wrong while submitting the form. Teams searching for the best AI agents for incident response automation rarely start from scratch. Most already Corelayer·https://www.corelayer.com**Best AI Agents** for Incident Response 2026 - Corelayer July 17, 2026 5 min read. Mitch Radhuber. by Mitch Radhuber. The Best AI Agents for Automated Incident Response in 2026, Ranked. AI agents can now handle the majority of an incident response loop on t Kiro·https://kiro.dev How We Learned to Trust an AI Agent to Triage Production Incidents By 2:46 AM PDT, 13 minutes and 35 seconds later, a defensible diagnosis sat on the ticket: streams stalling silently from either a production bug or capacity scaling, customers affected, every competi DrDroid·https://drdroid.io How to use AI for On-call Investigations | Engineering Tools - DrDroid By continuously monitoring your systems and analyzing trends, AI can predict potential issues before they escalate. This allows you to initiate proactive investigations and implement fixes preemptivel AI Agents Directory·https://aiagentsdirectory.com**Top** 5 Tools That Help AI Agents Fix Production Bugs Automatically Splunk SOAR; 5. PagerDuty; Automatic Bug Fixing Is the Next Evolution of SRE. Production engineering is entering a new phase. As AI-assisted development accelerates release velocity, engineering teams www.bland.ai·https://www.bland.ai/blog/best-ai-customer-support-tools-for-it-teams 15 Best AI Customer Support Tools for IT Teams and Workflows The best AI customer support tools for IT teams streamline ITSM workflows, reduce ticket resolution time, and improve service quality through scalable, automated support systems. These intelligent sol

Yes. For an incident-investigation agent, I’d build around observability + incident management + controlled code execution, rather than buying a generic “AI SRE” product and giving it production access.

Recommended stack

• Incident management: PagerDuty, Rootly, or incident.io. These give the agent incident context, responders, timelines, escalation state, and communication workflows. In 2026, incident.io, Rootly, and PagerDuty are differentiated particularly around Slack-native workflows, automation, and on-call management.
• Observability: your existing Datadog / Grafana Labs / New Relic stack. Expose APIs for logs, metrics, traces, deployments, monitors, and service ownership.
• Source control + CI/CD: GitHub or GitLab, plus your deployment system. The agent should be able to inspect recent commits, diffs, deployments, feature flags, and rollback history.
• Knowledge: runbooks, architecture docs, previous incidents, service catalogs, and postmortems. Retrieval/file-search is particularly useful here.
• Execution: a sandboxed shell/container where the agent can reproduce failures, query data, run tests, inspect repositories, and construct a proposed patch. The current OpenAI Agents SDK supports shell execution, file manipulation, code execution, MCP, and controlled sandbox environments.
• Agent framework: openai.github.io. It supports function tools, MCP, tool search, sandbox execution, agents-as-tools, guardrails, and tracing.

The important design choice

I'd give the agent roughly this tool surface:

Incident
 ├─ get_incident()
 ├─ get_timeline()
 ├─ get_oncall()
 └─ add_investigation_note()

Observability
 ├─ query_logs()
 ├─ query_metrics()
 ├─ query_traces()
 ├─ get_alert_history()
 └─ get_deployments()

Code
 ├─ inspect_repo()
 ├─ search_code()
 ├─ inspect_commit()
 ├─ run_tests()
 └─ create_patch()

Infrastructure
 ├─ get_service_status()
 ├─ inspect_config()
 └─ inspect_recent_changes()

Knowledge
 ├─ search_runbooks()
 ├─ search_postmortems()
 └─ search_architecture_docs()

Then make mutating production actions separate and approval-gated:

rollback_deployment()
change_feature_flag()
restart_service()
apply_config_change()
deploy_patch()

The agent can investigate and prepare the fix automatically, but a human approves the production mutation.

That distinction is critical: the model should have broad read access, constrained write access, and a hard boundary around irreversible/high-blast-radius operations. OpenAI's current agent tooling explicitly supports approval-sensitive tools and guardrails for this kind of workflow.

What I'd build first

For an MVP, I'd choose:

incident.io/Rootly/PagerDuty + your observability API + GitHub + runbook retrieval + sandboxed OpenAI agent.

The agent's workflow would be:

1. Receive alert/incident.
2. Establish impact and affected services.
3. Examine metrics/logs/traces.
4. Correlate with deployments/config/feature-flag changes.
5. Search runbooks and historical incidents.
6. Form competing hypotheses.
7. Run read-only tests to eliminate hypotheses.
8. Identify likely root cause.
9. Produce a proposed fix/rollback.
10. Run tests against the proposed change.
11. Ask for human approval.
12. Execute the approved action.
13. Verify recovery and document the incident.

That is considerably safer and more useful than an agent whose primary capability is “run kubectl and see what happens.”

For a serious production deployment, I'd also instrument the agent itself. The Agents SDK has built-in tracing/observability so you can inspect tool calls and agent behavior, which is especially valuable when debugging the incident investigator itself.

If you tell me your current stack (e.g. AWS + Kubernetes + Datadog + PagerDuty + GitHub), I can narrow this down to a concrete tool architecture and the exact APIs/MCP servers I'd expose.

For an on-call team, the best AI incident agent is usually not a single chatbot. The strongest setups combine:

1. Incident management/orchestration (pages, ownership, timelines, comms)
2. Observability access (logs, metrics, traces, deploys)
3. Knowledge access (runbooks, past incidents, code, architecture docs)
4. Safe remediation tools (scripts, tickets, PRs, rollbacks—with approvals)

Good options depend on where your operational data already lives.

A practical architecture I’d recommend:

Incident trigger

• PagerDuty / Datadog monitors / Kubernetes alerts

↓

AI investigator

• Reads:
  • recent deploys
  • Git commits
  • Kubernetes events
  • logs and traces
  • dashboards
  • previous incidents
  • runbooks

↓

Reasoning output

• “Likely cause”
• “Evidence”
• “Confidence”
• “Suggested next checks”
• “Possible fixes”

↓

Action layer (approval required)

• Create Jira ticket
• Roll back deployment
• Restart unhealthy workload
• Adjust feature flag
• Open a PR
• Update status page

The tools I’d prioritize connecting:

• GitHub — code changes, PRs, ownership
• Kubernetes — runtime state
• Terraform — infrastructure changes
• Slack or Microsoft Teams — incident conversation history
• Your observability platform — logs/metrics/traces

For a first version, I would avoid a fully autonomous “fix production” agent. A safer and usually more valuable first milestone is:

AI investigator → human approves fix → automation executes

The agent should be able to answer:

• “What changed?”
• “What is probably broken?”
• “What evidence supports that?”
• “What are the safest next actions?”

before it ever gets permission to change systems.

A good shortlist by team profile:

• Small engineering team: incident.io or Rootly + your observability stack
• Enterprise SRE team: PagerDuty + Datadog + custom AI tooling
• Platform-heavy Kubernetes shop: Datadog AI capabilities or a custom agent connected to Kubernetes/GitHub/runbooks
• Highly regulated environment: build a permissioned internal agent with audit logs and approval gates

The biggest differentiator is not the LLM; it is the quality of context and the permissions model. An AI agent with incomplete telemetry will confidently guess, while one connected to your operational history can become a useful SRE partner.

Building an AI agent for incident investigation and remediation requires a stack that spans observability, agent orchestration, code/command execution , and ticketing/collaboration.

Here are the best tools categorized by their function in the incident response lifecycle:

• Agent Frameworks & Orchestration
  • LangGraph / LangChain : Excellent for building stateful, multi-agent workflows where an investigator agent hands off tasks to a remediation-proposing agent.
  • CrewAI : Great for role-based multi-agent setups (e.g., assigning specific roles like "Log Investigator," "Metric Analyst," and "SRE Lead").
  • LlamaIndex : Ideal if your agent needs to deeply query internal post-mortems, runbooks, and architecture docs via RAG before troubleshooting.
• Observability & Data Retrieval
  • Datadog LLM Observability / APIs : Allows agents to programmatically query metrics, APM traces, and log streams during an active alert.
  • Grafana / Prometheus APIs : Essential for open-source stacks to let the agent fetch time-series data and query logs (e.g., via Loki or Tempo).
  • Honeycomb : Perfect for high-cardinality data investigation, helping the agent pinpoint the exact outlier dimensions causing a spike.
• Execution & Remediation Testing
  • Temporal : Not an AI tool per se, but critical for orchestrating reliable, durable workflows when your agent triggers automated scripts or multi-step remediation.
  • Port / Cortex : Internal developer portals that allow agents to safely trigger pre-approved remediation actions, restart services, or flip feature flags via secure backstage plugins.
  • Kubernetes API / kubectl wrappers : If your infrastructure is containerized, giving the agent a sandboxed, read-and-safe-write interface to inspect pod logs, restart deployments, or check cluster events.
• Collaboration & Ticketing
  • PagerDuty API : For the agent to automatically enrich an active incident with its preliminary investigation findings and proposed fixes.
  • Jira / Linear : To automatically draft tickets or PR descriptions detailing the root cause analysis and recommended patches.
  • Slack Bolt SDK : To build the bot interface where the on-call engineer can chat with the agent or approve/reject its proposed fixes ("human-in-the-loop").

If you'd like, tell me:

• What observability and cloud stack you currently use (e.g., Datadog, AWS, Kubernetes)
• Whether you want the agent to automatically apply fixes or just propose them for human approval

I can narrow down this list to a specific architectural blueprint for your team.

Building an AI agent for on-call incident investigation and remediation requires a mix of observability integrations, LLM orchestration frameworks, and secure execution environments.

Here are the best tools and categories to assemble an effective on-call troubleshooting agent:

• Observability & Data Sources (The Eyes and Ears): The agent needs real-time access to logs, metrics, and traces.
  • Datadog / New Relic: Essential for querying metrics, APM traces, and error logs via API.
  • Grafana / Prometheus: Great for pulling system performance dashboards and metric alerts.
  • Better Stack / PagerDuty: Used to ingest incident context, alerts, and metadata when a page fires.
• Code & Infrastructure Access (The Hands): To investigate codebases and safely propose fixes.
  • GitHub API / GitLab API: Allows the agent to search repository files, read pull requests, analyze recent commits, and open draft PRs with proposed fixes.
  • Kubernetes API / Cloud Provider SDKs (AWS, GCP, Azure): For checking pod health, restarting services, or viewing container logs (with strict read/execute guardrails).
• Orchestration & Agent Frameworks (The Brains): Frameworks designed to let agents plan, use tools in loops, and maintain context during an outage.
  • LangGraph: Excellent for building stateful, multi-actor agent workflows (e.g., triage -> log analysis -> root cause hypothesis -> fix proposal).
  • LlamaIndex: Strong if your agent needs to ingest internal runbooks, architecture diagrams, and historical post-mortems via RAG (Retrieval-Augmented Generation).
  • CrewAI / AutoGen: Useful if you want specialized multi-agent roles—like a "Log Analyst Agent" passing data to a "Developer Agent."
• Dedicated AI Incident Platforms (Turnkey Solutions): Platforms specifically built for SRE and on-call workflows.
  • Rootly / FireHydrant: Traditional incident management platforms increasingly adding native AI assistant features for summaries and triage.
  • Koro / Kubiya / AccessOwl: Emerging AI platform tools specifically marketed for DevOps and SRE workflows to handle automated runbook execution and Slack/PagerDuty integration.

To help narrow down the ideal stack, tell me:

• What is your primary cloud or infrastructure stack (e.g., Kubernetes, AWS, specific languages)?
• Which observability and ticketing tools (PagerDuty, Datadog, Jira, etc.) do you already use?

For an on-call team, I’d build this as an AI SRE agent sitting on top of your existing observability + incident stack, rather than buying a generic chatbot.

Best options

Rootly is particularly interesting right now: its AI Connectors can pull live evidence from observability, code, infrastructure, feature flags, tickets and documentation, while its AI agent can work from the incident channel.

For a custom agent, the OpenAI Agents SDK provides tool calling, handoffs, guardrails and built-in tracing of model calls, tool calls and agent execution.

The architecture I'd recommend

Give the agent read access broadly, write access narrowly:

Investigate

• Datadog / Grafana / Prometheus → metrics, logs, traces
• Sentry → exceptions
• GitHub/GitLab → recent commits and PRs
• Kubernetes / AWS / GCP → infrastructure state
• Feature flags → recent changes
• PagerDuty / Rootly / incident.io → incident history
• Slack → relevant conversations
• Runbooks / internal docs → known remediation procedures

Reason

1. Identify affected service.
2. Build a timeline.
3. Correlate deploys/config/traffic/errors.
4. Compare against historical incidents.
5. Produce 2–3 hypotheses with evidence and confidence.
6. Recommend the safest remediation.

Act

• First: propose commands/changes.
• Then: require human approval.
• Only automatically execute tightly constrained, reversible actions—e.g. rollback a known deployment—not arbitrary shell commands.

That distinction matters: recent reports have highlighted the risks of autonomous agents escaping their intended boundaries, so production incident agents should have explicit tool permissions, approval gates and auditing.

If I were choosing today

Already using Datadog + Slack: start with Rootly AI.

Want the most Slack-native incident workflow: evaluate incident.io alongside Rootly. Its current platform emphasizes investigation, coordination and automated incident workflows.

Want to build something differentiated: use OpenAI Agents SDK + your observability APIs + GitHub + Kubernetes/cloud APIs, with a strict approval layer. The SDK's tracing is especially useful for seeing why the agent reached a conclusion and what tools it actually invoked.

The key isn't finding the smartest model; it's giving the model excellent incident context and safe, verifiable tools.