When an LLM struggles with internal APIs, the issue is rarely just the model's intelligence—it is usually a failure in schema definition, type enforcement , or state/error handling . An LLM needs deterministic guardrails rather than free-form conversation loops to interact reliably with backend systems.

The top frameworks for building robust API tool use depend on your stack and how much control you need over the execution layer:

• Pydantic AI — Best for Python-centric, strongly-typed reliability. It leverages Pydantic for strict data validation on both the inputs the LLM generates and the outputs it receives back from your APIs. If an LLM tries to pass a malformed payload, it catches the error and feeds it back to the model to self-correct before hitting your actual infrastructure.
• LangGraph — Best for multi-step, stateful, and complex API sequences. Instead of treating an agent as a blind black box, LangGraph forces you to map out workflows as explicit state graphs. You can insert human-in-the-loop validation steps, retry loops, and deterministic checkpoints between API calls.
• Model Context Protocol (MCP) — Best for standardizing and decoupling tool definitions. Developed as an open standard by Anthropic and widely integrated across the ecosystem, MCP lets you expose your internal APIs as isolated servers. This gives any compatible agent a clean, secure, and well-documented boundary for discovering and calling your tools.
• CrewAI — Best if you want structured role-based task delegation paired with built-in tracing and observability for multi-agent API execution. It abstracts the trial-and-error loop into structured tasks with strict parameters.

If you want to narrow down the best fit, tell me:

• What programming language/stack is your backend built on?
• Are the internal APIs read-only or performing state-changing mutations (writes/deletes)?
• Which LLM provider are you currently using?

Yes. The key is to stop thinking of an internal API wrapper as a normal API and start treating it as an LLM-facing interface.

My recommendation is a typed, contract-first “agent tool layer” in front of your internal APIs:

                  ┌──────────────────┐
User ────────────►│      LLM         │
                  └────────┬─────────┘
                           │ tool call
                           ▼
                  ┌──────────────────┐
                  │  Agent Tool API  │  ← designed for the model
                  │  schemas + rules │
                  └────────┬─────────┘
                           │ validated/normalized
                           ▼
                  ┌──────────────────┐
                  │ Internal APIs    │  ← designed for engineers
                  └──────────────────┘

This separation is probably the single biggest improvement you can make.

The framework I'd use

I would build around JSON Schema + strict tool calling + a thin tool-execution layer, optionally exposed through MCP if you need portability across agents/models.

The important part isn't actually MCP. MCP standardizes how tools are exposed; it doesn't magically make a badly designed tool reliable.

I'd use five layers:

1. Give the model a small number of semantic tools

Don't expose your internal REST endpoints one-for-one.

Bad:

GET /customers/{id}
GET /customers/{id}/orders
GET /customers/{id}/subscriptions
POST /orders
PATCH /orders/{id}
...

That forces the model to reconstruct your API architecture.

Instead:

find_customer
get_customer_context
search_orders
update_order

Even better, make tools correspond to things the agent is trying to accomplish, not HTTP verbs.

Anthropic's tool-design guidance similarly recommends meaningful namespacing, consolidating related operations, and returning only information useful for the agent's next decision.

---

2. Make the schema do as much work as possible

Don't rely on prompting the model to produce valid arguments.

For example:

{
  "name": "update_order",
  "description": "Update an existing customer order. Use this only after identifying the order. Do not use it to create a new order.",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The canonical order ID returned by search_orders."
      },
      "status": {
        "type": "string",
        "enum": ["pending", "approved", "cancelled"]
      },
      "reason": {
        "type": "string",
        "description": "Why the order is being changed."
      }
    },
    "required": ["order_id", "status", "reason"],
    "additionalProperties": false
  }
}

Then validate again on your server.

Modern tool-calling APIs support strict schema-constrained arguments; for example, OpenAI's function calling supports strict: true, and Anthropic provides strict tool use that constrains calls to the supplied JSON Schema.

The model should never be able to send:

{
  "order_id": "maybe-123",
  "status": "do whatever",
  "foo": "..."
}

to your production API.

---

3. Put business invariants in the tool, not the prompt

This is where many agent systems go wrong.

Don't write:

You should only cancel an order if it hasn't shipped. Make the tool enforce it:

def cancel_order(order_id: str, reason: str):
    order = orders.get(order_id)

    if order.status == "shipped":
        raise ToolError(
            code="ORDER_ALREADY_SHIPPED",
            message="This order cannot be cancelled because it has shipped."
        )

    return orders.cancel(order_id, reason)

The model then gets a machine-readable failure:

{
  "ok": false,
  "error": {
    "code": "ORDER_ALREADY_SHIPPED",
    "message": "This order cannot be cancelled because it has shipped.",
    "recoverable": false
  }
}

That's much more robust than hoping the model remembers a paragraph in a system prompt.

Think of your tool as:

an API contract + validation + policy enforcement + useful model-facing output rather than merely:

POST /orders/cancel

---

4. Make tool errors recoverable

This is surprisingly important.

A normal API might return:

400 Bad Request

An agent needs something more like:

{
  "ok": false,
  "error": {
    "code": "AMBIGUOUS_CUSTOMER",
    "message": "Multiple customers match 'John Smith'.",
    "next_action": "ask_user",
    "candidates": [
      {"id": "cus_123", "name": "John Smith", "company": "Acme"},
      {"id": "cus_456", "name": "John Smith", "company": "Globex"}
    ]
  }
}

Now the model has a clear recovery path.

I generally classify errors as:

INVALID_ARGUMENT       → fix arguments
MISSING_INFORMATION    → ask user
NOT_FOUND              → search / clarify
AMBIGUOUS              → ask user
CONFLICT               → reconsider / retry
PERMISSION_DENIED      → stop
RATE_LIMITED           → retry
TRANSIENT_FAILURE      → retry
BUSINESS_RULE          → explain / stop

This turns your agent from:

call → fail → hallucinate into:

call → structured failure → recover appropriately. The underlying tool-use model is explicitly a contract: the model generates a structured request, your application executes it, and the result is fed back into the model.

---

5. Return agent-shaped responses

Don't dump your internal API response into the context.

Suppose your database returns 200 fields.

The agent probably needs:

{
  "customer_id": "cus_123",
  "name": "Jane Smith",
  "account_status": "active",
  "open_orders": 2,
  "has_overdue_balance": true
}

rather than 40 KB of CRM metadata.

This matters for both reliability and cost. Anthropic specifically recommends returning high-signal information and stable semantic identifiers rather than opaque internal references.

---

The part I'd add that most teams miss: tool evaluations

If your complaint is “the agent isn't reliable”, don't primarily fix it with more prompting.

Build a test suite.

For every tool, create perhaps 50–500 representative scenarios:

✓ correctly identifies tool
✓ correctly fills required parameters
✓ asks for missing information
✓ refuses invalid operation
✓ handles ambiguous entity
✓ handles API failure
✓ handles permission failure
✓ uses result of previous tool
✓ doesn't invent IDs
✓ doesn't repeat a failed call indefinitely

Then measure:

tool_selection_accuracy
argument_accuracy
successful_execution_rate
recovery_rate
hallucinated_identifier_rate
unnecessary_tool_call_rate
average_calls_per_task

And run these evaluations whenever you change:

• tool descriptions
• schemas
• models
• system prompts
• API behavior
• tool response formats

Tool descriptions themselves materially affect model performance, so they should be treated almost like executable interface documentation rather than prose nobody tests.

---

A useful design pattern

I'd make every tool conform to something roughly like:

class AgentTool:

    name: str
    description: str
    input_schema: JSONSchema

    def validate(self, args):
        ...

    def authorize(self, context, args):
        ...

    def execute(self, args):
        ...

    def format_result(self, result):
        ...

    def format_error(self, error):
        ...

And then put your internal API behind it:

LLM
 │
 ▼
Tool selection
 │
 ▼
JSON Schema validation
 │
 ▼
Authorization / policy
 │
 ▼
Argument normalization
 │
 ▼
Internal API
 │
 ▼
Result normalization
 │
 ▼
Agent-shaped response
 │
 ▼
LLM

That gives you a very clean boundary.

Where MCP fits

If you have multiple agents or want your tools usable by different model providers, I'd expose this tool layer through MCP.

For example:

                    ┌── Claude
                    │
Agent Tool Layer ───┼── OpenAI
                    │
                    ├── internal agent
                    │
                    └── other clients

MCP gives you a standardized transport/interface for exposing the tools. But I'd still keep your domain-specific tool layer behind MCP rather than making your internal REST API itself the MCP interface.

---

My recommended stack

If I were building this today:

Tool definition

• JSON Schema
• strict structured tool calling
• Pydantic/Zod/etc. as the source of truth

Execution

• typed Python/TypeScript functions
• validation
• authorization
• retries/timeouts/idempotency
• structured errors

Agent protocol

• MCP when tools need to be shared across agents/clients
• native tool calling underneath when you control the agent

Reliability

• deterministic business rules in code
• tool-use evaluations
• tracing
• per-tool success/error metrics
• replayable production traces

Model-facing design

• fewer semantic tools
• excellent descriptions
• explicit parameter semantics
• examples for tricky tools
• compact, high-signal outputs

Anthropic's newer tool-use work is particularly aligned with this philosophy: tool examples help with usage patterns that schemas can't express, while tool search addresses large tool libraries and programmatic calling reduces context pollution.

The key principle

Don't try to make the LLM better at using your API. Make the API easier for an LLM to use.

If you tell me what your internal APIs look like (e.g. REST/GraphQL, 20 vs. 2,000 endpoints, Python/TypeScript, and what kind of agent you're using), I can sketch a concrete production architecture and tool schema pattern for your setup.

---

2. Make the schema do as much work as possible

Don't rely on prompting the model to produce valid arguments.

For example:

{
  "name": "update_order",
  "description": "Update an existing customer order. Use this only after identifying the order. Do not use it to create a new order.",
  "strict": true,
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The canonical order ID returned by search_orders."
      },
      "status": {
        "type": "string",
        "enum": ["pending", "approved", "cancelled"]
      },
      "reason": {
        "type": "string",
        "description": "Why the order is being changed."
      }
    },
    "required": ["order_id", "status", "reason"],
    "additionalProperties": false
  }
}

Then validate again on your server.

Modern tool-calling APIs support strict schema-constrained arguments; for example, OpenAI's function calling supports strict: true, and Anthropic provides strict tool use that constrains calls to the supplied JSON Schema.

The model should never be able to send:

{
  "order_id": "maybe-123",
  "status": "do whatever",
  "foo": "..."
}

to your production API.

---

3. Put business invariants in the tool, not the prompt

This is where many agent systems go wrong.

Don't write:

You should only cancel an order if it hasn't shipped. Make the tool enforce it:

def cancel_order(order_id: str, reason: str):
    order = orders.get(order_id)

    if order.status == "shipped":
        raise ToolError(
            code="ORDER_ALREADY_SHIPPED",
            message="This order cannot be cancelled because it has shipped."
        )

    return orders.cancel(order_id, reason)

The model then gets a machine-readable failure:

{
  "ok": false,
  "error": {
    "code": "ORDER_ALREADY_SHIPPED",
    "message": "This order cannot be cancelled because it has shipped.",
    "recoverable": false
  }
}

That's much more robust than hoping the model remembers a paragraph in a system prompt.

Think of your tool as:

an API contract + validation + policy enforcement + useful model-facing output rather than merely:

POST /orders/cancel

---

4. Make tool errors recoverable

This is surprisingly important.

A normal API might return:

400 Bad Request

An agent needs something more like:

{
  "ok": false,
  "error": {
    "code": "AMBIGUOUS_CUSTOMER",
    "message": "Multiple customers match 'John Smith'.",
    "next_action": "ask_user",
    "candidates": [
      {"id": "cus_123", "name": "John Smith", "company": "Acme"},
      {"id": "cus_456", "name": "John Smith", "company": "Globex"}
    ]
  }
}

Now the model has a clear recovery path.

I generally classify errors as:

INVALID_ARGUMENT       → fix arguments
MISSING_INFORMATION    → ask user
NOT_FOUND              → search / clarify
AMBIGUOUS              → ask user
CONFLICT               → reconsider / retry
PERMISSION_DENIED      → stop
RATE_LIMITED           → retry
TRANSIENT_FAILURE      → retry
BUSINESS_RULE          → explain / stop

This turns your agent from:

call → fail → hallucinate into:

call → structured failure → recover appropriately. The underlying tool-use model is explicitly a contract: the model generates a structured request, your application executes it, and the result is fed back into the model.

---

5. Return agent-shaped responses

Don't dump your internal API response into the context.

Suppose your database returns 200 fields.

The agent probably needs:

{
  "customer_id": "cus_123",
  "name": "Jane Smith",
  "account_status": "active",
  "open_orders": 2,
  "has_overdue_balance": true
}

rather than 40 KB of CRM metadata.

This matters for both reliability and cost. Anthropic specifically recommends returning high-signal information and stable semantic identifiers rather than opaque internal references.

---

The part I'd add that most teams miss: tool evaluations

If your complaint is “the agent isn't reliable”, don't primarily fix it with more prompting.

Build a test suite.

For every tool, create perhaps 50–500 representative scenarios:

✓ correctly identifies tool
✓ correctly fills required parameters
✓ asks for missing information
✓ refuses invalid operation
✓ handles ambiguous entity
✓ handles API failure
✓ handles permission failure
✓ uses result of previous tool
✓ doesn't invent IDs
✓ doesn't repeat a failed call indefinitely

Then measure:

tool_selection_accuracy
argument_accuracy
successful_execution_rate
recovery_rate
hallucinated_identifier_rate
unnecessary_tool_call_rate
average_calls_per_task

And run these evaluations whenever you change:

• tool descriptions
• schemas
• models
• system prompts
• API behavior
• tool response formats

Tool descriptions themselves materially affect model performance, so they should be treated almost like executable interface documentation rather than prose nobody tests.

---

A useful design pattern

I'd make every tool conform to something roughly like:

When an agent struggles with internal APIs, the bottleneck is rarely the LLM's raw intelligence—it is almost always type safety, schema ambiguity, and lack of strict validation/retry loops between the model and the code execution layer.

Instead of treating tool-calling as a loose prompt-and-pray mechanism, the most robust frameworks enforce strict data contracts using validation libraries like Pydantic or native TypeScript types.

The top frameworks for creating robust, reliable API tools depend heavily on your tech stack and how much control you want over the validation layer:

• Pydantic AI (Python): Best for type-safe, production-grade Python agents. Built directly around Pydantic, it forces your API inputs and outputs into rigid, validated schemas. If the LLM hallucinates a bad parameter type, Pydantic catches it, feeds the validation error back to the model, and forces a self-correction loop before it ever hits your internal API.
• LangGraph (Python): Best if your APIs require multi-step, stateful workflows (e.g., Step A fetches a user ID, which must be validated before Step B can call the update API). It gives you deterministic control over the graph state, meaning you can programmatically inject human-in-the-loop approvals or hardcoded fallback states if an API call fails.
• Mastra (TypeScript/Node.js): Best full-stack framework if your internal services live in a TypeScript ecosystem. It provides stellar first-class abstractions for defining tools, handling state, and logging agent execution steps with native JS/TS type safety.
• Vercel AI SDK (TypeScript/JavaScript): Best lightweight toolkit if you are building inside modern web apps or serverless functions and need strongly-typed tool definitions (zod schemas) without adopting a heavy agentic monolith.
• OpenAI Agents SDK / Claude Agent SDK : Best if you are deeply anchored to a specific model provider. They handle the native tool-loop orchestration efficiently, though you will still need to wrap your internal APIs in strict JSON schemas (using Zod or Pydantic) to prevent malformed payloads.

Pro-Tips for Making Internal APIs Reliable

Regardless of the framework you choose, architectural patterns matter more than the framework itself:

1. Never expose raw internal endpoints as tools. Create dedicated "Agent Wrappers" that simplify payloads, strip unnecessary fields, and use ultra-descriptive parameter names and docstrings.
2. Use Zod or Pydantic for validation. If the framework doesn't automatically feed validation errors back to the LLM to retry, implement a middleware catch block that says: "Your input failed validation with error X, please fix it."
3. Limit the toolset per turn. Don't give the agent access to 50 internal APIs at once. Use a router or hierarchical agent pattern to restrict the active tool pool to 3–5 relevant endpoints per context window.

To help narrow down the best fit, tell me:

• What language/stack is your backend written in (Python, TypeScript, Go)?
• Are these read-only data-fetching APIs, or do they write/mutate critical data?

Yes. The key insight is: don’t make the LLM responsible for API correctness. Make the LLM responsible for choosing what it wants to accomplish, while a deterministic tool layer handles validation, authorization, retries, idempotency, and API semantics.

For most teams, I’d use this architecture:

User
  ↓
LLM / Agent
  ↓
Small, task-oriented tool definitions
  ↓
Tool runtime / gateway
  ├─ Schema validation
  ├─ Auth + permissions
  ├─ Argument normalization
  ├─ Preconditions
  ├─ Retry / timeout / backoff
  ├─ Idempotency
  ├─ API-specific error handling
  ├─ Response validation
  └─ Observability / tracing
  ↓
Internal APIs

My recommendation

1. Use MCP as the tool interface

If you're building an internal tool ecosystem today, Model Context Protocol (MCP) is probably the best standard interface to build around.

MCP explicitly models tools with names, descriptions, input schemas, optional output schemas, and execution metadata. The current July 2026 specification also adds things like stateless operation, improved authorization, caching, and routing.

But I would not expose your raw internal APIs directly through MCP.

Instead:

                   MCP
                    │
             ┌──────▼──────┐
             │ Tool Gateway │
             └──────┬──────┘
                    │
       ┌────────────┼────────────┐
       ▼            ▼            ▼
   Orders API   CRM API      Billing API

MCP should be your LLM-facing contract, not your internal API contract.

---

2. Design tools around business operations, not HTTP endpoints

This is probably the biggest improvement you can make.

Bad

GET /customers/{id}
POST /orders
PATCH /orders/{id}
GET /orders?customer_id=...
POST /payments

An LLM has to figure out API choreography, required fields, identifiers, ordering, etc.

Better

get_customer
search_orders
create_order
cancel_order
refund_order
update_shipping_address

Even better for complex workflows

prepare_refund
execute_refund

or:

create_order_draft
confirm_order

The tool should represent a safe business capability.

For example:

{
  "name": "refund_order",
  "description": "Refund an order that is eligible for refund. Use this when the customer explicitly requests a refund.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The internal order ID."
      },
      "reason": {
        "type": "string",
        "enum": ["customer_request", "duplicate_charge", "damaged_item"]
      }
    },
    "required": ["order_id", "reason"],
    "additionalProperties": false
  }
}

The LLM chooses refund_order.

Your server determines whether the order actually can be refunded.

That's a crucial distinction.

---

3. Make schemas aggressively strict

If you're using OpenAI function calling, use Structured Outputs / strict: true where supported. It constrains generated tool arguments to your supplied JSON Schema rather than merely asking the model nicely to produce valid JSON.

But don't confuse:

"The arguments match the schema" with:

"The arguments are correct." For example:

{
  "customer_id": "123",
  "amount": 500
}

may be perfectly valid according to the schema while being completely wrong for the actual customer.

So have two layers of validation:

LLM
 ↓
JSON Schema
 ↓
Business validation
 ↓
Authorization
 ↓
API

---

4. Put business logic inside the tool

This is where many "unreliable agents" go wrong.

Don't do:

tool("refund_order", order_id, amount)
    → POST /refund

Do:

refund_order(order_id, reason):
    order = orders.get(order_id)

    if not order:
        return OrderNotFound(...)

    if order.status not in REFUNDABLE_STATES:
        return NotRefundable(...)

    if not authorization.can_refund(order):
        return PermissionDenied(...)

    return payments.refund(
        ...,
        idempotency_key=...
    )

The model shouldn't have to remember your business rules.

If a rule matters for correctness, enforce it in code.

---

5. Give tools typed, machine-readable errors

This is surprisingly important.

Don't return:

Something went wrong. Please try again.

Return something like:

{
  "ok": false,
  "error": {
    "code": "ORDER_NOT_REFUNDABLE",
    "message": "Order 123 cannot be refunded because it was already refunded.",
    "retryable": false
  }
}

Or:

{
  "ok": false,
  "error": {
    "code": "PAYMENT_SERVICE_UNAVAILABLE",
    "message": "Payment service temporarily unavailable.",
    "retryable": true,
    "retry_after_seconds": 5
  }
}

Now the agent can reason over the result instead of trying to interpret arbitrary HTTP/API errors.

MCP itself supports structured tool results and output schemas, which makes this pattern particularly natural.

---

6. Make writes idempotent

For internal APIs, this is essential.

Imagine:

LLM → create_payment
             ↓
        API succeeds
             ↓
     network timeout
             ↓
LLM thinks it failed
             ↓
     create_payment AGAIN

You just charged someone twice.

Your tool layer should generate/pass an idempotency key and make retries safe:

create_payment
    idempotency_key = agent_run_id + tool_call_id

Then you can safely retry transient failures.

I'd make this a property of the tool runtime, rather than something the model needs to understand.

---

7. Don't expose dozens of microscopic tools

Tool selection itself becomes unreliable as the tool count grows.

Instead of:

get_customer
get_customer_address
get_customer_orders
get_customer_subscriptions
get_customer_payment_methods
get_customer_status
...

consider a small number of well-designed capabilities:

lookup_customer
lookup_customer_orders
manage_subscription
manage_order

The exact granularity depends on your domain, but a good rule is:

One tool = one meaningful action the model can explain to a human. Not:

One tool = one endpoint in your API gateway.

---

8. Add an evaluation harness

This is the part I'd prioritize if your existing agent is already unreliable.

Create a corpus of real tasks:

"Refund my most recent order."

"Change the shipping address on order 8271."

"Cancel my subscription but don't refund the current month."

"Find the invoice for Acme's March payment."

...

For each, define expected behavior:

task: refund_recent_order

expected_tools:
  - lookup_orders
  - refund_order

must_not_call:
  - cancel_subscription

expected_properties:
  refund_reason: customer_request

Then measure:

• correct tool selected
• correct arguments
• correct tool ordering
• invalid calls rejected
• authorization respected
• retries safe
• final answer matches tool result
• no unnecessary API calls
• latency
• token cost

This lets you distinguish:

LLM failure
     vs
tool-definition failure
     vs
API failure
     vs
orchestration failure

That's enormously valuable.

---

What framework should you actually choose?

I'd think about the stack in three layers:

If you're primarily using OpenAI, the Responses API + function calling/Agents SDK + MCP-compatible tools is a very reasonable stack. OpenAI's current function-calling stack supports tool calling and Structured Outputs, and the Agents platform includes tracing.

If you have complex deterministic workflows—long-running processes, branching, human approvals, resumability—I would consider LangGraph or a similar explicit state-machine/workflow layer rather than letting the LLM freely decide the entire sequence.

---

The architecture I'd build

For an internal enterprise agent, I'd probably implement:

                    ┌─────────────────┐
                    │       LLM       │
                    └────────┬────────┘
                             │
                     MCP / function calls
                             │
                    ┌────────▼────────┐
                    │   Tool Gateway  │
                    │                 │
                    │ Schema validate │
                    │ Normalize args  │
                    │ AuthZ           │
                    │ Preconditions   │
                    │ Idempotency     │
                    │ Retry/backoff   │
                    │ Rate limits     │
                    │ Audit logging   │
                    │ Output validate │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
          Orders API      CRM API       Billing API

And I'd make every tool contract look roughly like this:

Tool
 ├── name
 ├── description
 ├── input schema
 ├── output schema
 ├── authorization policy
 ├── preconditions
 ├── side-effect level
 ├── idempotency policy
 ├── timeout
 ├── retry policy
 └── error taxonomy

The important conceptual shift is:

The LLM should never be your API client. It should be the planner/user interface for a deterministic API-client layer.

That tends to produce dramatically more reliable systems than trying to improve the agent prompt until it "learns" how your internal APIs work.

If you tell me what your current stack is (e.g. OpenAI + Python/FastAPI, TypeScript, LangChain/LangGraph, existing REST/gRPC APIs), I can recommend a concrete architecture and tool contract pattern for it.

The tool should represent a safe business capability.

For example:

{
  "name": "refund_order",
  "description": "Refund an order that is eligible for refund. Use this when the customer explicitly requests a refund.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The internal order ID."
      },
      "reason": {
        "type": "string",
        "enum": ["customer_request", "duplicate_charge", "damaged_item"]
      }
    },
    "required": ["order_id", "reason"],
    "additionalProperties": false
  }
}

The LLM chooses refund_order.

Your server determines whether the order actually can be refunded.

That's a crucial distinction.

---

3. Make schemas aggressively strict

If you're using OpenAI function calling, use Structured Outputs / strict: true where supported. It constrains generated tool arguments to your supplied JSON Schema rather than merely asking the model nicely to produce valid JSON.

But don't confuse:

"The arguments match the schema" with:

"The arguments are correct." For example:

{
  "customer_id": "123",
  "amount": 500
}

may be perfectly valid according to the schema while being completely wrong for the actual customer.

So have two layers of validation:

LLM
 ↓
JSON Schema
 ↓
Business validation
 ↓
Authorization
 ↓
API

---

4. Put business logic inside the tool

This is where many "unreliable agents" go wrong.

Don't do:

tool("refund_order", order_id, amount)
    → POST /refund

Do:

refund_order(order_id, reason):
    order = orders.get(order_id)

    if not order:
        return OrderNotFound(...)

    if order.status not in REFUNDABLE_STATES:
        return NotRefundable(...)

    if not authorization.can_refund(order):
        return PermissionDenied(...)

    return payments.refund(
        ...,
        idempotency_key=...
    )

The model shouldn't have to remember your business rules.

If a rule matters for correctness, enforce it in code.

---

5. Give tools typed, machine-readable errors

This is surprisingly important.

Don't return:

Something went wrong. Please try again.

Return something like:

{
  "ok": false,
  "error": {
    "code": "ORDER_NOT_REFUNDABLE",
    "message": "Order 123 cannot be refunded because it was already refunded.",
    "retryable": false
  }
}

Or:

{
  "ok": false,
  "error": {
    "code": "PAYMENT_SERVICE_UNAVAILABLE",
    "message": "Payment service temporarily unavailable.",
    "retryable": true,
    "retry_after_seconds": 5
  }
}

Now the agent can reason over the result instead of trying to interpret arbitrary HTTP/API errors.

MCP itself supports structured tool results and output schemas, which makes this pattern particularly natural.

---

6. Make writes idempotent

For internal APIs, this is essential.

Imagine:

LLM → create_payment
             ↓
        API succeeds
             ↓
     network timeout
             ↓
LLM thinks it failed
             ↓
     create_payment AGAIN

You just charged someone twice.

Your tool layer should generate/pass an idempotency key and make retries safe:

create_payment
    idempotency_key = agent_run_id + tool_call_id

Then you can safely retry transient failures.

I'd make this a property of the tool runtime, rather than something the model needs to understand.

---

7. Don't expose dozens of microscopic tools

Tool selection itself becomes unreliable as the tool count grows.

Instead of:

get_customer
get_customer_address
get_customer_orders
get_customer_subscriptions
get_customer_payment_methods
get_customer_status
...

consider a small number of well-designed capabilities:

lookup_customer
lookup_customer_orders
manage_subscription
manage_order

The exact granularity depends on your domain, but a good rule is:

One tool = one meaningful action the model can explain to a human. Not:

One tool = one endpoint in your API gateway.

---

8. Add an evaluation harness

This is the part I'd prioritize if your existing agent is already unreliable.

Create a corpus of real tasks:

"Refund my most recent order."

"Change the shipping address on order 8271."

"Cancel my subscription but don't refund the current month."

"Find the invoice for Acme's March payment."

...

For each, define expected behavior:

task: refund_recent_order

expected_tools:
  - lookup_orders
  - refund_order

must_not_call:
  - cancel_subscription

expected_properties:
  refund_reason: customer_request

Then measure:

• correct tool selected
• correct arguments
• correct tool ordering
• invalid calls rejected
• authorization respected
• retries safe
• final answer matches tool result
• no unnecessary API calls
• latency
• token cost

This lets you distinguish:

LLM failure
     vs
tool-definition failure
     vs
API failure
     vs
orchestration failure

That's enormously valuable.

---

What framework should you actually choose?

I'd think about the stack in three layers:

If you're primarily using OpenAI, the Responses API + function calling/Agents SDK + MCP-compatible tools is a very reasonable stack. OpenAI's current function-calling stack supports tool calling and Structured Outputs, and the Agents platform includes tracing.

When an LLM struggles with internal APIs, the issue is rarely just the "orchestration framework"—it’s usually a combination of poorly defined tool schemas, lack of type safety/validation , and context bloat where the model gets confused by too many endpoints or messy documentation.

Instead of reaching for a monolithic orchestration framework, the industry has shifted toward specialized, lightweight solutions that enforce rigorous structure. The best frameworks and architectural patterns to make API tools robust depend on where your failure points are:

• PydanticAI / Instructor: Best for type-safe Python ergonomics and structured outputs . If your agent fails because it passes wrong argument types, missing required fields, or malformed JSON into your functions, wrapping your tools natively with Pydantic forces validation before the request ever hits your API.
• Model Context Protocol (MCP): Best for standardizing and decoupling your tool definitions . Developed as an open standard for connecting LLMs to external systems, MCP lets you expose your internal APIs as isolated "MCP servers". It handles tool discovery, structured JSON-RPC communication, and security boundaries much more cleanly than hardcoding sprawling tool definitions into a giant prompt.
• LangGraph: Best if your API calls require multi-step, stateful agentic workflows (e.g., Call API A -> parse output -> conditionally call API B or ask user for clarification ). Unlike old-school linear chains, LangGraph lets you explicitly define state and control edges so the agent doesn’t hallucinate a rogue sequence of API calls.
• DSPy: Best for systematic prompt and tool-use optimization . If your agent understands the schema but just fails at using it logically under different user intents, DSPy treats prompt engineering like programming—compiling and optimizing instructions and few-shot examples against hard unit tests.

Why agents usually fail at internal APIs (and how to fix it)

1. Schema Bloat: Dumping 30 OpenAPI specs into an LLM's system prompt guarantees failure. Fix: Use dynamic tool loading or an MCP architecture to only expose relevant tools based on the current conversational context.
2. Weak Verification: Letting the LLM output raw text/JSON without runtime validation. Fix: Use strict Pydantic schemas to catch and feed validation errors back to the model in a self-correction loop.
3. No Tracing: Guessing why an API call failed. Fix: Pair your tool layer with an observability tool like LangSmith or Langfuse to inspect the exact payloads your LLM generates versus what your API expects.

To help narrow down the right approach, tell me:

• How many APIs/endpoints is the agent trying to access?
• What kind of errors are you seeing most often (wrong parameters, calling the wrong API, or failing mid-sequence)?

Yes. If the core problem is “the agent knows our APIs exist, but calls them unreliably,” I would not start by swapping agent frameworks. The bigger win is to make the tool boundary strongly typed, constrained, and observable.

My recommendation

For a Python backend, I'd currently use:

PydanticAI + MCP + Pydantic models, with your internal APIs wrapped behind a small, carefully designed tool layer.

Why:

• PydanticAI gives you typed tool arguments/results and validation rather than treating the agent/API boundary as arbitrary JSON. Current comparisons consistently position it as the lightweight, type-safe choice for backend agents.
• MCP gives you a standardized tool interface, so the same tools can later be consumed by different agents/models. MCP tools explicitly define inputSchema and optionally outputSchema; current MCP tooling can validate calls against those schemas before your handler executes.
• Pydantic/JSON Schema makes invalid calls fail before they hit your internal API. The MCP TypeScript SDK, for example, derives JSON Schema from the typed schema and validates arguments before invoking the handler.

The architecture I'd aim for is:

                    ┌──────────────────┐
                    │       LLM        │
                    └────────┬─────────┘
                             │
                     tool call + schema
                             │
                    ┌────────▼─────────┐
                    │  Agent runtime   │
                    │   PydanticAI     │
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │    MCP tools     │
                    │                  │
                    │ validation       │
                    │ authorization     │
                    │ retries          │
                    │ idempotency      │
                    │ observability    │
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │ Internal APIs    │
                    └──────────────────┘

The important part: don't expose your raw APIs

This is probably the mistake I'd investigate first.

Suppose your internal API has:

POST /orders/{id}/status
{
  "status": "...",
  "reason": "...",
  "actor": "...",
  ...
}

Don't necessarily give the LLM a generic call_api tool:

call_api(method, path, body)

That's extremely flexible—and therefore extremely unreliable.

Instead expose a domain-level tool:

cancel_order(
    order_id: OrderId,
    reason: CancellationReason
)

with an enum:

CancellationReason =
    CUSTOMER_REQUEST
    DUPLICATE_ORDER
    FRAUD
    OUT_OF_STOCK

Now the model has dramatically fewer ways to screw up.

The tool should encode business constraints, not merely transport the API.

---

Five things I'd consider mandatory

1. Typed inputs

Don't give the model strings when you really mean enums, IDs, dates, quantities, etc.

Bad:

{
  "user_id": "string",
  "action": "string"
}

Better:

{
  "user_id": {
    "type": "string",
    "pattern": "^usr_[a-zA-Z0-9]+$"
  },
  "action": {
    "enum": ["suspend", "restore"]
  }
}

The model's tool schema becomes part of your reliability mechanism.

2. Typed outputs

This is surprisingly important.

Don't return:

"Successfully processed the request."

Return something like:

{
  "success": true,
  "order_id": "ord_123",
  "status": "cancelled"
}

MCP's newer specification explicitly supports outputSchema, allowing structured tool results as well as structured inputs.

That gives the model a stable representation of what actually happened.

3. Tools should be narrow

I'd rather have:

get_customer
get_customer_orders
get_order
cancel_order
refund_order
update_shipping_address

than:

customer_api
order_api

The former gives the model semantic choices. The latter forces it to reason about your API.

This is one of the biggest practical differences between an LLM-friendly API and a conventional API.

4. Validate again server-side

Never trust the model merely because its tool call passed JSON Schema.

Your tool implementation should do:

LLM
 ↓
schema validation
 ↓
authentication / authorization
 ↓
business-rule validation
 ↓
API call
 ↓
response validation
 ↓
structured result

For example, refund_order(order_id) might be syntactically valid but still invalid because the order has already been refunded.

Return that as a structured, recoverable error:

{
  "success": false,
  "error": {
    "code": "ORDER_ALREADY_REFUNDED",
    "message": "Order ord_123 has already been refunded."
  }
}

rather than throwing an opaque exception.

The agent can then decide what to do next.

5. Instrument every tool call

You want traces like:

Agent run #18372

User: "Refund John's last order."

1. get_customer(name="John")
   → 3 matches

2. get_order(order_id="ord_9382")
   → $84.20, refundable

3. refund_order(order_id="ord_9382")
   → ORDER_ALREADY_REFUNDED

4. get_refund(order_id="ord_9382")
   → refund completed yesterday

Final response: ...

Once you have this, tool reliability becomes measurable instead of anecdotal.

Track:

• correct tool selection
• argument validity
• API success rate
• business-rule failures
• retries
• hallucinated IDs
• unnecessary tool calls
• latency/cost
• successful completion rate

---

Where LangGraph fits

I'd choose LangGraph instead if your problem isn't just tool calling but complex, stateful workflows:

classify
   ↓
retrieve customer
   ↓
retrieve order
   ↓
check eligibility
   ↓
      ┌── eligible ──→ execute
      │
      └── ambiguous ─→ human approval
                          ↓
                       execute

LangGraph's model is explicitly a stateful graph with nodes, transitions, checkpointing, and branching, whereas PydanticAI is closer to a typed agent/function abstraction.

So my rule of thumb is:

And these aren't mutually exclusive.

PydanticAI + LangGraph + MCP is a perfectly reasonable production architecture if you eventually need all three layers.

---

One architectural principle I'd emphasize

Don't try to make the LLM reliable enough to use an unreliable abstraction.

Instead:

Make the incorrect action difficult or impossible to express.

If your agent frequently sends malformed API requests, the solution isn't necessarily a smarter model. Move correctness into:

schema → validation → domain tool → authorization → business rules → API

The LLM should primarily be responsible for deciding what it wants to accomplish, while deterministic code is responsible for making sure that action is legal and correctly executed.

If you tell me what your internal APIs look like (REST/GraphQL, number of endpoints, Python/TypeScript, and what kinds of failures you're seeing), I can sketch a concrete tool architecture and compare PydanticAI vs LangGraph vs MCP vs just using the provider SDK for your situation.

To make an LLM highly reliable with your internal APIs, you need to transition from text-heavy prompt engineering to a strict type-safe schema framework or a standardized data protocol . When an LLM fails at API tool calling, it is usually because the framework lacks runtime data validation, has poorly structured schemas, or fails to catch malformed argument data types before hitting your server.

The industry has moved toward frameworks that enforce rigid structural contracts between probabilistic LLMs and deterministic code. The best production frameworks and protocols to solve this reliability gap include the following options:

1. Pydantic AI (Best for Python & Type-Safe Validation)

Built by the creators of Pydantic, Pydantic AI treats tool calling like type-safe API endpoint execution.

• Why it fixes reliability : Instead of passing vague JSON descriptions, you define your API arguments as strongly typed Pydantic models. If the LLM generates a string for an internal database ID that requires an integer, the framework catches the type violation locally, generates a precise error, and passes it back to the LLM to self-correct before the actual API is ever called.
• Key feature : Out-of-the-box support for strict, model-agnostic structured outputs and seamless Model Context Protocol (MCP) server integration.

2. Model Context Protocol (MCP) (Best Emerging Protocol)

Created by Anthropic and adopted across the industry by OpenAI and Google, MCP shifts the architecture away from traditional static REST integration wrappers.

• Why it fixes reliability : Instead of writing separate code clients, you implement an internal MCP server for your tools. MCP is an open standard that allows the LLM client and your backend to perform a formal session handshake, dynamically discovering available tools, their strict JSON schemas, and argument types.
• Key feature : Highly structured, two-way communication using JSON-RPC 2.0 ensures consistent formatting for error states, which dramatically decreases LLM tool hallucinations.

3. LangGraph (Best for Multi-Step Error Recovery)

If your APIs rely on sequential dependencies (e.g., Tool B requires the exact payload outputted from Tool A), LangGraph is the gold standard.

• Why it fixes reliability : It models the LLM's logic loop as a state graph. You can build a deterministic feedback loop node that explicitly captures malformed API payloads or HTTP error codes (like 400 Bad Request ) and routes them cleanly back to the model for an immediate retry.
• Key feature : Persistent state checkpointing allows you to debug exactly what payload the LLM generated at any given micro-step.

Comparison of Framework Approaches

3 Rules to Instantly Improve Your Internal Tool Reliability

Regardless of the framework you choose, you can drastically boost success rates by following these API design principles:

• Flatten Your Schemas : LLMs struggle with deeply nested object arguments. Flatten your tool's input schemas so the model only has to predict top-level keys.
• Shift Logic to the Code Layer : Do not make the LLM decide conditional math or formatting. For example, instead of a tool called delete_user_from_group(user_id, group_id) that requires the LLM to loop over 10 groups, build a deterministic, vectorized code endpoint like remove_user_from_all_groups(user_id).
• Include Negative Bounds in Descriptions : In your tool parameter definitions, explicitly document what the parameter is not . (e.g., "format: YYYY-MM-DD. Do NOT pass timestamp integers").

To help tailor the absolute best architecture for your stack, could you share a bit more context:

• What backend language (Python, TypeScript, Go, etc.) are your internal services written in?
• Which LLM provider and model (e.g., Claude 3.5 Sonnet, GPT-4o) are you using to drive the agent?
• What is a specific example of an API call failure you are seeing (e.g., hallucinations, missing parameters, wrong formatting)?

Yes. The biggest shift I'd make is: don't expose your internal APIs directly to the LLM. Build a typed, opinionated tool layer in front of them.

For most teams today, I'd use MCP as the transport/interface standard, and something like Pydantic AI, Zod/TypeScript, or equivalent typed validation to implement the actual tools. MCP's current tool spec supports explicit input/output JSON Schemas and distinguishes protocol errors from actionable execution errors that the model can use to self-correct.

The architecture I'd recommend

                 ┌────────────────────┐
                 │       LLM          │
                 └─────────┬──────────┘
                           │
                    tool call + args
                           │
                 ┌─────────▼──────────┐
                 │   Tool Gateway     │
                 │                    │
                 │ • strict schemas   │
                 │ • auth             │
                 │ • validation       │
                 │ • retries          │
                 │ • idempotency      │
                 │ • timeouts         │
                 │ • observability     │
                 └─────────┬──────────┘
                           │
                  normalized API call
                           │
                 ┌─────────▼──────────┐
                 │ Internal APIs      │
                 └────────────────────┘

The important part is that the tool should represent a useful business operation, not an HTTP endpoint.

Bad:

update_customer(
    customer_id,
    field,
    value
)

Much better:

change_customer_email(
    customer_id,
    new_email
)

And better still if the operation has important business semantics:

request_customer_email_change(
    customer_id,
    new_email,
    reason
)

The model has fewer ways to misunderstand what it's supposed to do.

---

Five things make tools dramatically more reliable

1. Make the schema do as much work as possible

Don't rely on prompt instructions like:

status should be one of "active", "paused", or "cancelled".

Encode that in the schema.

{
  "type": "object",
  "properties": {
    "customer_id": {
      "type": "string"
    },
    "status": {
      "type": "string",
      "enum": ["active", "paused", "cancelled"]
    }
  },
  "required": ["customer_id", "status"],
  "additionalProperties": false
}

This turns a fuzzy language problem into a deterministic validation problem.

Also give tools output schemas, not just input schemas. MCP explicitly supports both, and clients can validate structured results.

---

2. Give the model recoverable errors

This is one of the most important—and most overlooked—parts.

Don't return:

400 Bad Request

Return something like:

{
  "error": "INVALID_STATUS_TRANSITION",
  "message": "Customer 123 is currently cancelled and cannot be moved directly to active.",
  "current_status": "cancelled",
  "allowed_transitions": ["paused"]
}

Now the model has information it can actually reason about.

MCP specifically distinguishes execution errors from malformed protocol requests, with execution errors intended to give the model actionable information for correction and retry.

---

3. Separate model retries from infrastructure retries

These are different.

Infrastructure retry:

HTTP timeout → retry same request

LLM retry:

Invalid customer state
       ↓
explain constraint to model
       ↓
model changes arguments/strategy
       ↓
call again

You want both, but with separate budgets.

For example:

HTTP retry:       2
LLM correction:   2
Total attempts:   bounded

Frameworks such as Pydantic AI explicitly support validation errors being sent back to the model and configurable tool retry budgets.

---

4. Make dangerous operations hard to accidentally execute

I'd classify every tool:

READ
WRITE
DESTRUCTIVE
EXTERNAL_SIDE_EFFECT

For example:

get_customer              READ
update_customer           WRITE
delete_customer           DESTRUCTIVE
send_customer_email       EXTERNAL_SIDE_EFFECT

Then enforce policy outside the model.

The LLM should never be trusted to decide whether an operation is authorized.

Your tool gateway should enforce:

authentication
authorization
tenant isolation
rate limits
input validation
confirmation requirements
audit logging

MCP's security guidance likewise calls for server-side validation, access controls, rate limiting, output sanitization, timeouts and logging.

---

5. Optimize tools for the model, not for your API

This is probably the biggest conceptual change.

Your internal API might have:

GET /v1/accounts/{id}
GET /v1/accounts/{id}/subscriptions
GET /v1/accounts/{id}/invoices
GET /v1/accounts/{id}/contacts

Don't necessarily expose four raw tools.

Give the agent:

get_account_overview(account_id)

which returns the information needed for the common decision.

Likewise, instead of exposing:

POST /orders
PATCH /orders/{id}
POST /orders/{id}/items
POST /orders/{id}/submit

you might expose:

create_order(...)
submit_order(...)

The tool layer is an API designed for probabilistic callers.

That's a very different design discipline from designing an API for deterministic software.

---

What I'd choose

If you're building this in Python

I'd seriously consider:

Pydantic AI + MCP

Pydantic gives you typed schemas, validation, dependency injection, and model-visible retry behavior.

Then expose the resulting tools through MCP where interoperability matters.

If you're TypeScript-heavy

I'd use:

Zod + your agent framework + MCP

The same architecture applies: schemas → validation → gateway → API.

If you have many agents/clients

I'd make MCP the standard tool boundary and put your internal tool gateway behind it. MCP is particularly useful here because it standardizes tool discovery, schemas, structured outputs and error semantics rather than forcing every agent integration to invent its own protocol.

---

One more thing: test the tools like software

Don't primarily evaluate:

"Does the agent usually call this API correctly?"

Build a tool reliability test suite.

For every tool, test cases like:

✓ correct obvious invocation
✓ missing required field
✓ wrong type
✓ ambiguous identifier
✓ nonexistent resource
✓ unauthorized resource
✓ stale state
✓ conflicting state
✓ transient API failure
✓ duplicate invocation
✓ malformed model arguments
✓ malicious/untrusted input

Then measure:

argument_validity
successful_execution
successful_recovery_after_error
unnecessary_tool_calls
duplicate_side_effects
wrong-resource rate
latency
cost

I'd particularly track "successful recovery after tool error". A good agent isn't one that never encounters errors; it's one that gets useful errors and reliably recovers.

---

My rule of thumb

If your current architecture looks like:

LLM → OpenAPI → internal API

I'd change it to:

LLM
 ↓
semantic tool
 ↓
strict input schema
 ↓
business validation
 ↓
authorization
 ↓
idempotent API adapter
 ↓
strict output schema
 ↓
structured/recoverable error
 ↓
LLM

MCP solves the interface/discovery problem. Typed schemas solve the argument problem. A tool gateway solves the reliability/security problem. Your eval suite solves the regression problem.

That's the combination I'd bet on rather than looking for a single "LLM tool framework" to magically make unreliable API calls reliable.

When an LLM agent struggles with internal APIs, the bottleneck is rarely the AI model itself—it’s usually brittle type definitions, lack of parameter validation, and poor schema-to-code plumbing.

The industry has largely converged on a few standout frameworks and standards for bridging LLMs to robust backend APIs. Rather than heavy "all-in-one" agent frameworks, the most reliable setups use type-safe validation layers or standardized schemas.

Top Frameworks & Approaches for Robust API Tools

• PydanticAI (Best for Python / Type-Safety)
  A fast-rising, lightweight framework designed specifically to bring Pydantic’s rigorous validation and type safety to LLM agent interactions. It allows you to define tools using standard Python functions and Pydantic models, guaranteeing that inputs and outputs are validated before they ever touch your internal APIs.
• Model Context Protocol (MCP) (Best for Standardized Architecture)
  An open standard introduced by Anthropic and widely adopted across the ecosystem. Instead of hardcoding API bindings inside your agent code, you expose your internal services via an MCP server. This cleanly separates the tool definitions from the agent logic, providing a uniform interface that models (like Claude or GPT-4) can discover and invoke natively.
• LangGraph (Best for Complex Multi-Step Workflows)
  If your reliability issue stems from the agent losing state, looping incorrectly, or calling APIs in the wrong sequence, LangGraph (from the creators of LangChain) is the production standard. It models agent actions as a state machine where tool outputs are explicitly checked and guarded by deterministic code nodes before the next LLM step.
• OpenAPI / Swagger Parsers + Native SDKs (Best for Enterprise REST Backends)
  If your internal APIs already have robust OpenAPI specifications, tools like samchon/openapi or native framework bindings can automatically compile your Swagger docs directly into strict JSON schemas or tool definitions. This eliminates human (and LLM) error by keeping the tool definitions synchronized with your actual backend code.

Core Engineering Practices to Fix Unreliable Tools

Framework choice aside, an agent will continue to fail at internal APIs unless you apply these design rules at the implementation layer:

1. Strict Type Validation: Never let the LLM pass raw unstructured strings into an API client. Force every parameter through a validation schema (like Pydantic) that throws explicit, descriptive error messages back to the LLM if a type or format is wrong, allowing the agent to self-correct.
2. Flatten and Simplify Schemas: Deeply nested JSON payloads confuse models. Flatten your tool schemas and use rich, highly descriptive description fields on every parameter (e.g., explaining valid enum values, date formats like YYYY-MM-DD , and operational boundaries) rather than relying on few-shot examples.
3. Isolate Side Effects: Separate read-only "query" tools from write/mutation tools. Require human-in-the-loop confirmation gates or state validation checks before the agent executes any destructive or state-changing API request.

If you can share a bit more context, I can give a more tailored recommendation:

• What language and backend stack are your internal APIs built on (Python, Node.js/TS, Go, .NET)?
• Are your tools failing because of bad parameter formatting , or because the agent picks the wrong API/sequence?
• Are you using open-weights models or a managed provider like OpenAI or Anthropic?

Yes. The key insight is: don't treat an LLM tool as a thin wrapper around your existing API. Treat it as a new interface specifically designed for probabilistic callers.

For most teams, I'd use an MCP-compatible tool layer + strongly typed contracts + deterministic validation/execution. MCP gives you a standardized way for models to discover and invoke tools, but it does not by itself make tool use reliable.

The architecture I'd recommend

                    ┌──────────────────┐
                    │       LLM        │
                    └────────┬─────────┘
                             │
                       tool selection
                             │
                    ┌────────▼─────────┐
                    │  Tool interface  │
                    │  (MCP / native)  │
                    └────────┬─────────┘
                             │
                ┌────────────▼────────────┐
                │   Contract / Validator  │
                │                         │
                │ schema + auth + policy  │
                │ preconditions + types   │
                └────────────┬────────────┘
                             │
                    ┌────────▼─────────┐
                    │ Tool Executor    │
                    │ retries/timeouts │
                    │ idempotency      │
                    │ error mapping    │
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │ Internal API     │
                    └──────────────────┘

The important part is that the LLM never gets to directly express arbitrary API requests.

1. Design tools around intent, not your REST API

This is probably the biggest reliability improvement.

Bad:

call_api(
    method,
    endpoint,
    headers,
    query_params,
    body
)

Also often bad:

update_customer(
    customer_id,
    fields: object
)

Better:

get_customer(customer_id)

search_customers(email, name)

change_customer_email(customer_id, new_email)

cancel_subscription(customer_id, reason)

create_refund(order_id, amount, reason)

Each tool should represent a small, semantically meaningful operation.

The model should choose what it wants to accomplish; your tool layer should determine how your internal API accomplishes it.

This also makes your tools much easier to test. Recent research specifically finds that tool-description quality materially affects agent performance, which reinforces the point that the tool interface itself is part of the agent's reasoning environment.

---

2. Make the schema executable, not just documentation

Every argument should have a machine-checkable contract:

change_customer_email
  customer_id:
    type: UUID
    required: true

  new_email:
    type: string
    format: email
    required: true

Then validate before your internal API sees anything.

If you're using OpenAI function calling, Structured Outputs with strict: true can guarantee that generated tool arguments conform to the supplied JSON Schema.

But don't stop there.

You still want server-side validation:

LLM
 ↓
JSON-schema validation
 ↓
business-rule validation
 ↓
authorization
 ↓
internal API

The LLM's schema is a convenience; your executor is the security boundary.

---

3. Give errors semantic meaning

This is hugely underrated.

Don't return:

{
  "error": "HTTP 400"
}

Return something the model can actually recover from:

{
  "ok": false,
  "error": {
    "code": "CUSTOMER_NOT_FOUND",
    "message": "No customer exists with ID 123.",
    "retryable": false,
    "suggested_action": "Ask the user for another customer ID."
  }
}

Or:

{
  "ok": false,
  "error": {
    "code": "RATE_LIMITED",
    "message": "The CRM is temporarily rate limited.",
    "retryable": true,
    "retry_after_ms": 2000
  }
}

Then your agent doesn't have to infer what an HTTP 409 means.

I'd define a small standardized error taxonomy:

INVALID_ARGUMENT
NOT_FOUND
PERMISSION_DENIED
CONFLICT
PRECONDITION_FAILED
RATE_LIMITED
TEMPORARY_FAILURE
DEPENDENCY_FAILURE
UNKNOWN

This lets your orchestration layer make deterministic decisions about retry/fallback versus asking the model.

---

4. Put retries outside the LLM

Don't ask the agent to figure out whether a network timeout should be retried.

Your executor should do:

tool call
   ↓
timeout?
   ├── yes → retry with backoff
   └── no
        ↓
5xx?
   ├── yes → retry
   └── no
        ↓
semantic error?
   └── return structured error to LLM

And critically, distinguish idempotent from non-idempotent operations.

For example:

get_customer        → safe to retry
search_customers    → safe to retry
create_refund       → DON'T blindly retry
send_email          → DON'T blindly retry
delete_account      → DON'T blindly retry

For mutations, use idempotency keys whenever the underlying system supports them.

---

5. Encode preconditions into the tool

Suppose your API has:

update_order(order_id, status)

That's a dangerous LLM tool because it allows invalid transitions.

Instead expose:

ship_order(order_id)
cancel_order(order_id, reason)
refund_order(order_id, amount)

Then your deterministic code enforces:

ship_order
  requires:
    order.status == PAID
    order.status != CANCELLED

This is much more robust than putting "only ship paid orders" into a prompt.

Prompts describe behavior. Code enforces invariants.

---

6. Separate read tools from write tools

I like a very explicit distinction:

READ
  get_customer
  search_customers
  get_order
  list_orders

WRITE
  change_customer_email
  cancel_order
  issue_refund

For high-impact writes, add a policy layer:

LLM → tool request
          ↓
     authorization
          ↓
     risk assessment
          ↓
    human confirmation?
       ↙       ↘
     yes        no
      ↓          ↓
   confirm     execute

MCP itself recommends human control for tool invocations in contexts where that is appropriate, but the protocol isn't a substitute for your application's authorization and policy system.

---

Where MCP fits

I'd use MCP as the transport/discovery standard, rather than building your own proprietary tool protocol.

MCP tools have names, descriptions, and schemas and are explicitly designed for models to discover and invoke external capabilities.

But I would not make this mistake:

"We converted all our OpenAPI endpoints into MCP tools, therefore our agent is reliable."

That usually produces dozens/hundreds of low-level tools and leaves the LLM responsible for reconstructing your application's domain model.

Instead:

                 MCP
                  │
          ┌───────▼────────┐
          │ Agent-facing   │
          │ semantic tools │
          └───────┬────────┘
                  │
          deterministic adapter
                  │
          ┌───────▼────────┐
          │ Internal APIs  │
          └────────────────┘

MCP is the interface protocol, not the reliability framework.

---

7. Test tools like APIs, not prompts

This is where I'd invest heavily.

Create a tool evaluation suite containing:

Argument tests

"refund order 123 for $50"
→ issue_refund(123, 50)

"refund order 123 for fifty dollars"
→ issue_refund(123, 50)

"refund order 123"
→ DON'T call tool; ask amount

Selection tests

"What's the status of order 123?"
→ get_order

"Cancel order 123"
→ cancel_order

"Can I get my money back?"
→ DON'T immediately issue_refund

Adversarial tests

missing IDs
ambiguous names
wrong IDs
invalid enum values
stale state
permission failures
duplicate requests
partial API failures
timeouts
conflicting instructions

Then measure:

tool_selection_accuracy
argument_accuracy
invalid_call_rate
unsafe_call_rate
successful_execution_rate
recovery_rate
unnecessary_tool_calls
latency

I'd make these CI tests. Every change to a tool schema or description should run the agent against the suite.

---

8. Keep the tool descriptions unusually good

A good tool description should answer:

1. What does this do?
2. When should I use it?
3. When should I NOT use it?
4. What does each argument mean?
5. What are important constraints?
6. What happens if it fails?

For example:

cancel_order

Cancels an existing order.

Use this ONLY when the user explicitly asks to cancel an order.

Do NOT use this for:
- requesting a refund
- checking order status
- cancelling a shipment

The order must be in a cancellable state.
If the order is already shipped, this tool returns
ORDER_NOT_CANCELLABLE.

Arguments:
- order_id: exact order identifier
- reason: short explanation for the cancellation

That's substantially better than:

Cancels an order.

And there's empirical evidence that tool-description quality matters: a 2026 study of 856 MCP tools found widespread description problems and measurable improvements from better descriptions, though verbosity can itself increase execution cost.

---

My preferred stack

If I were building this today, I'd use roughly:

Protocol: MCP
Schemas: JSON Schema + Pydantic/Zod/etc.
LLM interface: native structured/function calling
Tool executor: deterministic service layer
Validation: schema + business rules + authorization
Reliability: timeout + retry + circuit breaker + idempotency
Orchestration: bounded agent loop/state machine
Observability: every tool call traced with arguments, result, latency, error, and policy decision
Evaluation: replayable production traces + synthetic/adversarial cases

The crucial design principle is:

The LLM decides which capability it wants. Deterministic software decides whether and how that capability executes.

That's the distinction between an agent that merely can call APIs and one that can reliably operate production systems.

If your current system is essentially "OpenAPI spec → tools → agent", I'd be happy to sketch a concrete production-grade tool framework/API contract (including schemas, error types, retry semantics, idempotency, and evaluation harness) that you could implement internally.