Building Production AI Agents: MCP, Tools & Orchestration

The gap between a Jupyter notebook agent demo and a production system handling real business workflows is wider than most vendors admit. After a year of shipping agent-based systems for enterprise clients, a few patterns have emerged as genuinely useful — and a few have quietly died. Here's what actually works in 2025.

The MCP inflection point

Anthropic's Model Context Protocol (MCP), released late 2024 and now supported by OpenAI, Google DeepMind, and most major agent frameworks, has done for tool integration what LSP did for IDEs: standardized the wire between models and external systems.

Before MCP, each framework (LangChain, LlamaIndex, Semantic Kernel) had its own tool schema. Every integration was rewritten. With MCP, you expose a capability once as a server, and any MCP-compatible client — Claude Desktop, Cursor, your custom agent — can consume it.

A minimal MCP server in Python:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("invoicing")

@mcp.tool()
def get_overdue_invoices(customer_id: str) -> list[dict]:
    """Return unpaid invoices past due date for a customer."""
    return db.query(
        "SELECT id, amount, due_date FROM invoices "
        "WHERE customer_id = %s AND status = 'overdue'",
        (customer_id,),
    )

if __name__ == "__main__":
    mcp.run(transport="stdio")

That's it. This server is now consumable by any MCP client without framework lock-in. For internal platforms, this is the first architectural decision to make: expose your business capabilities as MCP servers, then layer agents on top.

Tool use: the real bottleneck

Model quality on tool calling has converged. Claude 3.5 Sonnet, GPT-4.1, and Gemini 2.5 all handle 20-30 tool schemas reliably. The bottleneck is no longer the model — it's tool design.

Common failure modes we see in audits:

• Overloaded tools: a single manage_customer tool with 15 optional parameters. Models hallucinate arguments. Split into create_customer, update_contact, merge_duplicates.
• Vague descriptions: "Gets data". Rewrite as: "Returns the last 90 days of transactions for a given IBAN. Use only when the user explicitly asks about recent activity."
• No error contracts: tools that throw stack traces. Return structured errors the model can reason about: {"error": "customer_not_found", "suggestion": "try search_customer first"}.

Anthropic's internal guidance is blunt: if a junior engineer can't understand your tool from its schema alone, neither can the model.

Single agent vs. multi-agent: pick with care

Multi-agent orchestration is oversold. Most enterprise use cases are better served by a single agent with well-designed tools. Add agents only when you have genuinely parallel, independent subtasks or clear domain boundaries.

| Pattern | When it fits | Framework examples | |---|---|---| | Single agent + tools | Support triage, data lookups, most RPA replacements | Claude Agent SDK, OpenAI Agents SDK | | Supervisor / worker | Research synthesis, multi-step reports | LangGraph, CrewAI | | Peer-to-peer swarm | Simulations, adversarial testing | AutoGen, OpenAI Swarm | | Deterministic pipeline with LLM steps | Regulated workflows, invoicing, KYC | Temporal + LLM activities |

A hard-earned lesson: Temporal (or a similar durable execution engine) plus LLM calls as activities beats a pure agent framework for anything touching money, contracts, or compliance. You get retries, idempotency, audit trails, and human-in-the-loop for free. The agent is just one node in a graph you can reason about.

Three enterprise use cases that actually ship

1. Tier-1 support deflection. A single agent with 8-12 tools (order lookup, refund initiation, KB search, ticket escalation). Measured impact on a recent retail client: 42% of tickets resolved without human touch, average handle time down from 6 min to 90 seconds on deflected cases. Key: aggressive scoping — the agent refuses anything outside its tool surface.

1. Internal data analyst. Agent with read-only SQL, semantic layer access (Cube, dbt Semantic Layer), and chart rendering. Replaces the "can someone pull numbers for me?" Slack channel. Watch out: without a semantic layer, hallucinated joins and wrong metric definitions will burn credibility fast.

1. Engineering workflow automation. Coding agents (Claude Code, Cursor Agent, Devin) doing scoped tasks: dependency upgrades, test generation, migration scripts. The unit of work must be small and verifiable — measurable by CI, not by human review.

A production readiness checklist

Before promoting an agent from POC to production:

• [ ] Every tool call is logged with inputs, outputs, latency, and cost
• [ ] Prompts and tool schemas are versioned in git, not in a Notion page
• [ ] There's a regression eval suite (at least 50 curated cases) run in CI
• [ ] Rate limits and budget caps per user/tenant are enforced at the gateway
• [ ] Sensitive tools require explicit confirmation or human approval
• [ ] Fallback path exists when the model or a tool is unavailable
• [ ] PII flowing to the model provider has been mapped and legally cleared
• [ ] On-call runbook covers "agent is doing something weird" scenarios

Skip any of these and you'll rediscover them the hard way in incident review.

Cost reality check

Agent loops are expensive. A support agent averaging 4 tool calls per session at ~8k input tokens per call on Claude Sonnet 4 costs roughly $0.10-0.15 per resolved ticket. Cheap versus a human agent, expensive versus a rules engine. Do the math before pitching agents where a decision table would suffice.

Prompt caching (Anthropic, OpenAI, Gemini all support it now) typically cuts input token cost by 60-90% on repeated system prompts. Turn it on from day one.

Key takeaways

• MCP is the integration standard. Expose business capabilities as MCP servers; keep agent frameworks swappable.
• Tool design beats model choice. Small, well-described, single-purpose tools with structured errors.
• Prefer single agents. Multi-agent orchestration is a last resort, not a starting point.
• Wrap agents in durable execution (Temporal, Restate) for any workflow that touches money or compliance.
• Ship evals and observability before ambition. No eval suite, no production.