Agents IA & automatisationNovember 6, 2026

Agentic RAG: when to turn retrieval into a tool an agent can call

A standard RAG follows a fixed sequence. As soon as multi-step reasoning or source selection is needed, an agent that invokes search as a tool becomes relevant.

Most teams putting an assistant into production start from a standard RAG pipeline: a user query, a vector search, an assembled prompt, a response. It is simple, predictable, and enough for many use cases. But as soon as you add multi-step reasoning, several heterogeneous sources or the need to trigger actions, this architecture hits its limits. That is precisely where the notion of an agent, and of agentic RAG, takes on an operational meaning for a CTO.

What a standard RAG actually does

In a classic RAG as described in the Azure architecture documentation, the orchestrator follows a fixed sequence: it receives the query, runs a search against an index, assembles the context and calls the language model. It does not decide whether to search, which index to query, or how many retrieval steps to take. Those choices are made at design time (Microsoft Learn).

This model works well when a query maps to a single search against a single index. Product FAQ, internal support over a homogeneous documentation base, contextualised summary: the fixed sequence is an asset, not a limitation. It makes the system testable and its cost predictable.

When the fixed sequence is no longer enough

The same documentation identifies several scenarios that exceed what a fixed pipeline can handle: multi-step reasoning, dynamic source selection, query decomposition at runtime, and combining retrieval with actions. In these cases, Microsoft recommends considering agentic RAG, in which an AI agent treats retrieval as a tool it can invoke on demand (Microsoft Learn).

The distinction matters. In a standard RAG, search is a mandatory step. In an agentic RAG, it becomes one tool among others, which the agent chooses to call — or not — depending on the query. That opens the door to decisions such as: querying a legal index rather than a technical one, chaining two searches where the second depends on the result of the first, or skipping search altogether because the question does not require external information.

The architectural consequences

Moving from a fixed pipeline to an agent changes several things for the technical team:

  • Latency becomes variable. An agent may decide to run one, three or five searches. The response-time distribution widens and latency budgets have to be thought in percentiles, not in averages.
  • Inference cost becomes less predictable. Every reasoning step consumes tokens. An explicit cap (maximum number of tool calls, global timeout) becomes an architectural constraint, not an option.
  • Evaluation gets harder. The metrics of a standard RAG — groundedness, completeness, utilization, relevancy recalled in the Azure guide — remain useful, but you need to add evaluations on the agent's trajectory: did it pick the right tool? Did it stop at the right moment?
  • Observability must trace decisions. Logging the query and the response is no longer enough. You need to capture the chain of tool calls, their arguments and their intermediate results.

A simple decision criterion

Before adopting an agentic architecture, ask the question the other way round: do your user queries mostly fit into a single search over a single index? If so, a well-instrumented standard RAG will be more robust, cheaper to operate, and easier to evolve. The agent's autonomy is only worthwhile if it solves a problem the fixed sequence cannot.

Conversely, if your use cases regularly involve several sources to arbitrate between, questions that decompose, or the need to chain retrieval and action in a third-party system, the rigidity of the standard pipeline becomes a ceiling. The agentic approach is then not a stylistic choice, but a response to a functional constraint.

Preparing the shift without rebuilding everything

An often underestimated point: the quality of an agentic RAG depends first on the quality of its retrieval tools. If the underlying index is poorly chunked, poorly enriched or poorly embedded, giving the agent the freedom to query it several times will not fix anything. The Azure guide devotes several phases to these choices — chunking strategy, chunk enrichment, embedding model selection, index configuration — which remain valid, and arguably more critical, in an agentic architecture (Microsoft Learn).

A reasonable trajectory therefore consists in stabilising a standard RAG first, measuring on which types of queries it fails, and introducing the agent only where those failures actually match the scenarios the agentic approach is designed to handle: multi-step, dynamic source selection, decomposition, actions. That avoids paying the complexity cost of an agent for marginal gains.

If you want to discuss the architecture suited to your use cases, our services cover the design and evaluation of RAG and agentic solutions; feel free to contact us to frame an initial analysis.

Sources

Share this article

Read also