The Most Important Capabilities in an AI Agent Platform

Platform feature lists are organised the way vendors think about their products, which is not the way buyers experience them. A more useful organisation is by failure mode: for each capability, what specifically goes wrong in production without it.

Grounded retrieval

Prevents: confident, fluent, wrong answers.

An agent answering from the model's parameters alone will occasionally produce something plausible and false, and it will do so in the same tone it uses when correct. Grounding answers in retrieved source material does not eliminate this, but it changes the failure from invention to retrieval error — which is detectable, attributable, and fixable.

What to look for beyond "we support RAG":

• Whether retrieved sources are visible in the interaction record, so an answer can be traced to what produced it.
• How the system behaves when retrieval returns nothing relevant. Declining to answer is a feature.
• How content updates propagate, and how long stale material can persist.
• Whether retrieval respects the permissions of the person being served, rather than running with a single privileged identity.

That last point is where deployments most often get into difficulty, and it is almost never visible in a demo.

A governed tool layer

Prevents: the agent taking an action nobody authorised.

Once an agent can write to systems rather than only read from them, the tool layer becomes the most safety-relevant part of the architecture. The capability that matters is not the number of available integrations but how tightly each one can be constrained.

Useful properties:

• Tools declared with explicit schemas and explicit side-effect classifications, so read-only and state-changing operations are distinguishable by the system rather than by convention.
• Per-tool permission scoping, ideally inheriting the end user's authorisation.
• Rate and value limits that the platform enforces, rather than instructions in a prompt asking the model to be careful.
• A complete record of which tool was called with which arguments.

Evaluation infrastructure

Prevents: shipping a regression you do not notice for weeks.

Agents fail probabilistically. Without a systematic way to measure behaviour across a set of representative cases, you learn about regressions from complaints, which is both slow and expensive.

Minimum viable evaluation capability:

• A case set you control, containing your own inputs and expected outcomes.
• Automated scoring for what can be scored automatically, and a workflow for human review of what cannot.
• The ability to compare two versions on the same case set and see what moved.
• Sampling of live traffic, so the case set can grow from real failures.

The exportability of all of the above matters more than it looks. Evaluation sets are among the most valuable artefacts a team builds, and a set that cannot leave the platform is a substantial part of the switching cost.

Interaction-level observability

Prevents: being unable to explain what happened.

When something goes wrong — and in a system with this failure profile something eventually will — the question is how quickly you can reconstruct the sequence of events. This is an operational requirement, and in regulated contexts frequently a compliance one.

A complete record includes what the agent was asked, what it retrieved, what it decided, which tools it called with which arguments, what came back, what it said, and which configuration version was live at the time. Retention needs to outlast your dispute window, not your debugging window.

The capabilities that strain in production

Four more that consistently look adequate in evaluation and prove awkward later:

Memory. Persistence across interactions is straightforward to demonstrate and difficult to govern. Ask what is retained, for how long, who can see it, how a user has it deleted, and what happens to agent behaviour when it is.

Escalation. Handing off to a human is easy; handing off well is not. The receiving person needs context, and the transition needs to work when the agent is uncertain rather than only when it detects an explicit request.

Versioning and rollback. Configuration is code. If you cannot see what changed, when, and by whom — and revert it — incident response becomes guesswork.

Multi-agent coordination. As deployments grow into several specialised agents, how work is routed between them becomes an architectural decision. Systems that treat this as a first-class concern are easier to reason about than ones where it emerges from prompts.

A test for any capability claim

For each capability on a vendor's list, ask two questions: can my team operate this without you, and can I take the output of it with me. A capability that requires a support ticket to exercise is a service, not a platform feature, and one whose artefacts cannot be exported is part of the lock-in rather than part of the value.