Why AI Agents Fail Even When They Understand the Problem Perfectly

There is a category of failure that is very easy to miss when you are watching an AI agent work. The agent reads the situation. It processes the relevant signals. It arrives, by most reasonable measures, at an accurate picture of what is happening. And then it does the wrong thing anyway. Or nothing at all. The outcome looks identical to the outcome you'd get from an agent that simply misread the situation from the start — but the underlying problem is completely different, and fixing one will do nothing about the other.

This distinction has a name in organizational psychology: the knowing-doing gap. It describes the persistent, maddening phenomenon of understanding what needs to happen and failing to make it happen regardless. Humans live inside this gap constantly. Now researchers are finding that LLM agents do too — and that the benchmarks currently used to evaluate them are not equipped to see it.

The Problem With Watching Only the Ending

Most AI agent evaluations work by pointing a system at a task, letting it run, and then measuring the final result. This is intuitive. It mimics how we judge human performance in most professional contexts. But it quietly assumes something that increasingly does not hold: that the final cost, score, or outcome can tell you what went wrong when things go wrong. According to new research introducing STOCKTAKE^[2], a 26-week supply-chain replenishment benchmark, this assumption breaks down completely in the class of tasks where AI agents are now being seriously deployed.

The core issue is partial observability. In supply-chain management — and in a widening range of real-world agentic tasks — the state of the world that actually drives cost is never directly visible to the agent. An inventory system might know how many units it has on hand, but not the demand signal that is quietly building in the background, not the supplier lead time that has just shifted, not the stockout that is three weeks away unless an order goes in today. The agent has to infer what is happening from incomplete, delayed, and often ambiguous information. It is making decisions under a kind of structured fog.

When an agent fails under these conditions, the final cost number tells you essentially nothing useful. The agent may have misread the fog — it perceived the state of the world incorrectly and acted on a mistaken model. Or it may have read the fog correctly and still failed to act appropriately, for reasons that have nothing to do with perception: poor planning, failure to translate understanding into the right sequence of decisions, an inability to execute intentions at the right moment. These are fundamentally different failure modes. One is a perception problem. The other is an action problem. They require different fixes. And yet, as the STOCKTAKE researchers note, existing evaluations cannot separate them.

“The final cost cannot say why an agent failed: it may have misread the world, or read it correctly and still failed to act.”

The Oracle Problem

Part of what makes this hard to fix is the reference policy problem. To evaluate whether an agent's actions were good, you need some benchmark to compare them against — a sense of what a well-functioning agent should have done in the same situation. Most evaluation frameworks either skip this step entirely, leaving the outcome score to do all the work, or they use a reference policy that has access to information the agent never had. The second approach is subtly corrosive: you are grading a student on an open-book test and then wondering why they performed worse than the answer key.

STOCKTAKE addresses this by building what the researchers call a "fair oracle" — a reference policy constructed to operate under the same informational constraints as the agent being evaluated. The benchmark is designed as a factored partially observable environment, which is a technical way of saying the world is structured so that hidden and visible states can be cleanly separated. This architecture lets the framework ask distinct questions: did the agent perceive the hidden state as well as it reasonably could have, given what was observable? And given its perception, did it act well? You can get credit for reading the situation accurately even if your action was wrong, and you can be penalized for acting badly even if your perception was sound.

The supply-chain setting is particularly well-suited to this kind of factoring. Inventory replenishment over 26 weeks involves a long horizon of sequential decisions where each week's order placement affects future stock levels, cost exposure, and service quality. Hidden demand patterns accumulate invisibly until they surface as either overstock or stockout. The delayed consequences mean an agent cannot simply react to what it sees; it has to model what it cannot see and act on that model weeks before the consequences materialize. This is not a toy problem. It is the kind of task that large logistics operations run at scale, and the kind of task that agentic AI systems are increasingly being pointed at.

A Measurement Gap That Is Getting More Expensive

The timing of this benchmark matters. The agentic AI space is moving quickly from short, contained, single-session tasks toward extended multi-week deployments where agents operate with meaningful autonomy over real systems. Supply chains are one of the earlier beachheads. Hotel pricing is another — separate recent research has examined agentic policy editors operating in hotel-pricing simulators where agents receive only region-level diagnostic feedback rather than direct outcome signals. Legal reasoning environments are beginning to use collaborative multi-agent systems. Code agents are being evaluated not just on whether code runs, but on the full trajectory of how they follow instructions, use tools, verify their own work, and recover from mistakes.

Across these settings, a common structural feature keeps appearing: the agent operates in an environment where what drives outcomes is not directly observable, and where the gap between understanding and action is a real, distinct source of failure. Yet the evaluation infrastructure has not caught up. If we are watching these systems at work without the right instruments, we are accumulating a kind of measurement debt — deploying systems we cannot properly diagnose, in settings where diagnosis increasingly matters.

Related work is circling the same problem from adjacent angles. PM-Bench, a benchmark inspired by cognitive science's Virtual Week paradigm^[1], evaluates something called prospective memory in LLM agents: the ability to execute an intention at a specific future cue while other activities are ongoing. The framework asks agents to maintain user intentions, execute delayed actions, and monitor latent environment changes across a simulated seven-day week. What it is measuring, in part, is the agent's capacity to hold a plan and carry it through — which is precisely the action side of the knowing-doing gap. Another line of work looks at tool-switching: what happens to an LLM agent's behavior when the reliable tool silently changes within an ongoing session. Agents, that research finds, tend to settle on a small recurring repertoire of tools and fail to adapt when hidden reliability shifts — a kind of rigidity that sits again in the action-not-perception quadrant.

“We are deploying systems we cannot properly diagnose, in settings where diagnosis increasingly matters.”

Why This Is Not Just a Technical Problem

The knowing-doing gap, when it shows up in humans, is rarely a knowledge deficiency. It is usually something more structural: an environment that does not translate understanding into timely action, a feedback loop that is too slow or too noisy to guide behavior, a planning failure that lets a correct diagnosis sit idle while the window for intervention closes. These are not random failures. They have patterns, and those patterns point toward specific remedies. The same logic should apply to AI agents — but only if you have measurement tools fine-grained enough to see the pattern.

What STOCKTAKE is offering, in essence, is a way to stop treating agent failure as a single undifferentiated thing. An agent that fails because it cannot read partial observations correctly needs different architectural attention than an agent that reads those observations well and then fails to translate them into a coherent sequence of actions. Collapsing both into a final cost score and optimizing against it is a little like treating a broken compass and a broken engine as the same malfunction because both will leave you in the wrong place. The score tells you that you ended up somewhere bad. It does not tell you which part of the system to fix. As AI agents take on longer, more autonomous, higher-stakes roles — in logistics, in pricing, in legal reasoning, in code — that distinction is going to cost someone real money before it becomes obvious enough to demand attention.

References

1. How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks (arxiv.org)
2. STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle (arxiv.org)
   Introduces STOCKTAKE benchmark that separates perception failures from action failures in LLM agents using a fair oracle reference policy.