When a large language model inside an agent workflow names the top factors behind its recommendation, operators often treat that list as a map of what the model actually used. A new research-stage paper, submitted to arXiv on 4 September 2026, tests that assumption directly — and finds it holds only loosely.
The study, by Urja Pawar and six co-authors, evaluates two interpretations of the factors models cite: necessity, meaning that changing a factor would change the output, and sufficiency, meaning that retaining it while removing other changeable information would preserve the output. The work is a preprint, not a peer-reviewed finding, and the authors scope it to individual LLM decisions rather than whole agent systems.
How the test works
The team built two synthetic use cases: recommending advisors to clients, and judging prompts for harmfulness or risk. In each, models returned an output plus the top three factors that most influenced it.
The researchers then ran controlled black-box interventions. A necessity score for each factor measures how often changing it changes the output. A sufficiency score measures how often retaining it — while stripping out other changeable information — preserves the output. Because the method only manipulates inputs and observes outputs, it requires no access to model internals.
What the numbers show
Across eight models from the Claude, GPT, and Gemini families, the mean Spearman correlations between the cited ranking and measured influence were moderate at best. For advisor recommendation, correlations with necessity and sufficiency scores were 0.349 and 0.354. For prompt monitoring, they were 0.431 and 0.580.
The more striking result concerns what the citations leave out. An uncited factor scored above the lowest-scoring cited factor in 57.6% of advisor responses under necessity and 58.1% under sufficiency. The corresponding prompt-monitoring rates were 25.8% and 8.9% — suggesting explanations were considerably more reliable in the safety-judgement task than in the recommendation task.
The authors' conclusion is measured: the cited top three contain useful information, but they do not reliably identify the three factors with the strongest measured influence under either interpretation.
Why it matters
Operators use named factors to monitor systems, diagnose errors, and decide when to escalate an output. If the cited factors diverge from behavioural influence, escalation decisions can rest on the wrong evidence — a human reviewer may watch the variable the model mentioned while the variable that actually moved the output goes unexamined. The gap also varied sharply by task, which means a reliability check passed on one workflow does not transfer to another.
The work sits alongside a growing line of research questioning free-text LLM explanations. A separate recent paper from NYU researchers formalized "self-consistent sufficiency" and found LLM explanations generally insufficient across nine models and four datasets, with weak correlation to model size or accuracy. The convergence matters: different methods are arriving at the same caution from different directions.
Our take
The practical contribution here is not the warning that explanations are unreliable — that suspicion is now well established — but a usable instrument. Because the necessity and sufficiency probes are black-box, an oversight team can run them against any vendor model without weights access, turning "trust the rationale" into a measurable property per deployment. The caveat the authors state themselves deserves equal weight: the framework checks individual decisions, not the chained behaviour of a full agent, which is where most production risk now lives.
