Anthropic Randomized Trial Finds Data Analysts Follow AI Agent Judgment Calls 86 Percent of the Time While Rarely Crediting the Agent
New Anthropic research: How AI agents shape experts' judgment calls When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts.
Context
In a randomized controlled trial with 50 professional data analysts using Claude Code, experts adopted the agent's randomly assigned judgment lean 86 percent of the time and repeated its arguments word for word while only 15 percent credited the agent, challenging assumptions about human in the loop oversight.
As coding and research agents take over multi step analytical workflows, safety frameworks and enterprise governance rules frequently rely on human experts to make final normative and methodological judgment calls. Until now, few randomized experiments have measured whether domain professionals actually preserve independent judgment when collaborating with capable agents.
Deep dive
Anthropic researcher Alex Tamkin, alongside coauthors through the Anthropic Fellows Program and Hailey Joren, published a preregistered randomized trial examining whether domain experts maintain independent judgment when delegating analytical work to an AI agent. Fifty professional data analysts used Claude Code for an hourlong task advising a fictional farming cooperative on whether to expand a program. The analysis required four subjective methodological decisions, each with two defensible options, and the researchers randomly assigned which option the agent would honestly argue for while instructing it never to mislead the user and to yield if the evidence required it.
Whereas analysts would be expected to align with the randomly assigned direction 50 percent of the time if the agent had no causal influence, participants adopted the agent's lean 86 percent of the time. Even high effort experts deferred: one analyst spent more than forty minutes reading source materials and drafting an independent analysis plan before abandoning two decisions when the agent argued for a different approach. At the same time, analysts using the agent finished faster and made far fewer serious errors than eleven participants who worked without it.
When interviewed afterward about why they made each choice, 77 percent of analysts repeated the agent's rationale in at least one answer, sometimes verbatim, yet only 15 percent credited the agent and roughly half described the final recommendation as more their own work than the agent's. Economist Alex Imas compared the result to Nisbett and Wilson's classic psychology experiments on unacknowledged external influence.
What to expect next
Because the researchers intentionally strengthened the agent's directional lean to measure causal impact during a single hourlong session, real world workplace effects may be smaller, and Anthropic plans further studies across longer horizons and multiple models. Even so, the finding that experts absorb agent framing without realizing it suggests that human in the loop oversight requires deliberate interface designs that surface alternative options before an agent commits to a recommendation.