A study published on arXiv in late July and covered by Nature on 13 August 2026 tested whether an AI system could carry out open-ended research by giving it the research questions from two NeurIPS papers, six days, and $3,000 in compute. The system, built around Claude Opus 4.8 inside a modified OpenClaw agent scaffold, succeeded at the engineering grind — running hundreds of experiments, doing literature reviews, and catching its own hallucinations — but failed at the research judgment that matters most. The original authors scored the outputs 2/6 and 1/6, noting the agent settled on hypotheses too early and its self-critique was too weak to backtrack. "I don't think full automation of open-ended research is on the horizon right now," said co-author Sayash Kapoor of Princeton.
Two weeks after the arXiv preprint, Anthropic's own Institute published "When AI builds itself," a first-party account of recursive self-improvement (RSI) progress inside the company. Anthropic states clearly: "We are not there yet, and recursive self-improvement is not inevitable." Yet the essay documents a striking acceleration in AI development labor. As of May 2026, more than 80% of code merged into Anthropic's codebase was authored by Claude, up from low single digits before Claude Code launched in February 2025. In Q2 2026, the typical engineer merged 8x as much code per day as in 2024. A March 2026 internal poll of 130 research staff found a median self-reported 4x output uplift with the Mythos Preview model. Task-length benchmarks show a doubling roughly every four months: Claude Opus 3 handled 4-minute tasks in March 2024; Sonnet 3.7 reached 90-minute tasks a year later; Opus 4.6 managed 12-hour tasks a year after that. METR reports Claude Mythos Preview working "at least" 16 hours, at the upper end of what they can measure.
Confirmed
- Nature / Kapoor et al. (arXiv:2607.27191, late July 2026): Shadow evaluation of two NeurIPS research directions using Claude Opus 4.8 in OpenClaw scaffold; 6 days, $3,000 compute per paper. Engineering/experiment loops succeeded; open-ended research judgment failed (scores 2/6 and 1/6). Failure mode: early hypothesis lock-in, insufficient backtracking, weak self-critique. Kapoor quote: full automation of open-ended research not on the horizon.
- Sakana AI Scientist background: 2024 unveiling; improved version published in Nature March 2026 (Lu et al., Nature 651, 914-919). Three papers submitted to workshop, one accepted. Kapoor argues peer review is unreliable for assessing quality in AI research.
- Anthropic Institute essay "When AI builds itself" (published August 2026): Explicit statement: RSI not achieved, not inevitable. Internal metrics: >80% of merged code authored by Claude (May 2026); 8x code/engineer/day in Q2 2026 vs 2024; median 4x self-reported output uplift in March 2026 poll (n=130). Task-length doubling ~every 4 months (METR). SWE-bench and CORE-Bench saturated in ~2 years and ~15 months respectively. Key distinction: strong at executing specified work (engineering and well-scoped experiments), weak at choosing goals (research judgment).
- Anthropic's own caveats: Lines of code is an imperfect measure (quantity over quality); 8x almost certainly overstates true productivity gain. Self-reported uplift likely somewhat lower than 4x. "Large performance gaps persist when it comes to Claude exercising judgement in choosing goals in both engineering and research."
Analysis
The two sources are not contradictory — they are measuring different layers of the stack. Kapoor's shadow evaluation tests the top layer: can an AI system, given a research direction, make the judgment calls that define original research? The answer is no. Anthropic's essay tests the layer below: can AI systems accelerate the engineering and execution that implements research decisions? The answer is yes, dramatically. The 8x code metric and the task-length doubling curve describe a real compression of implementation time. But Anthropic's own distinction — "humans supply the goal, but they no longer need to supply the method" — maps directly onto Kapoor's failure mode. The shadow agent had the method (it could run experiments endlessly) but lacked the judgment to choose which hypotheses to pursue and when to abandon them. Acceleration of AI-development labor does not equal automated scientific breakthroughs.
This tension also reframes the Sakana AI Scientist result. Peer-review acceptance of one workshop paper looked like progress in 2024-2025. Kapoor's shadow evaluation, using original authors as expert reviewers rather than harried conference reviewers, reveals a harsher standard: the engineering works, the research judgment does not. Anthropic's internal data suggests the engineering acceleration is real and compounding. The judgment gap remains.
Unknown
- How far Anthropic's internal productivity claims (8x code, 4x self-reported output) generalize beyond its specific codebase, tooling, and team composition.
- Whether shadow-evaluation failures on two NeurIPS directions transfer to other domains or to other labs' RSI timelines.
- Whether the task-length doubling trend (4-month doubling) holds once tasks exceed the current measurement ceiling (~16 hours per METR).
- What fraction of the 8x code metric reflects net new capability vs. refactoring, boilerplate, or test generation that inflates line counts.
Our take
The industry is conflating "speed of building" with "quality of open-ended research." Anthropic's own data shows the former is real and steep; Kapoor's shadow evaluation shows the latter is still missing. The policy and investment conversation should track them separately.
