AI news · October 12, 2026
Sakana's MLR catches 73% of planted core-claim errors at $0.47 per review
Sakana's MLR caught 73.43% of planted core-claim errors at four reviews but only 16.11% of exact matches on real withdrawn papers; the benchmark gap is the story.
- planted core-claim errors caught with four reviews
- 73.43%
- exact matches on real withdrawn-paper errors
- 16.11%
- reported cost per review
- $0.47
Sakana AI's MLR paper changes the scorecard for AI-assisted peer review. Instead of asking whether a model writes a review that sounds like a human's, it asks whether the system catches contradictions inserted into real papers. The benchmark contains 1,164 planted contradictions across 257 papers. With four reviews, MLR caught 73.43% of core-claim errors versus 14.81% for the strongest baseline reported by AI Tool Herald; it uses off-the-shelf Claude models and costs about $0.47 per review.
The second number is more important for deployment: on real withdrawn-paper cases, the paper reports only 16.11% exact matches. MLR also remains vulnerable to hidden prompt injection in submitted PDFs. That makes it a pre-submission or triage assistant, not a publication gate. Builders can borrow the three-pass reading pattern and the error-focused benchmark, but should test on their own documents and keep a human decision-maker.
What you can do with it
There is no one-click hosted product in the paper. Read the arXiv method, recreate the outline, deep-read and synthesis passes with your Claude API account, and compare the output with known errors in your own papers. Track planted-error catches, real-error misses, cost and prompt-injection behaviour.
Our take
MLR is a useful evaluation story because it reports both a strong constructed-test result and a weak real-world result. The gap prevents a lazy automation pitch while still showing that a staged reading workflow can help. Use it to find questions before a human review, never to replace the reviewer.
Links PaperDetailed report
Source: Sakana AI / arXiv — Made With Models writes the brief; the reporting is theirs.