en
Back to the list

AI agents built their own scientific journal and found that reviewing research was harder than expected

source-logo  cryptoslate.com 24 m
image

AI reviewers at an experimental journal caught flaws in 11 of 16 research drafts. When they read corrected versions of the same papers, they raised the same objection in 10 of 16 cases. The authors of the Nov. 21 preprint A Society of AI Scientists say the reviewers kept flagging the issue because the revised papers openly described the limitation.

In a separate audit at the agent-run journal, a guest editor turned reviewer objections into scripts and ran them on the authors’ uploaded data. The checks overturned the main statistical conclusion in 77 of 156 objections classed as major, the preprint reports. Several objections could concern the same study.

The authors describe how the Journal of Agent Research grew out of a 12-week study of 936 AI agents across 35 human-operated labs. Agents in one lab founded the journal and launched a standing panel of automated referees on Nov. 4. Human operators set up the computing systems, capped spending on model access and chose broad research themes for the study.

Know when your
coins move

Alerts, real-time prices, and market news — all in one app
qr-code Point your phone camera to install
4.8 based on 40K reviews in the App Store and Google Play

How the journal tested AI reviewers’ objections

The panel read manuscripts and raised concerns. Authors could also formally submit a study for an editorial decision. The editor could then recompute the main numbers from the uploaded code and data. A referee could point to something in the text that looked like a flaw. An editor could test what happened to the result when the criticism was applied.

The objection audit put that approach to work on 276 criticisms across 43 groups of papers. Running the scripts overturned the main result for 24 of 120 minor objections, less frequently than for major ones. In a separate case, the editor rejected a paper after finding that its code replaced failed attempts with the exact answer. That substitution undermined the paper’s claim that its verification method had improved performance.

The researchers’ public archive includes papers, code and agent discussions. The dataset card says the download excludes raw experiment data, which the journal used to test objections. The authors describe an exploratory, nonrandomized study of computational research. Each lab ran the agents on its own infrastructure, so the authors relied on participants to disclose whether they followed restrictions on manual work.

With a formal submission’s code and data, an editor could check whether the objection held up and whether it changed the result.

cryptoslate.com