Anthropic’s new AI leads freelance benchmarks, yet fails 58% of the tasks

Anthropic’s Fable 5.2 ranks first on the Remote Labor Index, a test of complete freelance projects, with 41.3% of its deliverables judged to meet a professional reference standard.

Scale’s Nov. 12 leaderboard lists Fable 5.2 at 41.3%, ahead of the next-highest model, Opus 5.5 at 21.25%. Anthropic reported the result in its Nov. 9 release announcement, labeling the RLI score “High effort” and indicating that safeguards were enabled.

The original RLI paper, submitted in October 2025 by researchers from the Center for AI Safety and Scale AI, describes a dataset of 240 projects across 23 freelance work categories. Each includes a brief, the files needed to complete the task, and a professional human-reference deliverable.

The selected projects include documents, audio, video, 3D models, and computer-aided design files. That range makes file production and finished-product quality part of the challenge, alongside understanding what the client requested.

The researchers’ original evaluation method uses trained human reviewers to inspect the AI’s work against the brief and reference deliverable.

Independent evaluations and majority voting determine outcomes, and a project passes when the AI’s deliverable satisfies the brief at least as well as the reference, at a quality a reasonable client would accept.

A useful prototype or a partial result can therefore fall below the pass threshold, even when it demonstrates relevant skills. Passing demands a deliverable that clears the professional reference standard.

What the Anthropic score leaves open

Fable 5.2’s score still leaves most evaluated projects below that acceptance threshold.

The original study excludes work that inherently requires client interaction, teamwork, or physical labor, as well as projects whose outcome must be awaited before it can be evaluated. Those exclusions limit how broadly the result can apply to the wider remote-work economy.

Anthropic’s release page specifies High effort, but does not establish the precise RLI execution setup, spending per project, or amount of human support.

RLI provides a demanding definition of success for a business assessing AI’s usefulness: carrying an assignment through to an inspectable product, with professional work setting the acceptance standard.