Claude Opus 5 scored 100% on all 20 attempts at four month-end close tasks. Then researchers brought in licensed CPAs, and the results got worse.
That’s the most cringe-inducing finding in a new study from Mercor, a startup that uses professionals to train and evaluate AI models.
David Leary and I covered it on Episode 509 of The Accounting Podcast.
What Mercor tested
Mercor gave 12 licensed CPAs four realistic month-end close scenarios. They averaged more than five years of experience. Six worked in public practice, six in industry, and half spent time at Big Four firms.
Each task asked them to search company files, find the right numbers, run the calculations, and deliver a table. But the tasks came with traps:
A hotel miscoded $4,000 of parking revenue as room revenue
A block of December hotel stays didn’t get billed until January 3, so the revenue wasn’t booked in December
A monthly lease reclassification needed a correction, and nearly every CPA working without AI skipped it
The results
Claude working alone hit 100% accuracy in a few minutes per task, at about 21 cents per correct rubric criterion.
CPAs working alone took 30 to 180 minutes and averaged 37%. Their cost came to $10.35 per criterion, nearly 50 times higher.
The most surprising part to me is that CPAs working with Claude took about 15 times as long as Claude alone.
That’s roughly 150 minutes, almost as slow as working solo.
And in four or five of the 20-something tasks, the CPA overrode or changed a number and undid a perfect Claude result.
Mercor built the study to measure how much AI helps accountants. Claude scored perfectly on its own, so the only direction left was down.
Read the caveats
Mercor says these results don’t make accountants replaceable, and I agree.
Participants started cold, with no client history, no coworkers to ask, and no way to clarify an ambiguous instruction.
Some tasks tested niche knowledge, like how a nonprofit splits shared costs. An accountant working outside that niche wouldn’t necessarily know that.
As one task designer said, the traps are realistic, but the setting and the scoring aren’t.
But that doesn’t explain the review result.
The review step is the bottleneck
When you use Claude in manual mode, you read every output, approve it, then read the next one. You work at human speed.
On the show, my cohost, David Leary, and I agreed that today’s AI setups don’t save as much time as they should.
You sit there approving, waiting, and answering questions.
Now picture that as a firm-wide workflow. You give staff an AI tool, tell them to review everything it produces, and call that a process.
The CPA becomes the slowest part of the system.
Worse, a reviewer who changes correct numbers adds errors instead of catching them.
If you hand your staff AI and expect efficiency to follow, you probably won’t see a return.
Think bank reconciliation
Nobody re-adds every cleared check by hand. You set the matching rules, let the system match, and spend your time on the unreconciled items.
The rules handle the volume while the accountant handles exceptions.
We described the same model for AI. The accountant defines the rules, the AI executes, automated checks run, and only exceptions reach a person.
That’s not the same as reviewing everything, and most firms haven’t built it yet.
It’s harder than it sounds. We haven’t figured out when to let an agent run and when to watch it. Agents also don’t know when to stop and ask a question.
Whoever solves that judgment call builds a workflow that finally pays off.
Take people out of the middle
If your firm rolled out AI by handing staff a tool, redesign the workflow instead of adding more review.
Put your people at the front, where they define the task and gather context from clients and coworkers. Put them at the back, where they handle exceptions.
Mercor expects skills like client communication and asking the right questions to fill more of an accountant’s day as models absorb the detail work.
Take people out of the middle, where they slow everything down.
