FEATUREDHacker News Frontpage· rssEN22:10 · 09·16
→Coding agent harnesses can 2× your cost with no real accuracy gain
UC Berkeley and Arena researchers tested 7 models across 3 harnesses—Claude Code, Codex CLI, and Pi—on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with 3 repetitions per pair. Harness choice barely moves success rates (±2–5%), but cost can vary up to 5×. Claude Fable 5 hits 97.8% in Claude Code at $1.33, and 96.7% in Pi at $0.67. Pi, a minimal open-source harness with just read, write, edit, and bash, reaches the Pareto frontier on both benchmarks. The post doesn't spell out the exact open-source licenses for Pi and Codex CLI, and doesn't link the full pricing sheet.
#Code#Agent#Benchmarking#UC Berkeley
why featured
Featured · importance 78 · hook + knowledge + resonance
editor take
Swapping coding-agent harnesses barely moves success rates (±2–5%), but cost can vary 5×; the minimal open-source Pi harness hits the Pareto frontier.
sharp
This one's worth opening because it tests something everyone assumes: that Claude models belong in Claude Code. The UC Berkeley and Arena team ran 7 models across 3 harnesses—Claude Code, Codex CLI, and Pi—on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with 3 repetitions per pair. Harness choice barely budged success rates, but cost swung up to 5×.
The clearest number: Claude Fable 5 hits 97.8% in Claude Code at $1.33, and 96.7% in Pi at $0.67. Pi is a minimal open-source harness with just read, write, edit, and bash—and it lands on the Pareto frontier on both benchmarks, meaning it's the cheapest way to hit a given success rate.
I'd discount this a bit on two fronts. The post doesn't link the full pricing sheet or spell out the exact open-source licenses for Pi and Codex CLI, so you'll need to dig into the repos yourself. And 30 tasks with 3 reps per pair isn't a huge sample—the ±5% confidence intervals on Terminal-Bench aren't that tight.
But the core takeaway holds: if you're running batch tasks through Claude Code, swapping to a lighter harness could halve your token costs with almost no success-rate hit. For solo devs and small teams, that's faster money saved than waiting for the next model price cut.
HKR breakdown
hook ✓knowledge ✓resonance ✓