FEATUREDLatent Space· rssEN06:12 · 06·09
→Cognition launches FrontierCode: a coding benchmark that asks 'would you actually merge this?'
Cognition built FrontierCode, a benchmark that scores code on mergeability and maintainability, not just passing unit tests. Tasks were designed with open-source maintainers, each taking 40+ hours, and evaluated on regression safety, cleanliness, scope, test correctness, and maintainability. The best model, Opus 4.8, hits only about 13% on the hardest tier—far below the 50%+ common on SWE-Bench-style evals. The post also notes METR found many SWE-bench-passing PRs wouldn't actually be merged, and FrontierCode directly measures that false-positive problem.
#Code#Benchmarking#Cognition#Opus 4.8
why featured
Featured · importance 78 · hook + knowledge + resonance
editor take
Cognition's FrontierCode asks 'would you merge this?' instead of 'does it pass tests?' — top model Opus 4.8 hits just 13% on the hardest tier.
sharp
This one's worth opening because it pokes a hole in how we've been measuring code ability. METR already found that many SWE-bench-passing PRs wouldn't actually get merged. Cognition took that insight and built a benchmark with open-source maintainers — each task took 40+ hours to design, and scoring covers regression safety, code cleanliness, scope, and test correctness. The result: Opus 4.8 scores about 13% on the hardest tier, way below the 50%+ you see on SWE-bench-style evals.
Don't read this as 'models got worse at code.' The cleaner take: old benchmarks treated 'it runs' as 'it ships,' and FrontierCode adds the maintainability half of the picture. We've only got the Latent Space summary so far — the full report and test set aren't public yet. I'd discount a bit until we see the actual tasks and rubrics.
HKR breakdown
hook ✓knowledge ✓resonance ✓