🔍 Read the full analysis: FrontierCode: Making Sense Of AI’s Role In Open-Source Pull Requests on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis’s October 12 evaluation of 23 models on FrontierCode found that Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts. The benchmark grades pull requests for factors including correctness, scope and review quality, and its results show that model performance and cost vary widely.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark for judging whether code changes would be acceptable in real open-source repositories, and found that even the top-ranked model cleared every blocking check in only 57.7% of attempts. Claude Opus 5.5 led the evaluation with a 52.2% overall score, underscoring that passing tests alone does not establish that an AI-generated pull request is ready to merge.
The results, added to Artificial Analysis’s evaluations on October 12, cover the benchmark’s 100-task Main set. Models were run through the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and up to three attempts per task. The review stage used GPT-6 Luna (medium) as the judging model.
GPT-6.1 Sol ranked second, scoring 49.1% at a reported cost of $0.81 per task. Claude Fable 5.1 scored 49.0%, while Claude Sonnet 5.5 scored 45.8%. The top nine models, from six labs, scored between 43.2% and 52.2%. Artificial Analysis reported that nine of the benchmark’s 100 tasks were not solved by any model on any attempt.
The reported overall score is not the same as the pass rate. FrontierCode gives an attempt zero if it fails any blocking check; attempts that pass all blocking checks receive a score between 70% and 100%, depending on quality checks. Opus’s 52.2% score therefore sits alongside a 57.7% rate of clearing all blocking checks, while GPT-6.1 Sol’s figures are 49.1% and 54.7%, respectively.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Merge Readiness Remains Limited
The results distinguish working code from a patch that maintainers can reasonably accept. A change may pass tests yet still introduce regressions, touch too many files, include weak tests or fall short of review standards. For project teams considering coding agents, the benchmark suggests that human review remains necessary: even the leading model failed at least one blocking check in a substantial share of attempts.
Cost adds another trade-off. Artificial Analysis reported task costs ranging from $0.09 for GPT-6 Luna to $10.15 for Claude Opus 5.5, with different performance levels. Moving from GPT-6.1 Sol to Opus raised the score by 3.1 percentage points while increasing the stated cost from $0.81 to $10.15 per task. Those figures may help teams compare options, but they do not establish what any model will cost or achieve on a particular project.
The ranking also differs from general intelligence evaluations in the source report. Claude Sonnet 5.5, described there as second among the compared models on the Intelligence Index, ranked fifth on FrontierCode; Grok 4.7 ranked 18th on the coding benchmark. That contrast cautions against treating broad model rankings as a substitute for task-specific testing.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Pull Requests
Cognition developed FrontierCode with open-source maintainers; the benchmark was announced in June 2026, followed by version 1.1 in July. According to the source report, more than 20 maintainers created tasks drawn from 36 established repositories, including Celery, Budibase, uppy and Mattermost. The full benchmark contains 150 tasks, while Artificial Analysis evaluated the 100-task Main set. Tasks are private, a design intended to reduce the chance that models have encountered their solutions during training.
FrontierCode examines patches on six axes: hidden tests for correctness; command checks for regressions and cleanliness; reverse tests to check that an agent’s own tests fail on the original code; adaptive tests for complex behavior; limits on touched files and diff size; and an LLM code review against maintainers’ criteria. The benchmark also penalizes oversized diffs, edits outside relevant files and unhelpful tests. Cognition’s trajectory scanner can assign a zero to attempts it flags for looking up a solution online.
In Cognition’s announcement, Celery maintainer Tomer Nosrati described the approach this way: “Where others grade like a CI, FrontierCode grades like a tech lead.” That is a characterization of the benchmark’s aim, not evidence that its automated assessment fully reproduces every maintainer’s judgment.
““Where others grade like a CI, FrontierCode grades like a tech lead.””
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
Limits of the October Results
The results do not show how each model would perform across the full 150-task benchmark, since Artificial Analysis ran the 100-task Main set. The source material does not provide uncertainty intervals or enough detail to determine how much scores might shift with different tasks, agent configurations or review judges. The reported rankings should be read as results under this evaluation setup, not as a guarantee of performance in every repository.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the results cannot establish how those systems would compare. The source report says nine tasks were not solved by any model on any attempt, but does not identify those tasks or explain whether later benchmark updates will change that count.
More Models and Benchmark Runs
The next useful comparison will come when missing models receive results or Artificial Analysis publishes further runs. Additional information on task-level performance, score variation and the benchmark’s full task set would help readers judge how broadly the headline rankings apply. Until then, teams can use these results as one comparison point, while testing candidate models against their own repositories and keeping human review in the pull-request process.
Key Questions
What does FrontierCode measure?
FrontierCode evaluates whether AI-generated repository changes meet maintainers’ acceptance criteria. It checks correctness, regressions, test quality, scope and code-review standards, rather than relying only on whether tests pass.
Which model ranked first in the reported evaluation?
Claude Opus 5.5 (max with fallback) ranked first with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does a 52.2% score mean the model passed 52.2% of checks?
No. The overall score and blocking-check pass rate are distinct measures. Artificial Analysis reported a 52.2% score and a 57.7% blocking-check pass rate for Opus.
Does the benchmark show that AI pull requests can be merged without review?
No. The reported results do not establish that autonomous merging is reliable. Even the top model failed at least one blocking check in many attempts, and FrontierCode’s review component is only one evaluation method.
Were all major models included?
No. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview did not have FrontierCode results as of October 13, 2026.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
