CheckNet.NETWORK DIAGNOSTICS / TOOLKIT
WorkspaceAI Model TrackerNETWORK TOOLKIT

AI Model Tracker

Recent model launches with their API prices and the benchmark results published by each lab, linked back to the official announcement.

Anthropic · launched 9 Nov 2026

Claude Fable 5.2

Released alongside the access-limited Claude Mythos 5.2. Tokens cost 20% less than Fable 5.1, with output generated up to 30% faster and an optional fast mode at $16 input and $80 output.

Input / 1M
$8.00
Output / 1M
$40
Context
1M

OpenAI · launched 26 Oct 2026

GPT-6.1 Terra

A fast, efficient model for computer use, visual reasoning and writing, priced 50% below GPT-6.1 Sol.

Input / 1M
$1.00
Output / 1M
$5.00
Context
1.05M

Listed rates apply up to 272K input tokens. Above 272K, input costs $2, cached input $0.10 and output $7.50 per million tokens for the full request.

Anthropic · launched 7 Oct 2026

Claude Haiku 5.5

Anthropic's fastest small model for high-volume tasks, live support and subagents, with adaptive thinking and adjustable effort.

Input / 1M
$0.10
Output / 1M
$0.50
Context
1M

Listed prices and comparisons apply to prompts up to 100K tokens. Above 100K, input costs $0.50, output $2.50 and cache reads $0.05 per million tokens.

Google DeepMind · launched 30 Sept 2026

Gemini 4 Argon

Google's newest frontier model for long-running coding, knowledge work and cybersecurity defense, generally available in the Gemini API and app since 13 October 2026.

Input / 1M
$2.00
Output / 1M
$10
Context
1.05M

Introductory rates through 30 November 2026. From 1 December 2026, input is $4, output $20 and cached input $0.20 per million tokens. Caching is implicit only. Long decode raises the output limit to 1,048,576 tokens.

OpenAI · launched 29 Sept 2026

GPT-6.1 Sol

An upgrade to GPT-6 Sol for coding, professional work and computer use, with cached input priced 50% lower.

Input / 1M
$2.00
Output / 1M
$10
Context
1.05M

Listed rates apply up to 272K input tokens. Above 272K, input costs $4, cached input $0.20 and output $15 per million tokens for the full request.

Anthropic · launched 28 Sept 2026

Claude Sonnet 5.5

The second model in the Claude 5.5 family. Input and output prices match Sonnet 5, it runs more than 30% faster and needs fewer tokens per task.

Input / 1M
$2.00
Output / 1M
$10
Context
1M

Cache reads were reduced from $0.20 to $0.10 per million tokens on 7 October 2026.

Anthropic · launched 22 Sept 2026

Claude Opus 5.5

Anthropic's newest Opus model. Tokens cost 20% less than Opus 5 and cache reads 60% less, with output generated more than 30% faster.

Input / 1M
$4.00
Output / 1M
$20
Context
1M

OpenAI · launched 22 Sept 2026

GPT-6 Luna

The lowest-cost model in the GPT-6 family, positioned below GPT-6 Sol and the flagship GPT-6 Astra.

Input / 1M
$0.10
Output / 1M
$0.50
Context
1.05M

SpaceXAI · launched 21 Sept 2026

Grok 4.7

Grok's latest main model for coding, agentic tasks and professional knowledge work, with adjustable reasoning effort.

Input / 1M
$2.00
Output / 1M
$6.00
Context
500K

Base rates apply up to 200K context tokens; higher-context requests use different rates. Batch API is not supported. The separate fast variant costs twice as much.

OpenAI · launched 4 Sept 2026

GPT-6 Astra

OpenAI's flagship GPT-6 model, positioned above GPT-6 Sol and Luna for the most demanding professional work, coding and computer use.

Input / 1M
$10
Output / 1M
$50
Context
1.05M

Google DeepMind · launched 2 Sept 2026

Gemini 3.8 Flash

Google's latest stable Flash model for long-horizon software engineering, autonomous agents and complex enterprise workflows.

Input / 1M
$0.75
Output / 1M
$3.75
Context
1.05M

Introductory rates through 31 December 2026. From 1 January 2027, input is $1.50, output $7.50 and cached input $0.15 per million tokens. Cache storage and grounding are billed separately.

Google DeepMind · launched 21 Jul 2026

Gemini 3.5 Flash-Lite

Google's fastest, most economical 3.5-class model for high-volume agentic tasks, translation and simple data processing.

Input / 1M
$0.30
Output / 1M
$2.50
Context
1.05M

Text, image, video and audio inputs share the listed rate. Cache storage and grounding are billed separately.

Google DeepMind · launched 19 Feb 2026

Gemini 3.1 Pro

Preview

Google's publicly available Pro model for complex reasoning, multimodal understanding and agentic coding, currently in preview.

Input / 1M
$2.00
Output / 1M
$12
Context
1.05M

Rates apply to prompts up to 200K tokens. Above 200K, input is $4, output $18 and cached input $0.40 per million tokens. Cache storage and grounding are billed separately.

API pricing

ModelLaunchedInput / 1MOutput / 1MCached / 1MBlended / 1M
Claude Fable 5.2Anthropic9 Nov 2026$8.00$40$0.20$16
GPT-6.1 TerraOpenAIListed rates apply up to 272K input tokens. Above 272K, input costs $2, cached input $0.10 and output $7.50 per million tokens for the full request.26 Oct 2026$1.00$5.00$0.05$2.00
Claude Haiku 5.5AnthropicListed prices and comparisons apply to prompts up to 100K tokens. Above 100K, input costs $0.50, output $2.50 and cache reads $0.05 per million tokens.7 Oct 2026$0.10$0.50$0.01$0.20
Gemini 4 ArgonGoogle DeepMindIntroductory rates through 30 November 2026. From 1 December 2026, input is $4, output $20 and cached input $0.20 per million tokens. Caching is implicit only. Long decode raises the output limit to 1,048,576 tokens.30 Sept 2026$2.00$10$0.10$4.00
GPT-6.1 SolOpenAIListed rates apply up to 272K input tokens. Above 272K, input costs $4, cached input $0.20 and output $15 per million tokens for the full request.29 Sept 2026$2.00$10$0.10$4.00
Claude Sonnet 5.5AnthropicCache reads were reduced from $0.20 to $0.10 per million tokens on 7 October 2026.28 Sept 2026$2.00$10$0.10$4.00
Claude Opus 5.5Anthropic22 Sept 2026$4.00$20$0.20$8.00
GPT-6 LunaOpenAI22 Sept 2026$0.10$0.50$0.01$0.20
GPT-6 SolOpenAIListed rates apply up to 272K input tokens. Above 272K, input costs $4, cached input $0.40 and output $15 per million tokens for the full request.22 Sept 2026$2.00$10$0.20$4.00
Grok 4.7SpaceXAIBase rates apply up to 200K context tokens; higher-context requests use different rates. Batch API is not supported. The separate fast variant costs twice as much.21 Sept 2026$2.00$6.00$0.50$3.00
GPT-6 AstraOpenAI4 Sept 2026$10$50$1.00$20
Gemini 3.8 FlashGoogle DeepMindIntroductory rates through 31 December 2026. From 1 January 2027, input is $1.50, output $7.50 and cached input $0.15 per million tokens. Cache storage and grounding are billed separately.2 Sept 2026$0.75$3.75$0.075$1.50
Claude Fable 5.1Anthropic1 Sept 2026$10$50$0.25$20
Claude Opus 5Anthropic24 Jul 2026$5.00$25$0.50$10
Gemini 3.5 Flash-LiteGoogle DeepMindText, image, video and audio inputs share the listed rate. Cache storage and grounding are billed separately.21 Jul 2026$0.30$2.50$0.03$0.85
GPT-5.6 LunaOpenAI9 Jul 2026$0.20$1.20$0.02$0.45
GPT-5.6 SolOpenAIPrice before the GPT-6 launch, from OpenAI's announcement. OpenRouter now lists $2 / $10.9 Jul 2026$4.00$20–$8.00
GPT-5.6 TerraOpenAIOpenAI's model documentation supports the Batch API but does not list a batch price.9 Jul 2026$2.00$12$0.20$4.50
Claude Sonnet 5Anthropic30 Jun 2026$2.00$10$0.20$4.00
Claude Fable 5Anthropic9 Jun 2026$10$50$1.00$20
Gemini 3.1 ProGoogle DeepMind · PreviewRates apply to prompts up to 200K tokens. Above 200K, input is $4, output $18 and cached input $0.40 per million tokens. Cache storage and grounding are billed separately.19 Feb 2026$2.00$12$0.20$4.50
Claude Haiku 4.5Anthropic15 Oct 2025$1.00$5.00$0.10$2.00

Standard API prices in USD per million tokens, checked 2 Oct 2026 via OpenRouter. Blended assumes 3 input tokens for every output token. Batch discounts vary by model. Google and Grok prices were rechecked on 27 October 2026 against their official sources, linked in each model's details. Argon rates are introductory prices through 30 November 2026. Haiku 5.5 and Sonnet 5.5 prices were updated from Anthropic's 7 October 2026 announcement. GPT-6.1 Sol and GPT-6 Sol prices were checked on 8 October 2026 against OpenAI's announcement and model documentation. GPT-6.1 Terra prices were checked on 27 October 2026 against OpenAI's announcement and model documentation. Claude Fable 5.2 prices are from Anthropic's 9 November 2026 announcement.

Official benchmark results

Anthropic: Introducing Claude Fable 5.2 and Claude Mythos 5.2Read announcement

Claude Fable 5.2 launch results

Reported by Anthropic on 9 Nov 2026 · exact values

BenchmarkClaude Fable 5.2Claude Fable 5.1Claude Opus 5.5GPT-6 Astra
FrontierCode 2.0Agentic coding49.1%32.4%45.5%42.8%
FrontierSWE v2Agentic coding69.9%56.3%62.3%65.5%
AutomationBenchBusiness workflows51.4%31.4%42.5%41.4%
Terminal-Bench-Science 0.1Agentic scientific research74.6%52.6%58.7%68.1%
WANDRAgentic search · relaxed score82.1%68.7%72.3%–
OSWorld 2.1Computer use · offline set, partial89.3%83.5%87.2%79.5%
RoboDojo SimRobotics64.9%27.6%39.4%29.8%
AA-Briefcase v1.1Knowledge work · rubric66.1%61.5%61.2%52%
AA-Briefcase v1.1 (Elo)Knowledge work1785 Elo1669 Elo1800 Elo1569 Elo
Remote Labor IndexKnowledge work41.3%17.9%21.3%20.8%
Humanity's Last ExamMultidisciplinary reasoning · with tools74.2%65.6%67.7%57.2%
ChartographyVisual chart recognition · no tools73.4%44.8%64.4%71%
  • Bold marks the best reported score in each row.
  • Scores for the Claude models are with production safeguards on and at each model's highest effort, except FrontierCode 2.0 (Fable 5.2 and 5.1 at xhigh) and Remote Labor Index (Fable 5.2 at high).
  • Claude Mythos 5.2, the same underlying model with access limited to vetted organizations, scores 75.3% on Terminal-Bench-Science 0.1. The cybersecurity and protein design rows report Mythos scores, so they are not shown here.
  • Anthropic's table also includes Gemini 4 Carbon, which has no published API price, so it is not tracked here. RoboDojo Sim scores were run internally by Anthropic.
  • A dash means the lab did not report a result for that model.
Source: Introducing Claude Fable 5.2 and Claude Mythos 5.2 (Anthropic)
FrontierCode 2.0

Accuracy against cost · reported by Anthropic on 9 Nov 2026 · exact values

Whether a model's code changes would be merged by the maintainers of open-source repositories.

  • Claude Fable 5.2
  • Claude Opus 5.5
  • Claude Fable 5.1
  • GPT-6 Astra
Show data table
ModelEffortScoreCost per task
Claude Fable 5.2low34.2%$2.10
Claude Fable 5.2medium37.6%$2.48
Claude Fable 5.2high45.3%$6.72
Claude Fable 5.2xhigh49.1%$14.2
Claude Fable 5.2max45.6%$44.7
Claude Opus 5.5low14.7%$0.75
Claude Opus 5.5medium30.1%$1.70
Claude Opus 5.5high34.5%$2.42
Claude Opus 5.5xhigh43.4%$5.18
Claude Opus 5.5max45.5%$12.8
Claude Fable 5.1low20.6%$3.01
Claude Fable 5.1medium21.5%$3.89
Claude Fable 5.1high25.7%$5.39
Claude Fable 5.1xhigh32.4%$8.68
Claude Fable 5.1max31.7%$11.5
GPT-6 Astralow31.1%$2.44
GPT-6 Astramedium36.8%$3.49
GPT-6 Astrahigh38.3%$4.28
GPT-6 Astraxhigh41.4%$4.54
GPT-6 Astramax42.8%$6.68
  • Exact score and cost values from the chart data published with Anthropic's announcement.
  • Anthropic's chart also shows Gemini 4 Carbon, which has no published API price, so it is not tracked here.
Source: Introducing Claude Fable 5.2 and Claude Mythos 5.2 (Anthropic)
WANDR (relaxed score)

Accuracy against cost · reported by Anthropic on 9 Nov 2026 · exact values

Perplexity's benchmark of agents on large data-collection tasks.

  • Claude Fable 5.2
  • Claude Opus 5.5
  • Claude Fable 5.1
Show data table
ModelEffortScoreCost per attempt
Claude Fable 5.2low71.4%$12.7
Claude Fable 5.2medium73.6%$15.7
Claude Fable 5.2high79.3%$29.2
Claude Fable 5.2xhigh80.6%$39.8
Claude Fable 5.2max82.1%$53.1
Claude Opus 5.5low31.2%$1.25
Claude Opus 5.5medium62.8%$11.4
Claude Opus 5.5high67.3%$17.3
Claude Opus 5.5xhigh71.3%$29.3
Claude Opus 5.5max72.3%$38.7
Claude Fable 5.1low63.3%$24.4
Claude Fable 5.1medium64.5%$28.8
Claude Fable 5.1high66.7%$37.6
Claude Fable 5.1xhigh67.7%$44.5
Claude Fable 5.1max68.7%$51.1
  • Exact score and cost values from the chart data published with Anthropic's announcement.
Source: Introducing Claude Fable 5.2 and Claude Mythos 5.2 (Anthropic)
Humanity's Last Exam, with tools

Pass rate against cost · reported by Anthropic on 9 Nov 2026 · exact values

Expert-level questions across many disciplines, answered with access to tools.

  • Claude Fable 5.2
  • Claude Opus 5.5
  • Claude Fable 5.1
Show data table
ModelEffortPass rateCost per attempt
Claude Fable 5.2low68.4%$0.19
Claude Fable 5.2medium68.6%$0.21
Claude Fable 5.2high71.5%$0.63
Claude Fable 5.2xhigh73.4%$1.48
Claude Fable 5.2max74.2%$4.51
Claude Opus 5.5low57.4%$0.08
Claude Opus 5.5medium63%$0.17
Claude Opus 5.5high63.9%$0.29
Claude Opus 5.5xhigh66.4%$0.69
Claude Opus 5.5max67.7%$2.18
Claude Fable 5.1low60.2%$0.59
Claude Fable 5.1medium62.4%$0.85
Claude Fable 5.1high63.1%$1.37
Claude Fable 5.1xhigh65.2%$2.70
Claude Fable 5.1max65.6%$3.77
  • Exact score and cost values from the chart data published with Anthropic's announcement.
  • GPT-6 Astra scores 57.2% at max effort; Anthropic reports no cost for it.
Source: Introducing Claude Fable 5.2 and Claude Mythos 5.2 (Anthropic)
Chartography, no tools

Accuracy against cost · reported by Anthropic on 9 Nov 2026 · exact values

How accurately a model reads charts and figures.

  • Claude Fable 5.2
  • Claude Opus 5.5
  • Claude Fable 5.1
Show data table
ModelEffortScoreCost per attempt
Claude Fable 5.2low64.2%$0.03
Claude Fable 5.2medium66.4%$0.03
Claude Fable 5.2high69.8%$0.07
Claude Fable 5.2xhigh71.4%$0.15
Claude Fable 5.2max73.4%$0.75
Claude Opus 5.5low57%$0.02
Claude Opus 5.5medium59%$0.03
Claude Opus 5.5high59.2%$0.03
Claude Opus 5.5xhigh62.4%$0.08
Claude Opus 5.5max64.4%$0.49
Claude Fable 5.1low36.6%$0.03
Claude Fable 5.1medium42.2%$0.05
Claude Fable 5.1high46%$0.08
Claude Fable 5.1xhigh47%$0.38
Claude Fable 5.1max44.8%$0.87
  • Exact score and cost values from the chart data published with Anthropic's announcement.
  • GPT-6 Astra scores 71.0% at max effort and Gemini 4 Carbon 72.5%; Anthropic reports no cost for either.
Source: Introducing Claude Fable 5.2 and Claude Mythos 5.2 (Anthropic)

OpenAI: Introducing GPT-6.1 TerraRead announcement

Agents' Last Exam v1

Score against cost · reported by OpenAI on 26 Oct 2026 · exact values

Professional computer-use tasks, such as turning architectural drawings into a 3D model or logging an animated film shot by shot.

  • GPT-6.1 Terra
  • GPT-5.6 Terra
  • GPT-6.1 Sol
  • GPT-6 Astra
  • GPT-6 Luna
  • Claude Opus 5.5
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Terralow55.9%$0.16
GPT-6.1 Terramedium57.6%$0.29
GPT-6.1 Terrahigh57.2%$0.39
GPT-6.1 Terraxhigh58.5%$0.66
GPT-6.1 Terramax60%$1.01
GPT-5.6 Terralow40.2%$0.59
GPT-5.6 Terramedium42.6%$0.84
GPT-5.6 Terrahigh46.6%$1.72
GPT-5.6 Terraxhigh48.5%$2.50
GPT-5.6 Terramax50.7%$3.58
GPT-6.1 Sollow52.4%$0.70
GPT-6.1 Solmedium53.8%$1.31
GPT-6.1 Solhigh55.3%$1.45
GPT-6.1 Solxhigh56.5%$1.45
GPT-6.1 Solmax58.6%$2.00
GPT-6 Astralow53.4%$3.55
GPT-6 Astramedium57.6%$4.82
GPT-6 Astrahigh57.8%$5.15
GPT-6 Astraxhigh58.3%$5.91
GPT-6 Astramax59.3%$7.24
GPT-6 Lunalow36.3%$0.022
GPT-6 Lunamedium46.8%$0.093
GPT-6 Lunahigh43.6%$0.087
GPT-6 Lunaxhigh47.9%$0.093
GPT-6 Lunamax50.9%$0.14
Claude Opus 5.5max63.2%$8.82
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
Source: Introducing GPT-6.1 Terra (OpenAI)
MMMU-Pro (Vision)

Score against cost · reported by OpenAI on 26 Oct 2026 · exact values

Visual reasoning across expert-level, multi-discipline questions.

  • GPT-6.1 Terra
  • GPT-5.6 Terra
  • GPT-6.1 Sol
  • GPT-6 Astra
  • GPT-6 Luna
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Terralow83.7%$0.0038
GPT-6.1 Terramedium84.6%$0.0041
GPT-6.1 Terrahigh85%$0.0045
GPT-6.1 Terraxhigh85.4%$0.0055
GPT-6.1 Terramax85.5%$0.012
GPT-5.6 Terralow74.9%$0.011
GPT-5.6 Terramedium76.4%$0.012
GPT-5.6 Terrahigh77.5%$0.014
GPT-5.6 Terraxhigh78.7%$0.02
GPT-5.6 Terramax81.7%$0.079
GPT-6.1 Sollow84%$0.0076
GPT-6.1 Solmedium84.7%$0.0079
GPT-6.1 Solhigh85.8%$0.0088
GPT-6.1 Solxhigh85.8%$0.01
GPT-6.1 Solmax86.3%$0.013
GPT-6 Astralow85.3%$0.038
GPT-6 Astramedium85.3%$0.039
GPT-6 Astrahigh86.4%$0.044
GPT-6 Astraxhigh86.5%$0.051
GPT-6 Astramax86.7%$0.065
GPT-6 Lunalow73.5%$0.00049
GPT-6 Lunamedium75.1%$0.00061
GPT-6 Lunahigh76.5%$0.00074
GPT-6 Lunaxhigh77.9%$0.00095
GPT-6 Lunamax79.9%$0.0021
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
Source: Introducing GPT-6.1 Terra (OpenAI)
OSWorld 2.1 (offline set)

Score against cost · reported by OpenAI on 26 Oct 2026 · exact values

Multistep tasks completed by agents using desktop apps, such as building a 3D model from a technical drawing.

  • GPT-6.1 Terra
  • GPT-5.6 Terra
  • GPT-6.1 Sol
  • GPT-6 Astra
  • GPT-6 Luna
  • Claude Sonnet 5.5
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Terralow53.8%$0.36
GPT-6.1 Terramedium66.7%$0.50
GPT-6.1 Terrahigh68.7%$0.51
GPT-6.1 Terraxhigh71.3%$0.62
GPT-6.1 Terramax74.2%$1.19
GPT-5.6 Terralow24.1%$0.44
GPT-5.6 Terramedium27.4%$0.59
GPT-5.6 Terrahigh37.7%$1.19
GPT-5.6 Terraxhigh48.4%$2.25
GPT-5.6 Terramax57.1%$4.76
GPT-6.1 Sollow60.9%$0.38
GPT-6.1 Solmedium73.5%$0.65
GPT-6.1 Solhigh75.5%$0.90
GPT-6.1 Solxhigh75.9%$0.95
GPT-6.1 Solmax78%$1.15
GPT-6 Astralow63.3%$2.32
GPT-6 Astramedium75.4%$4.37
GPT-6 Astrahigh78.3%$5.76
GPT-6 Astraxhigh79%$6.60
GPT-6 Astramax80.9%$8.17
GPT-6 Lunalow7.5%$0.02
GPT-6 Lunamedium26.3%$0.079
GPT-6 Lunahigh39.7%$0.14
GPT-6 Lunaxhigh44.6%$0.19
GPT-6 Lunamax55.5%$0.30
Claude Sonnet 5.5low57.8%$0.68
Claude Sonnet 5.5medium65.9%$0.93
Claude Sonnet 5.5high73.3%$1.39
Claude Sonnet 5.5xhigh81%$2.22
Claude Sonnet 5.5max83.9%$5.74
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
Source: Introducing GPT-6.1 Terra (OpenAI)
Factual error rate on difficult prompts

Error rate against cost · reported by OpenAI on 26 Oct 2026 · exact values

Answers containing at least one factual error on deliberately difficult prompts. Lower is better.

  • GPT-6.1 Terra
  • GPT-5.6 Terra
  • GPT-6.1 Sol
  • GPT-6 Astra
  • GPT-6 Luna
Show data table
ModelEffortAnswers with any factual error (%)Cost per task
GPT-6.1 Terralow16.7%$0.024
GPT-6.1 Terramedium10.1%$0.029
GPT-6.1 Terrahigh7.7%$0.036
GPT-6.1 Terraxhigh5%$0.05
GPT-6.1 Terramax4.3%$0.12
GPT-5.6 Terralow53.6%$0.055
GPT-5.6 Terramedium51.5%$0.061
GPT-5.6 Terrahigh41.1%$0.078
GPT-5.6 Terraxhigh33.9%$0.11
GPT-5.6 Terramax22.3%$0.25
GPT-6.1 Sollow14.7%$0.048
GPT-6.1 Solmedium12%$0.056
GPT-6.1 Solhigh6.6%$0.079
GPT-6.1 Solxhigh5.5%$0.099
GPT-6.1 Solmax4.4%$0.12
GPT-6 Astralow13.3%$0.26
GPT-6 Astramedium9.7%$0.29
GPT-6 Astrahigh5.9%$0.43
GPT-6 Astraxhigh4.6%$0.55
GPT-6 Astramax4.2%$0.66
GPT-6 Lunalow44.3%$0.0025
GPT-6 Lunamedium31.2%$0.0035
GPT-6 Lunahigh27.6%$0.0047
GPT-6 Lunaxhigh24.5%$0.0061
GPT-6 Lunamax18.4%$0.013
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
  • Prompts are de-identified conversations where users flagged an earlier model error. They are not representative of typical usage.
Source: Introducing GPT-6.1 Terra (OpenAI)
Terminal-Bench 4.0

Score against cost · reported by OpenAI on 26 Oct 2026 · exact values

Coding agents solving problems from the command line, such as fixing a webpage whose layout shifts as it loads.

  • GPT-6.1 Terra
  • GPT-5.6 Terra
  • GPT-6.1 Sol
  • GPT-6 Astra
  • Claude Sonnet 5.5
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Terralow24.8%$0.24
GPT-6.1 Terramedium35.8%$0.46
GPT-6.1 Terrahigh41.8%$0.65
GPT-6.1 Terraxhigh46.4%$0.99
GPT-6.1 Terramax50.3%$1.84
GPT-5.6 Terramax21.5%$5.25
GPT-6.1 Sollow47.1%$0.76
GPT-6.1 Solmedium52.7%$1.02
GPT-6.1 Solhigh53%$1.28
GPT-6.1 Solxhigh53.4%$1.28
GPT-6.1 Solmax55.9%$1.98
GPT-6 Astralow50.6%$4.93
GPT-6 Astramedium54.4%$6.06
GPT-6 Astrahigh58.1%$7.18
GPT-6 Astraxhigh57.9%$7.45
GPT-6 Astramax58.2%$10.3
Claude Sonnet 5.5low20%$0.62
Claude Sonnet 5.5medium28.8%$0.68
Claude Sonnet 5.5high43%$1.46
Claude Sonnet 5.5xhigh61.5%$4.34
Claude Sonnet 5.5max70.6%$10.4
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
  • GPT-5.6 Terra is shown at its published max-effort score only.
Source: Introducing GPT-6.1 Terra (OpenAI)
Terminal-Bench Science 0.1

Score against cost · reported by OpenAI on 26 Oct 2026 · exact values

Scientific research tasks completed through a terminal, such as tracking cell divisions in a time-lapse video.

  • GPT-6.1 Terra
  • GPT-6.1 Sol
  • GPT-6 Astra
  • Claude Opus 5.5 with fallbacks
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Terralow24.3%$0.47
GPT-6.1 Terramedium34.3%$0.95
GPT-6.1 Terrahigh39.7%$1.24
GPT-6.1 Terraxhigh43.4%$1.72
GPT-6.1 Terramax58.3%$4.15
GPT-6.1 Sollow43.7%$1.79
GPT-6.1 Solmedium47.6%$2.34
GPT-6.1 Solhigh51.1%$2.76
GPT-6.1 Solxhigh53.7%$2.89
GPT-6.1 Solmax57%$5.47
GPT-6 Astralow55.4%$11.4
GPT-6 Astramedium57.4%$12.3
GPT-6 Astrahigh62%$14.9
GPT-6 Astraxhigh60.9%$15.8
GPT-6 Astramax68.1%$23.8
Claude Opus 5.5 with fallbacksmax63.3%$23.2
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
  • Opus 5.5 results use fallback models as reported by OpenAI; these are not standalone Opus 5.5 results.
Source: Introducing GPT-6.1 Terra (OpenAI)

SpaceXAI: Introducing Grok 4.7Read announcement

Grok 4.7 launch results

Reported by SpaceXAI on 21 Sept 2026 · exact values

Exact published scores · Higher is better. Unreported results are omitted.

BenchmarkGrok 4.7Grok 4.6GPT-5.6 SolClaude Fable 5.1GPT-6 Astra
CursorBench 4.0Software engineering46.3%40.4%41.7%51.8%–
DeepSWE v1.1Software engineering · Grok 4.7 uses high effort; other Grok 4.7 rows use xhigh71%65.2%72.7%70%–
EEBenchElectrical engineering64%53%39.4%56.4%–
AA-Briefcase v1.1Knowledge work1657 Elo1546 Elo1487 Elo1678 Elo–
Terminal-Bench 4.0Agentic terminal work37.6%20.3%37.3%57.9%–
Harvey's Legal Agent BenchmarkLegal work19.6%15.8%2.5%6.7%–
HealthBench ProfessionalClinical reasoning56.7%48.5%60.5%62.1%–
GDPvalProfessional knowledge work · Tabbed chart: Grok 4.7 xhigh, Grok 4.6 high, Fable 5.1 and GPT-6 Astra max1695 Elo1605 Elo–1735 Elo1542 Elo
AA BriefcaseMulti-hour office work · Tabbed chart: Grok 4.7 xhigh, Grok 4.6 high, Fable 5.1 and GPT-6 Astra max1657 Elo1546 Elo–1678 Elo1569 Elo
EEBenchElectrical engineering · Tabbed chart: Grok 4.7 xhigh, Grok 4.6 high, Fable 5.1 and GPT-6 Astra max64%53%–56.4%69.3%
  • Bold marks the best reported score in each row.
  • Exact values from the announcement comparison table and embedded tabbed chart data. Grok 4.7 uses xhigh effort except DeepSWE v1.1, which uses high.
  • The comparison table uses GPT-5.6 Sol and Fable 5.1 at max effort, and Grok 4.6 at high effort.
  • The table and tabbed charts use different competitor sets; each is preserved with its source qualifiers. Grok 4.6 is a benchmark comparison only.
Source: Introducing Grok 4.7 (SpaceXAI)
CursorBench 4.0: software engineering

Score against average cost per task · reported by SpaceXAI on 21 Sept 2026 · exact values

Long-running coding task scores and average costs, with exact values exposed in the official interactive chart.

  • Grok 4.7
  • Claude Fable 5.1
  • Claude Opus 5
  • GPT-5.6 Sol
  • Claude Sonnet 5
Show data table
ModelEffortScoreCost per task
Grok 4.7low33.1%$1.58
Grok 4.7medium41.6%$3.49
Grok 4.7high43.9%$4.69
Grok 4.7xhigh46.3%$6.01
Claude Fable 5.1low45.1%$5.44
Claude Fable 5.1medium46.8%$7.05
Claude Fable 5.1high49.2%$9.08
Claude Fable 5.1xhigh51.6%$13
Claude Fable 5.1max51.8%$17.3
Claude Opus 5low40.7%$4.87
Claude Opus 5medium43.3%$6.94
Claude Opus 5high44.7%$9.00
Claude Opus 5xhigh46.1%$11.4
Claude Opus 5max46.6%$11.9
GPT-5.6 Sollow24.6%$0.87
GPT-5.6 Solmedium31.1%$1.77
GPT-5.6 Solhigh35.7%$2.85
GPT-5.6 Solxhigh37.7%$4.40
GPT-5.6 Solmax41.7%$8.23
Claude Sonnet 5low24.1%$1.39
Claude Sonnet 5medium28%$2.31
Claude Sonnet 5high30.8%$3.48
Claude Sonnet 5xhigh32%$4.55
Claude Sonnet 5max34.1%$7.17
  • Exact scores and costs from the chart's accessible point labels. All five published model series and their effort levels are preserved.
  • Reported by SpaceXAI; the task costs reflect the evaluation harness and pricing used for this announcement.
Source: Introducing Grok 4.7 (SpaceXAI)

Google DeepMind: Introducing Gemini 3.8 Flash and 3.8 Flash CyberRead announcement

Gemini 3.8 Flash: DeepSWE v1.1

Score against average cost per task · reported by Google DeepMind on 2 Sept 2026

Google's original comparison of long-horizon software engineering scores and task costs.

Google DeepMind has not published the numbers behind this chart, so the original image is shown instead. Open the announcement for the full context.

Google DeepMind: DeepSWE v1.1 scores against average cost per task, with lower cost to the right.
Original chart published by Google DeepMind, shown for reference.
  • The image reverses the cost axis: lower cost is to the right. Exact numeric task costs and per-effort point values were not published with this chart, so no reconstructed points are shown.
  • The exact launch scores are available in the Gemini 3.8 Flash result table.
Source: Introducing Gemini 3.8 Flash and 3.8 Flash Cyber (Google DeepMind)

OpenAI: Introducing GPT-6.1 SolRead announcement

GPT-6.1 Sol safety evaluations

Reported by OpenAI on 29 Sept 2026 · exact values

BenchmarkGPT-6.1 SolGPT-6 SolGPT-6 AstraGPT-6 Luna
Failure to disclose a broken search toolSafety evaluation · Max effort2.1%4.92%1.5%28.67%
Reviewer bypass attemptsSafety evaluation · Max effort0%0%0%0.26%
Warning circumventionSafety evaluation · Max effort23.48%64.39%17.42%42.37%
Computer-use safety stress testSafety evaluation · Xhigh effort4.32%17.39%2.4%13.7%
  • Bold marks the best reported score in each row.
  • All rates are lower-is-better. These targeted evaluations stress challenging scenarios and do not represent typical usage.
  • Exact values from the embedded announcement chart data. No cost values were published for these evaluations.
Source: Introducing GPT-6.1 Sol (OpenAI)
DeepSWE v1.1

Score against cost · reported by OpenAI on 29 Sept 2026 · exact values

Agentic software engineering performance across reasoning effort settings.

  • GPT-6.1 Sol
  • GPT-6 Sol
  • GPT-6 Astra
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Sollow64.38%$0.17
GPT-6.1 Solmedium73.01%$0.42
GPT-6.1 Solhigh75.22%$0.65
GPT-6.1 Solxhigh71.9%$0.79
GPT-6.1 Solmax71.9%$1.57
GPT-6 Sollow37.17%$0.16
GPT-6 Solmedium56.64%$0.38
GPT-6 Solhigh65.27%$0.64
GPT-6 Solxhigh66.59%$1.00
GPT-6 Solmax68.81%$2.74
GPT-6 Astralow67.04%$1.60
GPT-6 Astramedium72.79%$3.08
GPT-6 Astrahigh73.23%$3.92
GPT-6 Astraxhigh74.12%$4.43
GPT-6 Astramax73.23%$7.50
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
Source: Introducing GPT-6.1 Sol (OpenAI)
GDP.pdf

Score against cost · reported by OpenAI on 29 Sept 2026 · exact values

Accuracy on professional questions using complex PDF documents.

  • GPT-6.1 Sol
  • GPT-6 Sol
  • GPT-6 Astra
  • Claude Opus 5.5 with fallbacks
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Sollow27%$0.33
GPT-6.1 Solmedium30%$0.34
GPT-6.1 Solhigh32%$0.35
GPT-6.1 Solxhigh31.8%$0.37
GPT-6.1 Solmax31%$0.42
GPT-6 Sollow21.8%$0.33
GPT-6 Solmedium25.4%$0.34
GPT-6 Solhigh28%$0.35
GPT-6 Solxhigh23.8%$0.37
GPT-6 Solmax24.8%$0.43
GPT-6 Astralow30.4%$1.70
GPT-6 Astramedium30.4%$1.72
GPT-6 Astrahigh31%$1.79
GPT-6 Astraxhigh32.2%$1.91
GPT-6 Astramax31%$2.08
Claude Opus 5.5 with fallbackslow25.6%$0.76
Claude Opus 5.5 with fallbacksmedium25.6%$0.80
Claude Opus 5.5 with fallbackshigh28.8%$0.83
Claude Opus 5.5 with fallbacksxhigh26.6%$0.96
Claude Opus 5.5 with fallbacksmax26.2%$1.55
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
  • Opus 5.5 results use fallback models as reported by OpenAI; these are not standalone Opus 5.5 results.
Source: Introducing GPT-6.1 Sol (OpenAI)
AutomationBench 1.0.6

Score against cost · reported by OpenAI on 29 Sept 2026 · exact values

End-to-end business workflows using 47 tools across sales, marketing, operations, support, finance and HR.

  • GPT-6.1 Sol
  • GPT-6 Sol
  • GPT-6 Astra
  • Claude Opus 5.5 with fallbacks
  • Claude Fable 5.1 with Opus 5 fallback
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Sollow24.7%$0.16
GPT-6.1 Solmedium31.7%$0.19
GPT-6.1 Solhigh33.2%$0.23
GPT-6.1 Solxhigh35.5%$0.25
GPT-6.1 Solmax36.1%$0.30
GPT-6 Sollow21.16%$0.19
GPT-6 Solmedium26.94%$0.21
GPT-6 Solhigh31.2%$0.24
GPT-6 Solxhigh33.18%$0.27
GPT-6 Solmax31.96%$0.34
GPT-6 Astralow30.3%$1.08
GPT-6 Astramedium34.1%$1.27
GPT-6 Astrahigh37.1%$1.44
GPT-6 Astraxhigh39%$1.50
GPT-6 Astramax41.4%$1.73
Claude Opus 5.5 with fallbackslow24.2%$0.51
Claude Opus 5.5 with fallbacksmedium29.53%$0.65
Claude Opus 5.5 with fallbackshigh33.03%$0.71
Claude Opus 5.5 with fallbacksxhigh35.77%$0.89
Claude Opus 5.5 with fallbacksmax42.47%$1.44
Claude Fable 5.1 with Opus 5 fallbackmax31.4%$2.45
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
  • Opus 5.5 results use fallback models as reported by OpenAI; these are not standalone Opus 5.5 results.
  • Fable 5.1 is reported at max effort only. Its cost omits fallbacks, which occurred on approximately 40% of tasks, so its actual cost is higher.
Source: Introducing GPT-6.1 Sol (OpenAI)
OSWorld 2.0 (offline set)

Score against cost · reported by OpenAI on 29 Sept 2026 · exact values

Partial reward on long-horizon computer-use workflows, using the offline set from release v2026.08.08.

  • GPT-6.1 Sol
  • GPT-6 Sol
  • GPT-6 Astra
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Sollow58.96%$0.42
GPT-6.1 Solmedium66.84%$0.77
GPT-6.1 Solhigh69.56%$0.96
GPT-6.1 Solxhigh69.38%$1.05
GPT-6.1 Solmax71.42%$1.27
GPT-6 Sollow43.9%$1.01
GPT-6 Solmedium54%$1.38
GPT-6 Solhigh58.29%$1.71
GPT-6 Solxhigh60.54%$2.30
GPT-6 Solmax64.43%$3.37
GPT-6 Astralow62.17%$2.72
GPT-6 Astramedium69.25%$5.36
GPT-6 Astrahigh70.02%$6.91
GPT-6 Astraxhigh71.27%$7.49
GPT-6 Astramax73.49%$9.44
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
Source: Introducing GPT-6.1 Sol (OpenAI)
Terminal-Bench Science 0.1

Score against cost · reported by OpenAI on 29 Sept 2026 · exact values

Scientific research tasks completed through a terminal.

  • GPT-6.1 Sol
  • GPT-6 Sol
  • GPT-6 Astra
  • Claude Opus 5.5 with fallbacks
Show data table
ModelEffortScore (%)Cost per task
GPT-6.1 Sollow43.71%$1.79
GPT-6.1 Solmedium47.56%$2.34
GPT-6.1 Solhigh51.14%$2.76
GPT-6.1 Solxhigh53.71%$2.89
GPT-6.1 Solmax57.02%$5.47
GPT-6 Sollow9.17%$3.00
GPT-6 Solmedium14.49%$4.41
GPT-6 Solhigh14.61%$4.63
GPT-6 Solxhigh25.29%$6.77
GPT-6 Solmax27.59%$12.2
GPT-6 Astralow55.43%$11.4
GPT-6 Astramedium57.43%$12.3
GPT-6 Astrahigh62%$15
GPT-6 Astraxhigh60.86%$15.8
GPT-6 Astramax68.1%$23.8
Claude Opus 5.5 with fallbacksmax63.33%$23.2
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
  • Opus 5.5 results use fallback models as reported by OpenAI; these are not standalone Opus 5.5 results.
Source: Introducing GPT-6.1 Sol (OpenAI)
Factual error rate on difficult prompts

Error rate against cost · reported by OpenAI on 29 Sept 2026 · exact values

Answers containing at least one factual error on deliberately difficult prompts. Lower is better.

  • GPT-6.1 Sol
  • GPT-6 Sol
  • GPT-6 Astra
Show data table
ModelEffortAnswers with any factual error (%)Cost per task
GPT-6.1 Sollow7.72%$0.045
GPT-6.1 Solmedium6.29%$0.056
GPT-6.1 Solhigh4.52%$0.082
GPT-6.1 Solxhigh4.12%$0.10
GPT-6.1 Solmax4.61%$0.13
GPT-6 Sollow11.42%$0.05
GPT-6 Solmedium6.86%$0.069
GPT-6 Solhigh5.14%$0.099
GPT-6 Solxhigh4.52%$0.13
GPT-6 Solmax4.57%$0.18
GPT-6 Astralow6.26%$0.24
GPT-6 Astramedium4.41%$0.31
GPT-6 Astrahigh3.9%$0.48
GPT-6 Astraxhigh3.99%$0.60
GPT-6 Astramax3.91%$0.79
  • Exact score and cost values from the chart data embedded in OpenAI's announcement.
  • Prompts are de-identified conversations where users flagged an earlier model error. They are not representative of typical usage.
Source: Introducing GPT-6.1 Sol (OpenAI)

Anthropic: Introducing Claude Haiku 5.5Read announcement

Claude Haiku 5.5 launch results

Reported by Anthropic on 7 Oct 2026 · exact values

BenchmarkClaude Haiku 5.5Claude Haiku 4.5GPT-6 LunaClaude Sonnet 5.5
GDPval-AA v2.1Knowledge work1620 Elo735 Elo1437 Elo1840 Elo
AA-Briefcase v1.1Knowledge work1578 Elo614 Elo1336 Elo1824 Elo
OSWorld 2.1Computer use · Offline subset72.4%15.7%48.9%83.9%
Humanity's Last ExamMultidisciplinary reasoning · No tools45.9%10.2%–56.9%
Humanity's Last ExamMultidisciplinary reasoning · With tools57.4%18.7%–64.5%
Terminal-Bench 4.0Agentic coding39.2%0%16.4%70.6%
FrontierCode 1.1 (Main)Agentic coding46.4%–42.4%52.1%
ChartographyVisual reasoning · No tools46.4%6.4%29.1%61.6%
  • Bold marks the best reported score in each row.
  • Scores are reported by Anthropic in the Haiku 5.5 announcement. Sonnet 5.5 is included for reference; its FrontierCode result uses xhigh effort.
  • A dash means the lab did not report a result for that model.
Source: Introducing Claude Haiku 5.5 (Anthropic)
OSWorld 2.1 (offline subset)

Accuracy against cost · reported by Anthropic on 7 Oct 2026 · exact values

OSWorld 2.1 measures how well agents can operate a real computer to finish long, multi-step tasks.

  • Claude Haiku 5.5
  • Claude Haiku 4.5
  • Claude Sonnet 5.5
  • GPT-6 Luna
Show data table
ModelEffortPartial-credit score (%)Cost per attempt
Claude Haiku 5.5low42%$0.07
Claude Haiku 5.5medium53.3%$0.13
Claude Haiku 5.5high61.3%$0.18
Claude Haiku 5.5xhigh67.6%$0.28
Claude Haiku 5.5max72.4%$0.61
Claude Haiku 4.5max15.7%$1.45
Claude Sonnet 5.5low57.9%$0.68
Claude Sonnet 5.5medium66%$0.93
Claude Sonnet 5.5high73.2%$1.38
Claude Sonnet 5.5xhigh81.1%$2.22
Claude Sonnet 5.5max83.9%$5.73
GPT-6 Lunalow19.2%$0.038
GPT-6 Lunamedium37.5%$0.13
GPT-6 Lunahigh42.3%$0.14
GPT-6 Lunaxhigh44.8%$0.17
GPT-6 Lunamax48.9%$0.20
  • Exact score and cost values from the chart data embedded in Anthropic's announcement.
  • Haiku 4.5 is reported at max effort only.
Source: Introducing Claude Haiku 5.5 (Anthropic)
GDPval-AA v2.1

Elo against cost · reported by Anthropic on 7 Oct 2026 · exact values

Artificial Analysis's GDPval-AA v2.1 evaluates agents on real-world professional work across 44 occupations.

  • Claude Haiku 5.5
  • Claude Haiku 4.5
  • Claude Sonnet 5.5
  • GPT-6 Luna
Show data table
ModelEffortElo, as reportedCost per task
Claude Haiku 5.5low1125 Elo$0.012
Claude Haiku 5.5medium1277 Elo$0.03
Claude Haiku 5.5high1420 Elo$0.089
Claude Haiku 5.5xhigh1513 Elo$0.27
Claude Haiku 5.5max1620 Elo$0.87
Claude Haiku 4.5max735 Elo$0.24
Claude Sonnet 5.5low1179 Elo$0.22
Claude Sonnet 5.5medium1324 Elo$0.27
Claude Sonnet 5.5high1551 Elo$0.62
Claude Sonnet 5.5xhigh1731 Elo$1.88
Claude Sonnet 5.5max1840 Elo$6.78
GPT-6 Lunalow1036 Elo$0.0038
GPT-6 Lunamedium1262 Elo$0.02
GPT-6 Lunahigh1344 Elo$0.03
GPT-6 Lunaxhigh1364 Elo$0.05
GPT-6 Lunamax1437 Elo$0.09
  • Exact score and cost values from the chart data embedded in Anthropic's announcement.
  • Haiku 4.5 is reported at max effort only.
Source: Introducing Claude Haiku 5.5 (Anthropic)
Humanity's Last Exam (no tools)

Accuracy against cost · reported by Anthropic on 7 Oct 2026 · exact values

Humanity's Last Exam (HLE) is a test of expert-level academic knowledge and reasoning.

  • Claude Haiku 5.5
  • Claude Haiku 4.5
  • Claude Sonnet 5.5
Show data table
ModelEffortScore (%)Cost per attempt
Claude Haiku 5.5low30.9%$0.0033
Claude Haiku 5.5medium35.5%$0.0069
Claude Haiku 5.5high39.7%$0.02
Claude Haiku 5.5xhigh44.1%$0.065
Claude Haiku 5.5max45.9%$0.21
Claude Haiku 4.5max10.2%$0.093
Claude Sonnet 5.5low41.4%$0.029
Claude Sonnet 5.5medium43.2%$0.033
Claude Sonnet 5.5high47.9%$0.077
Claude Sonnet 5.5xhigh53%$0.20
Claude Sonnet 5.5max56.9%$0.89
  • Exact score and cost values from the chart data embedded in Anthropic's announcement.
  • Haiku 4.5 is reported at max effort only.
Source: Introducing Claude Haiku 5.5 (Anthropic)
Terminal-Bench 4.0

Accuracy against cost · reported by Anthropic on 7 Oct 2026 · exact values

Terminal-Bench 4.0 measures how well a model can complete complex, multi-step professional tasks within a command-line interface.

  • Claude Haiku 5.5
  • Claude Haiku 4.5
  • Claude Sonnet 5.5 ($0.10 cache reads)
  • Claude Sonnet 5.5 ($0.20 cache reads)
Show data table
ModelEffortScore (pass@1, %)Cost per attempt
Claude Haiku 5.5low12.7%$0.42
Claude Haiku 5.5medium20.3%$0.68
Claude Haiku 5.5high24.8%$1.04
Claude Haiku 5.5xhigh31.5%$1.75
Claude Haiku 5.5max39.2%$2.64
Claude Haiku 4.5max0%$0.79
Claude Sonnet 5.5 ($0.10 cache reads)low20%$0.62
Claude Sonnet 5.5 ($0.10 cache reads)medium28.8%$0.68
Claude Sonnet 5.5 ($0.10 cache reads)high43%$1.46
Claude Sonnet 5.5 ($0.10 cache reads)xhigh61.5%$4.34
Claude Sonnet 5.5 ($0.10 cache reads)max70.6%$10.4
Claude Sonnet 5.5 ($0.20 cache reads)low20%$0.79
Claude Sonnet 5.5 ($0.20 cache reads)medium28.8%$0.86
Claude Sonnet 5.5 ($0.20 cache reads)high43%$1.97
Claude Sonnet 5.5 ($0.20 cache reads)xhigh61.5%$5.83
Claude Sonnet 5.5 ($0.20 cache reads)max70.6%$14.2
  • Exact score and cost values from the chart data embedded in Anthropic's announcement.
  • Haiku 4.5 is reported at max effort only.
  • Sonnet 5.5 is shown before and after the 7 October cache-read price cut, with identical scores and different costs.
Source: Introducing Claude Haiku 5.5 (Anthropic)

OpenAI: Introducing GPT-6 Sol and LunaRead announcement

GPT-6 Sol and Luna alignment evaluation

Reported by OpenAI on 22 Sept 2026 · exact values

BenchmarkGPT-6 AstraGPT-6 SolGPT-6 LunaGPT-5.6 SolGPT-5.6 Luna
Coding deception rateAlignment · max effort · lower is better0.5%1.3%2.8%10.4%9.5%
  • Bold marks the best reported score in each row.
  • OpenAI's internal evaluation uses tasks deliberately chosen to provoke dishonesty, so deception is much rarer in typical use.
  • The announcement also covers broken search, reviewer bypass, warning circumvention and unauthorized interaction. See OpenAI's system card for those results.
Source: Introducing GPT-6 Sol and Luna (OpenAI)
AutomationBench 1.0.6

Score against cost · reported by OpenAI on 22 Sept 2026 · exact values

AI agents are tested on end-to-end workflows using 47 tools across sales, marketing, operations, support, finance and HR.

  • GPT-6 Luna
  • GPT-5.6 Luna
  • GPT-6 Sol
  • GPT-5.6 Sol
  • GPT-6 Astra
  • Claude Opus 5
  • Claude Fable 5.1
Show data table
ModelEffortScoreCost per task
GPT-6 Lunalow1.2%$0.006
GPT-6 Lunamedium9.4%$0.016
GPT-6 Lunahigh14.5%$0.021
GPT-6 Lunaxhigh12.6%$0.025
GPT-6 Lunamax20.7%$0.037
GPT-5.6 Lunalow1.8%$0.01
GPT-5.6 Lunamedium4.3%$0.02
GPT-5.6 Lunahigh9.1%$0.05
GPT-5.6 Lunaxhigh12.9%$0.06
GPT-5.6 Lunamax17%$0.07
GPT-6 Sollow21.2%$0.19
GPT-6 Solmedium26.9%$0.21
GPT-6 Solhigh31.2%$0.24
GPT-6 Solxhigh33.2%$0.27
GPT-6 Solmax32%$0.34
GPT-5.6 Sollow11.7%$0.31
GPT-5.6 Solmedium19.6%$0.42
GPT-5.6 Solhigh24.8%$0.47
GPT-5.6 Solxhigh26.3%$0.54
GPT-5.6 Solmax28.8%$0.67
GPT-6 Astralow30.3%$1.08
GPT-6 Astramedium34.1%$1.27
GPT-6 Astrahigh37.1%$1.44
GPT-6 Astraxhigh39%$1.50
GPT-6 Astramax41.4%$1.73
Claude Opus 5low20.4%$1.64
Claude Opus 5medium23.9%$2.22
Claude Opus 5high20.5%$2.27
Claude Opus 5xhigh25.3%$2.71
Claude Opus 5max26.9%$3.05
Claude Fable 5.1max31.4%$2.45
  • Exact values from the data labels on OpenAI's charts. OpenAI took competitor scores from publicly available reports.
  • Claude Fable 5.1 ran with Claude Opus 5 as a fallback. OpenAI notes its cost omits the fallback runs (about 40% of tasks), so its real cost is higher.
Source: Introducing GPT-6 Sol and Luna (OpenAI)
Agents' Last Exam V1

Score against cost · reported by OpenAI on 22 Sept 2026 · exact values

AI agents are evaluated on long-horizon, economically valuable tasks spanning 55 sub-industries of professional computer work.

  • GPT-6 Luna
  • GPT-5.6 Luna
  • GPT-6 Sol
  • GPT-5.6 Sol
  • GPT-6 Astra
  • Claude Opus 5
  • Claude Fable 5
Show data table
ModelEffortScoreCost per task
GPT-6 Lunalow36.3%$0.025
GPT-6 Lunamedium46.8%$0.11
GPT-6 Lunahigh43.6%$0.11
GPT-6 Lunaxhigh47.9%$0.11
GPT-6 Lunamax50.9%$0.15
GPT-5.6 Lunalow31%$0.18
GPT-5.6 Lunamedium38.5%$0.38
GPT-5.6 Lunahigh46.1%$0.84
GPT-5.6 Lunaxhigh49.4%$1.55
GPT-5.6 Lunamax50.4%$2.57
GPT-6 Sollow48.7%$0.86
GPT-6 Solmedium53.1%$1.27
GPT-6 Solhigh52.6%$1.53
GPT-6 Solxhigh55.4%$1.67
GPT-6 Solmax56.4%$2.93
GPT-5.6 Sollow45.1%$1.63
GPT-5.6 Solmedium52.1%$3.41
GPT-5.6 Solhigh52.4%$3.79
GPT-5.6 Solxhigh53.6%$5.08
GPT-5.6 Solmax52.8%$7.13
GPT-6 Astralow53.4%$3.07
GPT-6 Astramedium57.6%$4.10
GPT-6 Astrahigh57.8%$4.64
GPT-6 Astraxhigh58.3%$5.40
GPT-6 Astramax59.3%$6.23
Claude Opus 5low51.9%$4.06
Claude Opus 5medium53%$5.29
Claude Opus 5high55.9%$7.29
Claude Opus 5xhigh55.5%$10
Claude Opus 5max52.7%$9.76
Claude Fable 5xhigh48.7%$28.6
Claude Fable 5adaptive41.3%$15.2
  • Exact values from the data labels on OpenAI's charts. OpenAI took competitor scores from publicly available reports.
  • Claude Fable 5 ran with Claude Opus 4.8 as a fallback.
Source: Introducing GPT-6 Sol and Luna (OpenAI)
Factual error rate on difficult prompts

Answers with any factual error against cost · lower is better · reported by OpenAI on 22 Sept 2026 · exact values

OpenAI's internal evaluation on de-identified ChatGPT conversations where users had flagged a factual error from a prior model.

  • GPT-6 Luna
  • GPT-5.6 Luna
  • GPT-6 Sol
  • GPT-5.6 Sol
  • GPT-6 Astra
Show data table
ModelEffortError rateCost per task
GPT-6 Lunalow27.7%$0.0024
GPT-6 Lunamedium17.5%$0.0033
GPT-6 Lunahigh12.5%$0.0045
GPT-6 Lunaxhigh10.2%$0.0062
GPT-6 Lunamax7.6%$0.012
GPT-5.6 Lunalow36.8%$0.0045
GPT-5.6 Lunamedium24.7%$0.0062
GPT-5.6 Lunahigh17.3%$0.011
GPT-5.6 Lunaxhigh12.2%$0.015
GPT-5.6 Lunamax12%$0.022
GPT-6 Sollow11.4%$0.05
GPT-6 Solmedium6.9%$0.069
GPT-6 Solhigh5.1%$0.099
GPT-6 Solxhigh4.5%$0.13
GPT-6 Solmax4.6%$0.18
GPT-5.6 Sollow19.4%$0.10
GPT-5.6 Solmedium15%$0.15
GPT-5.6 Solhigh10.8%$0.23
GPT-5.6 Solxhigh8.4%$0.39
GPT-5.6 Solmax8.5%$0.87
GPT-6 Astralow6.3%$0.24
GPT-6 Astramedium4.4%$0.31
GPT-6 Astrahigh3.9%$0.48
GPT-6 Astraxhigh4%$0.60
GPT-6 Astramax3.9%$0.79
  • Exact values from the data labels on OpenAI's charts. OpenAI took competitor scores from publicly available reports.
  • These prompts were chosen because they caused errors before, so error rates are far higher than in typical use.
Source: Introducing GPT-6 Sol and Luna (OpenAI)
FrontierCode 1.1, main set

Score against cost · reported by OpenAI on 22 Sept 2026 · exact values

AI agents write code graded on correctness and mergeability: test quality, scope discipline, code style and codebase standards.

  • GPT-6 Luna
  • GPT-5.6 Luna
  • GPT-6 Sol
  • GPT-5.6 Sol
  • GPT-6 Astra
  • Claude Opus 5
  • Claude Fable 5.1
Show data table
ModelEffortScoreCost per task
GPT-6 Lunalow25.7%$0.021
GPT-6 Lunamedium35.5%$0.053
GPT-6 Lunahigh37.3%$0.067
GPT-6 Lunaxhigh37.1%$0.073
GPT-6 Lunamax42.4%$0.11
GPT-5.6 Lunalow15.4%$0.06
GPT-5.6 Lunamedium25.7%$0.13
GPT-5.6 Lunahigh35.9%$0.23
GPT-5.6 Lunaxhigh38.9%$0.31
GPT-5.6 Lunamax39.8%$0.37
GPT-6 Sollow37.3%$0.45
GPT-6 Solmedium45.9%$0.80
GPT-6 Solhigh47.7%$1.08
GPT-6 Solxhigh48.4%$1.37
GPT-6 Solmax49.3%$2.14
GPT-5.6 Sollow35.4%$1.89
GPT-5.6 Solmedium39.9%$2.69
GPT-5.6 Solhigh45.1%$3.48
GPT-5.6 Solxhigh46.8%$4.15
GPT-5.6 Solmax47.5%$5.19
GPT-6 Astralow45.3%$1.70
GPT-6 Astramedium48.8%$2.43
GPT-6 Astrahigh50.9%$3.01
GPT-6 Astraxhigh50.6%$3.28
GPT-6 Astramax53.3%$4.59
Claude Opus 5low41.9%$2.68
Claude Opus 5medium53.4%$4.31
Claude Opus 5high48%$7.24
Claude Opus 5xhigh43.6%$9.14
Claude Opus 5max48%$11.4
Claude Fable 5.1low49.8%$2.38
Claude Fable 5.1medium50.9%$3.28
Claude Fable 5.1high50.3%$5.27
Claude Fable 5.1xhigh48.7%$9.27
Claude Fable 5.1max50.3%$12.8
  • Exact values from the data labels on OpenAI's charts. OpenAI took competitor scores from publicly available reports.
Source: Introducing GPT-6 Sol and Luna (OpenAI)
DeepSWE 1.1

Score against cost · reported by OpenAI on 22 Sept 2026 · exact values

AI agents solve original, long-horizon software engineering tasks in real codebases.

  • GPT-6 Luna
  • GPT-5.6 Luna
  • GPT-6 Sol
  • GPT-5.6 Sol
  • GPT-6 Astra
  • Claude Opus 5
  • Claude Fable 5
Show data table
ModelEffortScoreCost per task
GPT-6 Lunalow2.4%$0.0057
GPT-6 Lunamedium44.5%$0.052
GPT-6 Lunahigh59.3%$0.084
GPT-6 Lunaxhigh61.3%$0.11
GPT-6 Lunamax66.6%$0.22
GPT-5.6 Lunalow1.2%$0.011
GPT-5.6 Lunamedium9.3%$0.031
GPT-5.6 Lunahigh42.4%$0.13
GPT-5.6 Lunaxhigh56.2%$0.27
GPT-5.6 Lunamax62.2%$0.53
GPT-6 Sollow37.2%$0.16
GPT-6 Solmedium56.6%$0.38
GPT-6 Solhigh65.3%$0.64
GPT-6 Solxhigh66.6%$1.00
GPT-6 Solmax68.8%$2.74
GPT-5.6 Sollow45.4%$0.82
GPT-5.6 Solmedium61.1%$1.42
GPT-5.6 Solhigh69.4%$2.66
GPT-5.6 Solxhigh70.7%$3.60
GPT-5.6 Solmax72.7%$6.46
GPT-6 Astralow67%$1.60
GPT-6 Astramedium72.8%$3.08
GPT-6 Astrahigh73.2%$3.92
GPT-6 Astraxhigh74.1%$4.43
GPT-6 Astramax73.2%$7.50
Claude Opus 5low58.1%$1.66
Claude Opus 5medium68.9%$3.29
Claude Opus 5high72.8%$6.08
Claude Opus 5xhigh73.2%$9.07
Claude Opus 5max73.7%$11.8
Claude Fable 5low59.6%$3.76
Claude Fable 5medium65.4%$6.09
Claude Fable 5high68.6%$9.18
Claude Fable 5xhigh69.9%$13.4
Claude Fable 5max69.7%$21.6
  • Exact values from the data labels on OpenAI's charts. OpenAI took competitor scores from publicly available reports.
  • OpenAI used Claude Fable 5 scores because Fable 5.1 scores were unavailable.
Source: Introducing GPT-6 Sol and Luna (OpenAI)
OSWorld 2.0, offline set

Partial reward against cost · release v2026.08.08 · reported by OpenAI on 22 Sept 2026 · exact values

AI agents attempt long-horizon computer-use workflows covering everyday and professional tasks.

  • GPT-6 Luna
  • GPT-5.6 Luna
  • GPT-6 Sol
  • GPT-5.6 Sol
  • GPT-6 Astra
  • Claude Opus 5
Show data table
ModelEffortScoreCost per task
GPT-6 Lunalow8.3%$0.03
GPT-6 Lunamedium31.5%$0.062
GPT-6 Lunahigh41.4%$0.12
GPT-6 Lunaxhigh46.7%$0.16
GPT-6 Lunamax52.7%$0.27
GPT-5.6 Lunalow12%$0.019
GPT-5.6 Lunamedium22.7%$0.052
GPT-5.6 Lunahigh35.9%$0.16
GPT-5.6 Lunaxhigh48%$0.36
GPT-5.6 Lunamax52.7%$0.49
GPT-6 Sollow43.9%$0.97
GPT-6 Solmedium54%$1.32
GPT-6 Solhigh58.3%$1.64
GPT-6 Solxhigh60.5%$2.21
GPT-6 Solmax64.4%$3.25
GPT-5.6 Sollow29.8%$0.91
GPT-5.6 Solmedium49.7%$2.73
GPT-5.6 Solhigh56.5%$4.46
GPT-5.6 Solxhigh60.9%$5.93
GPT-5.6 Solmax66.2%$7.71
GPT-6 Astralow62.2%$2.55
GPT-6 Astramedium69.3%$5.10
GPT-6 Astrahigh70%$6.60
GPT-6 Astraxhigh71.3%$7.17
GPT-6 Astramax73.5%$9.07
Claude Opus 5low55.2%$9.88
Claude Opus 5medium60.3%$12.7
Claude Opus 5high65.9%$15.9
Claude Opus 5xhigh70.1%$23.9
Claude Opus 5max70.2%$24.1
  • Exact values from the data labels on OpenAI's charts. OpenAI took competitor scores from publicly available reports.
  • Claude Opus 5 values match the official OSWorld 2.0 leaderboard (osworld-v2.xlang.ai).
Source: Introducing GPT-6 Sol and Luna (OpenAI)

Anthropic: Introducing Claude Opus 5.5Read announcement

Claude Opus 5.5 launch results

Reported by Anthropic on 22 Sept 2026 · exact values

BenchmarkClaude Opus 5.5Claude Fable 5.1Claude Opus 5GPT-6 AstraGPT-5.6 Sol
Terminal-Bench 4.0Agentic coding66.4%55.8%52.3%57.9%37.3%
FrontierCode v1.1 (main)Agentic coding54.4%50.3%48%53.3%47.5%
CursorBench 4.0Agentic coding57.8%51.8%46.6%–41.7%
GDPval-AA v2.1Knowledge work1846 Elo1735 Elo1708 Elo1542 Elo1588 Elo
AutomationBenchBusiness workflows40%31.4%26.9%41.4%28.8%
Humanity's Last ExamMultidisciplinary reasoning · with tools67.7%65.6%63.6%57.2%–
Terminal-Bench-Science 0.1Agentic scientific research58.7%52.6%29%64.6%22.4%
OSWorld 2.1Computer use · partial81.8%80.7%74%––
ChartographyVisual chart recognition · with tools89%88.4%83.4%––
  • Bold marks the best reported score in each row.
  • Claude Opus 5.5 results use adaptive thinking at max effort unless noted.
  • Terminal-Bench 4.0 shows each model's highest score: Claude Opus 5.5 at xhigh effort and GPT-6 Astra at high effort. GPT-6 Astra and GPT-5.6 Sol figures there are as reported by OpenAI.
  • A dash means the lab did not report a result for that model.
Source: Introducing Claude Opus 5.5 (Anthropic)
Terminal-Bench 4.0

Accuracy against cost · reported by Anthropic on 22 Sept 2026 · exact values

Measures how well a model completes complex, multi-step professional tasks within a command line interface.

  • Claude Opus 5.5
  • Claude Fable 5.1
  • Claude Opus 5
  • GPT-6 Astra
  • GPT-5.6 Sol
Show data table
ModelEffortScoreCost per attempt
Claude Opus 5.5low38.5%$1.29
Claude Opus 5.5medium57.6%$2.94
Claude Opus 5.5high64.2%$3.88
Claude Opus 5.5xhigh66.4%$7.35
Claude Opus 5.5max64.8%$11.2
Claude Fable 5.1low40.2%$5.70
Claude Fable 5.1medium43.4%$7.80
Claude Fable 5.1high49.4%$10.5
Claude Fable 5.1xhigh51.3%$15.8
Claude Fable 5.1max55.8%$19.5
Claude Opus 5low28.5%$4.25
Claude Opus 5medium41.2%$7.00
Claude Opus 5high47%$10.6
Claude Opus 5xhigh50.6%$13.5
Claude Opus 5max52.3%$15.8
GPT-6 Astralow49.7%$4.95
GPT-6 Astramedium53.9%$6.15
GPT-6 Astrahigh57.9%$7.21
GPT-6 Astraxhigh57.6%$7.48
GPT-6 Astramax56.7%$10.3
GPT-5.6 Sollow7.9%$1.46
GPT-5.6 Solmedium20.9%$2.69
GPT-5.6 Solhigh26.1%$4.12
GPT-5.6 Solxhigh28.5%$5.39
GPT-5.6 Solmax37.3%$7.89
  • Exact values from the data published with Anthropic's announcement.
  • GPT-6 Astra and GPT-5.6 Sol figures are as reported by OpenAI.
Source: Introducing Claude Opus 5.5 (Anthropic)
FrontierCode v1.1, main set

Accuracy against cost · reported by Anthropic on 22 Sept 2026 · exact values

Measures whether an agent's code changes would be merged.

  • Claude Opus 5.5
  • Claude Fable 5.1
  • Claude Opus 5
  • GPT-6 Astra
  • GPT-5.6 Sol
Show data table
ModelEffortScoreCost per task
Claude Opus 5.5low47.3%$0.40
Claude Opus 5.5medium54.64%$0.80
Claude Opus 5.5high53.99%$1.09
Claude Opus 5.5xhigh51.42%$2.25
Claude Opus 5.5max54.43%$6.19
Claude Fable 5.1low52.8%$2.47
Claude Fable 5.1medium50.91%$3.28
Claude Fable 5.1high50.34%$5.27
Claude Fable 5.1xhigh48.73%$9.27
Claude Fable 5.1max50.28%$12.8
Claude Opus 5low41.95%$2.64
Claude Opus 5medium53.38%$4.61
Claude Opus 5high47.99%$7.62
Claude Opus 5xhigh43.65%$8.99
Claude Opus 5max48.04%$12.3
GPT-6 Astralow45.27%$1.59
GPT-6 Astramedium48.83%$2.28
GPT-6 Astrahigh50.94%$2.85
GPT-6 Astraxhigh50.62%$3.10
GPT-6 Astramax53.26%$4.36
GPT-5.6 Sollow35.44%$1.75
GPT-5.6 Solmedium39.93%$2.50
GPT-5.6 Solhigh45.06%$3.25
GPT-5.6 Solxhigh46.84%$3.88
GPT-5.6 Solmax47.49%$4.85
  • Exact values from the data published with Anthropic's announcement.
Source: Introducing Claude Opus 5.5 (Anthropic)
CursorBench 4.0

Accuracy against cost · reported by Anthropic on 22 Sept 2026 · exact values

Evaluates coding agents on ambiguous, multi-file tasks taken from real Cursor sessions.

  • Claude Opus 5.5
  • Claude Fable 5.1
  • Claude Opus 5
  • GPT-5.6 Sol
Show data table
ModelEffortScoreCost per task
Claude Opus 5.5low43.7%$1.18
Claude Opus 5.5medium52.5%$2.90
Claude Opus 5.5high56%$3.97
Claude Opus 5.5xhigh56%$6.99
Claude Opus 5.5max57.8%$13.4
Claude Fable 5.1low45.1%$5.44
Claude Fable 5.1medium46.8%$7.05
Claude Fable 5.1high49.2%$9.08
Claude Fable 5.1xhigh51.6%$13
Claude Fable 5.1max51.8%$17.3
Claude Opus 5low40.7%$4.87
Claude Opus 5medium43.3%$6.94
Claude Opus 5high44.7%$9.00
Claude Opus 5xhigh46.1%$11.4
Claude Opus 5max46.6%$11.9
GPT-5.6 Sollow24.6%$0.87
GPT-5.6 Solmedium31.1%$1.77
GPT-5.6 Solhigh35.7%$2.85
GPT-5.6 Solxhigh37.7%$4.40
GPT-5.6 Solmax41.7%$8.23
  • Exact values from the data published with Anthropic's announcement.
Source: Introducing Claude Opus 5.5 (Anthropic)
GDPval-AA v2.1

Elo against cost · reported by Anthropic on 22 Sept 2026 · exact values

Artificial Analysis's evaluation of agents on real-world professional work across 44 occupations.

  • Claude Opus 5.5
  • Claude Fable 5.1
  • Claude Opus 5
  • GPT-6 Astra
  • GPT-5.6 Sol
Show data table
ModelEffortEloCost per task
Claude Opus 5.5low1224 Elo$0.21
Claude Opus 5.5medium1576 Elo$0.86
Claude Opus 5.5high1692 Elo$1.54
Claude Opus 5.5xhigh1820 Elo$4.21
Claude Opus 5.5max1846 Elo$8.92
Claude Fable 5.1low1450 Elo$1.41
Claude Fable 5.1medium1536 Elo$2.17
Claude Fable 5.1high1617 Elo$3.43
Claude Fable 5.1xhigh1721 Elo$7.09
Claude Fable 5.1max1735 Elo$9.59
Claude Opus 5low1294 Elo$0.58
Claude Opus 5medium1476 Elo$1.36
Claude Opus 5high1581 Elo$3.03
Claude Opus 5xhigh1676 Elo$4.96
Claude Opus 5max1708 Elo$6.76
GPT-6 Astralow1366 Elo$0.85
GPT-6 Astramedium1468 Elo$1.82
GPT-6 Astrahigh1485 Elo$2.43
GPT-6 Astraxhigh1516 Elo$3.04
GPT-6 Astramax1542 Elo$4.53
GPT-5.6 Sollow1289 Elo$0.27
GPT-5.6 Solmedium1403 Elo$0.60
GPT-5.6 Solhigh1480 Elo$1.11
GPT-5.6 Solxhigh1548 Elo$1.70
GPT-5.6 Solmax1588 Elo$2.81
  • Exact values from the data published with Anthropic's announcement.
Source: Introducing Claude Opus 5.5 (Anthropic)
AutomationBench

Pass rate against cost · reported by Anthropic on 22 Sept 2026 · exact values

Built by Zapier, tests whether an agent can carry out real business workflows across many connected apps.

  • Claude Opus 5.5
  • Claude Opus 5
  • GPT-6 Astra
  • GPT-5.6 Sol
Show data table
ModelEffortPass rateCost per task
Claude Opus 5.5low23.3%$0.50
Claude Opus 5.5medium28.6%$0.64
Claude Opus 5.5high32%$0.70
Claude Opus 5.5xhigh34.4%$0.86
Claude Opus 5.5max40%$1.37
Claude Opus 5low20.4%$0.75
Claude Opus 5medium23.9%$0.89
Claude Opus 5high20.6%$1.03
Claude Opus 5xhigh25.3%$1.15
Claude Opus 5max26.9%$1.27
GPT-6 Astralow30.3%$1.08
GPT-6 Astramedium34.1%$1.28
GPT-6 Astrahigh37.1%$1.45
GPT-6 Astraxhigh39%$1.53
GPT-6 Astramax41.4%$1.77
GPT-5.6 Sollow11.7%$0.42
GPT-5.6 Solmedium19.6%$0.57
GPT-5.6 Solhigh24.8%$0.64
GPT-5.6 Solxhigh26.3%$0.74
GPT-5.6 Solmax28.8%$0.91
  • Exact values from the data published with Anthropic's announcement.
  • Runs were performed by Zapier without fallback models, so safeguard interventions counted as failures.
Source: Introducing Claude Opus 5.5 (Anthropic)
WANDR

Accuracy against cost · reported by Anthropic on 22 Sept 2026 · exact values

Perplexity's benchmark measuring agents on large data collection tasks.

  • Claude Opus 5.5
  • Claude Fable 5.1
  • Claude Opus 5
Show data table
ModelEffortScoreCost per attempt
Claude Opus 5.5low31.2%$1.20
Claude Opus 5.5medium62.8%$11.2
Claude Opus 5.5high67.3%$16.9
Claude Opus 5.5xhigh71.3%$29.1
Claude Opus 5.5max72.3%$37.9
Claude Fable 5.1low63.3%$23.3
Claude Fable 5.1medium64.5%$27.4
Claude Fable 5.1high66.7%$33.6
Claude Fable 5.1xhigh67.7%$42.8
Claude Fable 5.1max68.7%$49
Claude Opus 5low50.5%$10.7
Claude Opus 5medium58.1%$24
Claude Opus 5high64.6%$43.5
Claude Opus 5xhigh67%$53.8
Claude Opus 5max67.2%$61.6
  • Exact values from the data published with Anthropic's announcement.
  • Claude models ran with offline web search and fetch tools and a 980K-token task budget. This differs from Perplexity's published setup, so scores are not comparable with Perplexity's leaderboard.
Source: Introducing Claude Opus 5.5 (Anthropic)

OpenAI: GPT-6 Astra: The next generation in intelligence for workRead announcement

Terminal-Bench 4.0

Accuracy against API cost · reported by OpenAI on 9 Sept 2026 · exact values

Tests agents on complex terminal-based tasks, including software engineering, system configuration and data analysis.

  • GPT-6 Astra
  • GPT-5.6 Sol
  • Claude Fable 5.1
  • Claude Fable 5
  • Claude Opus 5
Show data table
ModelEffortAccuracyCost per task
GPT-6 Astralow49.7%$4.95
GPT-6 Astramedium53.9%$6.15
GPT-6 Astrahigh57.9%$7.21
GPT-6 Astraxhigh57.6%$7.48
GPT-6 Astramax56.7%$10.3
GPT-5.6 Sollow7.9%$1.46
GPT-5.6 Solmedium20.9%$2.69
GPT-5.6 Solhigh26.1%$4.12
GPT-5.6 Solxhigh28.5%$5.39
GPT-5.6 Solmax37.3%$7.89
Claude Fable 5.1low40.2%$5.70
Claude Fable 5.1medium43.4%$7.80
Claude Fable 5.1high49.4%$10.5
Claude Fable 5.1xhigh51.3%$15.8
Claude Fable 5.1max55.8%$19.5
Claude Fable 5max44.5%$22.2
Claude Opus 5max52.6%$18.4
  • Exact values from the data labels on OpenAI's charts. OpenAI took competitor scores from publicly available reports.
  • Claude Fable 5 and Claude Opus 5 are shown at max effort only, as published.
Source: GPT-6 Astra: The next generation in intelligence for work (OpenAI)

Anthropic: Introducing Claude Sonnet 5.5Read announcement

Claude Sonnet 5.5 launch results

Reported by Anthropic on 28 Sept 2026 · exact values

BenchmarkClaude Sonnet 5.5Claude Sonnet 5Claude Opus 5.5GPT-6 Sol
Terminal-Bench 4.0Agentic coding70.6%10.3%66.4%–
FrontierCode 1.1 (main)Agentic coding · Sonnet 5.5 at max effort, 52.1% at xhigh46.2%42.4%54.4%49.3%
CursorBench 4.0Agentic coding55.5%34.1%57.8%–
GDPval-AA v2.1Knowledge work1844 Elo1449 Elo1846 Elo1487 Elo
AA-Briefcase v1.1Knowledge work1811 Elo1359 Elo1822 Elo1483 Elo
Humanity's Last ExamMultidisciplinary reasoning · with tools64.5%54.9%67.7%–
OSWorld 2.1Computer use · partial80.1%57%81.8%–
ChartographyVisual chart recognition · no tools61.6%15.6%64.4%53.6%
  • Bold marks the best reported score in each row.
  • Claude Opus 5.5's Terminal-Bench 4.0 score is at xhigh effort, its highest.
  • OpenAI recently fixed a bug affecting GPT-6 Sol's image understanding; its GDPval-AA, AA-Briefcase and Chartography scores may predate the fix.
  • A dash means the lab did not report a result for that model.
Source: Introducing Claude Sonnet 5.5 (Anthropic)
Terminal-Bench 4.0

Accuracy against cost · reported by Anthropic on 28 Sept 2026 · exact values

Measures how well a model completes complex, multi-step professional tasks within a command-line interface.

  • Claude Sonnet 5.5
  • Claude Opus 5.5
  • Claude Sonnet 5
  • GPT-5.6 Sol
Show data table
ModelEffortScoreCost per attempt
Claude Sonnet 5.5low20%$0.76
Claude Sonnet 5.5medium28.8%$0.83
Claude Sonnet 5.5high43%$1.94
Claude Sonnet 5.5xhigh61.5%$5.30
Claude Sonnet 5.5max70.6%$12.5
Claude Opus 5.5low38.5%$1.29
Claude Opus 5.5medium57.6%$2.94
Claude Opus 5.5high64.2%$3.88
Claude Opus 5.5xhigh66.4%$7.35
Claude Opus 5.5max64.8%$11.2
Claude Sonnet 5low3.2%$3.78
Claude Sonnet 5medium4.4%$4.83
Claude Sonnet 5high4.5%$8.20
Claude Sonnet 5xhigh7%$9.95
Claude Sonnet 5max10.3%$11.6
GPT-5.6 Sollow7.9%$1.46
GPT-5.6 Solmedium20.9%$2.69
GPT-5.6 Solhigh26.1%$4.12
GPT-5.6 Solxhigh28.5%$5.39
GPT-5.6 Solmax37.3%$7.89
  • Exact values from the data published with Anthropic's announcement.
  • OpenAI did not report GPT-6 Sol on this benchmark, so Anthropic shows GPT-5.6 Sol.
Source: Introducing Claude Sonnet 5.5 (Anthropic)
FrontierCode v1.1, main set

Accuracy against cost · reported by Anthropic on 28 Sept 2026 · exact values

Measures whether an agent's code changes would be merged.

  • Claude Sonnet 5.5
  • Claude Opus 5.5
  • Claude Sonnet 5
  • GPT-6 Sol
Show data table
ModelEffortScoreCost per task
Claude Sonnet 5.5low29.3%$0.19
Claude Sonnet 5.5medium36.5%$0.24
Claude Sonnet 5.5high49.4%$0.42
Claude Sonnet 5.5xhigh52.1%$1.59
Claude Sonnet 5.5max46.2%$20.8
Claude Opus 5.5low47.3%$0.40
Claude Opus 5.5medium54.6%$0.80
Claude Opus 5.5high54%$1.09
Claude Opus 5.5xhigh51.4%$2.25
Claude Opus 5.5max54.4%$6.19
Claude Sonnet 5low28.7%$2.39
Claude Sonnet 5medium35.2%$3.81
Claude Sonnet 5high39.4%$6.10
Claude Sonnet 5xhigh42.7%$10.1
Claude Sonnet 5max42.4%$17.1
GPT-6 Sollow37.3%$0.43
GPT-6 Solmedium45.9%$0.77
GPT-6 Solhigh47.7%$1.04
GPT-6 Solxhigh48.4%$1.32
GPT-6 Solmax49.3%$2.07
  • Exact values from the data published with Anthropic's announcement.
Source: Introducing Claude Sonnet 5.5 (Anthropic)
CursorBench 4.0

Accuracy against cost · reported by Anthropic on 28 Sept 2026 · exact values

Evaluates coding agents on ambiguous, multi-file tasks taken from real Cursor sessions.

  • Claude Sonnet 5.5
  • Claude Opus 5.5
  • Claude Sonnet 5
  • GPT-5.6 Sol
Show data table
ModelEffortScoreCost per task
Claude Sonnet 5.5low35.8%$0.50
Claude Sonnet 5.5medium39.2%$0.70
Claude Sonnet 5.5high47.8%$1.67
Claude Sonnet 5.5xhigh53.1%$3.88
Claude Sonnet 5.5max55.5%$9.67
Claude Opus 5.5low43.7%$1.17
Claude Opus 5.5medium52.5%$2.91
Claude Opus 5.5high56%$3.97
Claude Opus 5.5xhigh56%$6.98
Claude Opus 5.5max57.8%$13.4
Claude Sonnet 5low24.1%$1.39
Claude Sonnet 5medium28%$2.31
Claude Sonnet 5high30.8%$3.48
Claude Sonnet 5xhigh32%$4.55
Claude Sonnet 5max34.1%$7.17
GPT-5.6 Sollow24.6%$0.87
GPT-5.6 Solmedium31.1%$1.77
GPT-5.6 Solhigh35.7%$2.85
GPT-5.6 Solxhigh37.7%$4.40
GPT-5.6 Solmax41.7%$8.23
  • Exact values from the data published with Anthropic's announcement.
  • CursorBench does not report GPT-6 Sol publicly, so Anthropic shows GPT-5.6 Sol.
Source: Introducing Claude Sonnet 5.5 (Anthropic)
AA-Briefcase v1.1

Elo against cost · reported by Anthropic on 28 Sept 2026 · exact values

Artificial Analysis's benchmark of long-horizon knowledge work.

  • Claude Sonnet 5.5
  • Claude Opus 5.5
  • Claude Sonnet 5
  • GPT-6 Sol
Show data table
ModelEffortEloCost per task
Claude Sonnet 5.5low1264 Elo$0.87
Claude Sonnet 5.5medium1461 Elo$1.64
Claude Sonnet 5.5high1634 Elo$3.95
Claude Sonnet 5.5xhigh1746 Elo$9.63
Claude Sonnet 5.5max1811 Elo$29.2
Claude Opus 5.5low1285 Elo$1.15
Claude Opus 5.5medium1642 Elo$4.40
Claude Opus 5.5high1705 Elo$6.27
Claude Opus 5.5xhigh1780 Elo$12.3
Claude Opus 5.5max1822 Elo$21
Claude Sonnet 5low923 Elo$0.82
Claude Sonnet 5medium1056 Elo$1.73
Claude Sonnet 5high1177 Elo$3.76
Claude Sonnet 5xhigh1274 Elo$7.56
Claude Sonnet 5max1359 Elo$14.4
GPT-6 Sollow905 Elo$0.12
GPT-6 Solmedium1142 Elo$0.34
GPT-6 Solhigh1289 Elo$0.63
GPT-6 Solxhigh1364 Elo$1.19
GPT-6 Solmax1483 Elo$2.67
  • Exact values from the data published with Anthropic's announcement.
  • Artificial Analysis ran Sonnet 5.5 on a pre-release deployment with a since-fixed structured output bug, which may slightly understate its score.
Source: Introducing Claude Sonnet 5.5 (Anthropic)

Anthropic: Introducing Claude Fable 5.1 and Claude Mythos 5.1Read announcement

Claude Fable 5.1 launch results

Reported by Anthropic on 1 Sept 2026 · exact values

BenchmarkClaude Fable 5.1Claude Fable 5Claude Opus 5GPT-5.6 Sol
Terminal-Bench-Science 0.1Agentic scientific research52.6%24.7%29%22.4%
Terminal-Bench 4.0Agentic coding55.8%42%52.3%37.3%
GDPval-AA v2Knowledge work1853 Elo1723 Elo1824 Elo1711 Elo
OSWorld 2.0Computer use · partial77.9%72.9%75.4%–
OSWorld 2.0 (strict)Computer use · strict41.7%36.1%39.6%–
Humanity's Last Exam (no tools)Multidisciplinary reasoning · no tools60.9%57.8%56.6%–
Humanity's Last Exam (with tools)Multidisciplinary reasoning · with tools65%63.8%63.6%–
AutomationBenchBusiness workflows31.4%17.1%26.9%19.6%
CursorBench 3.2.0Agentic coding73.4%70.5%70%67.2%
  • Bold marks the best reported score in each row.
  • Fable 5.1 was evaluated with production safeguards enabled. Where they intervened, Fable 5.1 and Fable 5 scored zero on OSWorld 2.0 and Fable 5 scored zero on AutomationBench.
  • Claude Mythos 5.1, the same underlying model with access limited to vetted users, scores 60.9% on Terminal-Bench 4.0. Mythos has no public API price, so it is not tracked here.
  • A dash means the lab did not report a result for that model.
Source: Introducing Claude Fable 5.1 and Claude Mythos 5.1 (Anthropic)
Terminal-Bench-Science 0.1

Accuracy against cost · reported by Anthropic on 1 Sept 2026 · exact values

Terminal-based agentic scientific research tasks, in Anthropic's agentic scientific research category.

  • Claude Fable 5.1
  • Claude Fable 5
Show data table
ModelEffortScoreCost per task
Claude Fable 5.1low26.3%$11.1
Claude Fable 5.1medium35.7%$14.9
Claude Fable 5.1high40%$20.3
Claude Fable 5.1xhigh49.5%$31.8
Claude Fable 5.1max52.6%$37.9
Claude Fable 5low12.3%$17.1
Claude Fable 5medium21.4%$25
Claude Fable 5high25%$34.3
Claude Fable 5xhigh23.4%$36
Claude Fable 5max24.7%$44.1
  • Exact values from the data published with Anthropic's announcement.
  • Standard error is about 3.5 to 4.5 points per model.
Source: Introducing Claude Fable 5.1 and Claude Mythos 5.1 (Anthropic)
Humanity's Last Exam, with tools

Pass rate against cost · reported by Anthropic on 1 Sept 2026 · exact values

Expert-level questions across many disciplines, answered with access to tools.

  • Claude Fable 5.1
  • Claude Fable 5
Show data table
ModelEffortPass rateCost per task
Claude Fable 5.1low60%$0.52
Claude Fable 5.1medium62.96%$0.67
Claude Fable 5.1high64.76%$1.05
Claude Fable 5.1xhigh65.08%$2.28
Claude Fable 5.1max65%$3.20
Claude Fable 5low59.64%$0.61
Claude Fable 5medium61.4%$1.01
Claude Fable 5high63.16%$1.42
Claude Fable 5xhigh63.52%$1.92
Claude Fable 5max63.8%$3.44
  • Exact values from the data published with Anthropic's announcement.
Source: Introducing Claude Fable 5.1 and Claude Mythos 5.1 (Anthropic)
Humanity's Last Exam, no tools

Pass rate against cost · reported by Anthropic on 1 Sept 2026 · exact values

Expert-level questions across many disciplines, answered without tools.

  • Claude Fable 5.1
  • Claude Fable 5
Show data table
ModelEffortPass rateCost per task
Claude Fable 5.1low53.16%$0.30
Claude Fable 5.1medium55.92%$0.46
Claude Fable 5.1high57.96%$0.75
Claude Fable 5.1xhigh60.36%$1.53
Claude Fable 5.1max60.92%$2.23
Claude Fable 5low50.6%$0.17
Claude Fable 5medium55.88%$0.40
Claude Fable 5high56.88%$0.62
Claude Fable 5xhigh57.44%$0.91
Claude Fable 5max57.76%$1.70
  • Exact values from the data published with Anthropic's announcement.
Source: Introducing Claude Fable 5.1 and Claude Mythos 5.1 (Anthropic)
CursorBench 3.2.0

Accuracy against cost · reported by Anthropic on 1 Sept 2026 · exact values

Evaluates coding agents on tasks taken from real Cursor sessions (earlier release than CursorBench 4.0).

  • Claude Fable 5.1
  • Claude Fable 5
Show data table
ModelEffortScoreCost per task
Claude Fable 5.1low66.2%$2.90
Claude Fable 5.1medium68%$3.53
Claude Fable 5.1high69.4%$4.80
Claude Fable 5.1xhigh72.8%$6.96
Claude Fable 5.1max73.4%$9.64
Claude Fable 5low62.1%$4.46
Claude Fable 5medium65.2%$6.80
Claude Fable 5high66.5%$8.77
Claude Fable 5xhigh68.4%$11.7
Claude Fable 5max70.5%$17.3
  • Exact values from the data published with Anthropic's announcement.
Source: Introducing Claude Fable 5.1 and Claude Mythos 5.1 (Anthropic)

Google DeepMind: Gemini 4 Argon performance and methodologyRead announcement

Gemini 4 Argon launch results

Reported by Google DeepMind on 30 Sept 2026 · exact values

Exact published scores · Higher is better. Unreported results are omitted.

BenchmarkGemini 4 ArgonGPT-6 AstraClaude Fable 5.1Claude Opus 5.5
Vals IndexKnowledge work68.9%63.1%65.8%67%
AutomationBenchKnowledge work · Score51.3%41.4%31.4%42.5%
Vals Finance Agent v2Knowledge work65.4%53.5%58.9%58.6%
Harvey's Legal Agent BenchmarkKnowledge work19.6%5.4%6.7%3.8%
DeepSWE v1.1Agentic coding77.9%74.1%67.4%74.2%
FrontierSWE v2Agentic coding55%65.5%56.3%62.3%
Vibe Code BenchAgentic coding91.9%89.6%90.3%90.3%
Terminal-bench 4.0Agentic coding57.4%58.2%57.9%66.4%
PostTrainBenchML engineering45.3%44.3%40.2%49.3%
Terminal-Bench Science 0.1Science and math57.6%68.1%52.6%63.3%
LABBench 2Science and math88.8%85.4%68.6%73.1%
RiemannBenchScience and math76%72%65.6%69.6%
GraphWalksLong context · Up to 128k, BFS (F1)99.7%98.7%91.4%90.6%
GraphWalksLong context · 256k to 1M, BFS (F1)84.2%71.8%65%66.8%
Agent's Last ExamComputer use · Pass rate39.5%34.2%–38.2%
OSWorld-2.0Computer use · Offline subset Partial score69.2%72.6%––
ChartographyMultimodal understanding71.6%71%46.2%66.3%
LVBenchMultimodal understanding91.7%87.5%79.7%83.7%
CWE-bench v1Cybersecurity68%68%58%67%
View Google DeepMind's original charts
Google DeepMind: Gemini 4 Argon DeepSWE v1.1 scores compared with GPT-6 Astra, Claude Fable 5.1 and Claude Opus 5.5.
Original chart published by Google DeepMind, shown for reference.
Google DeepMind: Gemini 4 Argon Vals Index scores compared with GPT-6 Astra, Claude Fable 5.1 and Claude Opus 5.5.
Original chart published by Google DeepMind, shown for reference.
Google DeepMind: Gemini 4 Argon Vals Finance Agent v2 scores compared with GPT-6 Astra, Claude Fable 5.1 and Claude Opus 5.5.
Original chart published by Google DeepMind, shown for reference.
Google DeepMind: Gemini 4 Argon Harvey's Legal Agent Benchmark scores compared with GPT-6 Astra, Claude Fable 5.1 and Claude Opus 5.5.
Original chart published by Google DeepMind, shown for reference.
Google DeepMind: Gemini 4 Argon AutomationBench scores compared with GPT-6 Astra, Claude Fable 5.1 and Claude Opus 5.5.
Original chart published by Google DeepMind, shown for reference.
Google DeepMind: Gemini 4 Argon CWE-bench v1 scores.
Original chart published by Google DeepMind, shown for reference.
  • Bold marks the best reported score in each row.
  • Exact scores from Google DeepMind's performance table. Argon is initially available to trusted testers.
  • OSWorld-2.0 uses the offline subset and partial scoring. GraphWalks reports BFS F1, with separate context ranges.
  • Scores reflect Google's published evaluation setup. Benchmark versions and harnesses can differ from other labs' launch results. A dash means no result was published.
Source: Gemini 4 Argon performance and methodology (Google DeepMind)

Google DeepMind: Gemini 3.8 Flash model cardRead announcement

Gemini 3.8 Flash launch results

Reported by Google DeepMind on 2 Sept 2026 · exact values

Exact published scores · Higher is better. Unreported results are omitted.

BenchmarkGemini 3.8 FlashGemini 3.7 FlashClaude Opus 5Claude Sonnet 5GPT-5.6 SolGPT-5.6 Terra
DeepSWE v1.1Long-horizon software engineering73.7%65.3%74%53.8%72.7%69.6%
GDPVal-AA v2Knowledge work · Elo1545 Elo1482 Elo1824 Elo1584 Elo1710 Elo1528 Elo
Vals Finance Agent v2Financial analyst tasks61.4%59%58.6%53.9%53.8%54.4%
Harvey's Legal Agent BenchmarkComplex legal workflows · All pass rate10%8.8%6.7%5%2.5%0.8%
Terminal-bench 2.1Agentic terminal coding89.4%85.8%89.1%80.4%88.8%87.4%
Terminal-bench 4.0General agent capabilities19.1%11.2%51.8%12.4%37.3%23.6%
GDP.PDFExpert PDF document comprehension · All pass rate35%34%37%28%40%29%
CharXiv ReasoningInformation synthesis from complex charts · No tools86.2%84.5%83.7%70.1%85.8%85.9%
LVBenchLong video understanding · (agentic)87.8%85.4%75.4%68.5%82.1%78.9%
LVBenchLong video understanding · (static)87.1%–––––
HLE-VerifiedMultidisciplinary expert reasoning54.9%53.6%54.4%31%54.5%51.1%
OSWorld-2.0Agentic computer use · Partial score batch tool enabled59%50.6%75.4%42.6%62.6%50.2%
BioMysteryBenchBioinformatics research workflows · Human Solvable88.8%87.1%90.1%87.5%79.5%83.8%
BioMysteryBenchBioinformatics research workflows · Human Difficult56.5%43.5%49.4%34.1%44.7%49.4%
LABBench2Biology real-world research tasks86.2%82.1%84.2%80.1%82.1%81.2%
View Google DeepMind's original charts
Google DeepMind: Gemini 3.8 Flash Vals Finance Agent v2 scores.
Original chart published by Google DeepMind, shown for reference.
Google DeepMind: Gemini 3.8 Flash Harvey's Legal Agent Benchmark scores.
Original chart published by Google DeepMind, shown for reference.
Google DeepMind: Gemini 3.8 Flash HLE-Verified scores.
Original chart published by Google DeepMind, shown for reference.
  • Bold marks the best reported score in each row.
  • Exact scores from the Google DeepMind model card. Gemini 3.7 Flash and GPT-5.6 Terra are benchmark comparisons only.
  • LVBench reports separate agentic and static results for Gemini 3.8 Flash. Competitor cells span both rows in the source; they appear alongside the agentic result here, with no separate static values inferred. OSWorld-2.0 uses partial scoring with the batch tool enabled.
  • These are Google-reported results; keep benchmark versions and evaluation harnesses in mind when comparing other launch tables.
Source: Gemini 3.8 Flash model card (Google DeepMind)

Google DeepMind: Gemini 3.5 Flash-Lite model cardRead announcement

Gemini 3.5 Flash-Lite launch results

Reported by Google DeepMind on 21 Jul 2026 · exact values

Exact published scores · Higher is better. Unreported results are omitted.

BenchmarkGemini 3.5 Flash-LiteGemini 3.1 Flash-LiteGPT-5.4 miniClaude Haiku 4.5
SWE-Bench Pro (Public)Diverse agentic coding tasks54.2%38.3%54.4%39.5%
Terminal-bench 2.1Agentic terminal coding · Terminus-2 harness54%31%59.2%44.2%
MLE-BenchMachine Learning Engineering39.2%22%––
GDPVal-AA v2Knowledge work · Elo1140 Elo642 Elo1171 Elo907 Elo
OSWorld-VerifiedAgentic computer use74%54.3%72.1%50.7%
CharXiv ReasoningInformation synthesis from complex charts · No tools74.5%73.2%80.3%61.7%
CharXiv ReasoningInformation synthesis from complex charts · With tools76.5%75.6%––
GDM-MRCR v2 (8-needle)Long context performance · 128k (average)72.2%60.1%42.7%35.3%
GDM-MRCR v2 (8-needle)Long context performance · 1M (pointwise)21.3%12.3%––
  • Bold marks the best reported score in each row.
  • Exact scores from the Google DeepMind model card. Earlier competitor models are preserved as benchmark comparisons only.
  • CharXiv reports results with and without tools. Long-context results use separate 128K average and 1M pointwise settings. A dash means no result was published.
Source: Gemini 3.5 Flash-Lite model card (Google DeepMind)

Google DeepMind: Gemini 3.1 Pro model cardRead announcement

Gemini 3.1 Pro launch results

Reported by Google DeepMind on 19 Feb 2026 · exact values

Exact published scores · Higher is better. Unreported results are omitted.

BenchmarkGemini 3.1 Pro (high)Gemini 3 Pro (high)Claude Sonnet 4.6 (max)Claude Opus 4.6 (max)GPT-5.2 (xhigh)GPT-5.3-Codex (xhigh)
Humanity's Last ExamAcademic reasoning (full set, text + MM) · No tools44.4%37.5%33.2%40%34.5%–
Humanity's Last ExamAcademic reasoning (full set, text + MM) · Search (blocklist) + Code51.4%45.8%49%53.1%45.5%–
ARC-AGI-2Abstract reasoning puzzles · ARC Prize Verified77.1%31.1%58.3%68.8%52.9%–
GPQA DiamondScientific knowledge · No tools94.3%91.9%89.9%91.3%92.4%–
Terminal-Bench 2.0Agentic terminal coding · Terminus-2 harness68.5%56.9%59.1%65.4%54%64.7%
Terminal-Bench 2.0Agentic terminal coding · Other best self-reported harness · (Codex)––––62.2%77.3%
SWE-Bench VerifiedAgentic coding · Single attempt80.6%76.2%79.6%80.8%80%–
SWE-Bench Pro (Public)Diverse agentic coding tasks · Single attempt54.2%43.3%––55.6%56.8%
LiveCodeBench ProCompetitive coding problems from Codeforces, ICPC, and IOI · Elo2887 Elo2439 Elo––2393 Elo–
SciCodeScientific research coding59%56%47%52%52%–
APEX-AgentsLong horizon professional tasks33.5%18.4%–29.8%23%–
GDPval-AA EloExpert tasks1317 Elo1195 Elo1633 Elo1606 Elo1462 Elo–
τ2-benchAgentic and tool use · Retail90.8%85.3%91.7%91.9%82%–
τ2-benchAgentic and tool use · Telecom99.3%98%97.9%99.3%98.7%–
MCP AtlasMulti-step workflows using MCP69.2%54.1%61.3%59.5%60.6%–
BrowseCompAgentic search · Search + Python + Browse85.9%59.2%74.7%84%65.8%–
MMMU-ProMultimodal understanding and reasoning · No tools80.5%81%74.5%73.9%79.5%–
MMMLUMultilingual Q&A92.6%91.8%89.3%91.1%89.6%–
MRCR v2 (8-needle)Long context performance · 128k (average)84.9%77%84.9%84%83.8%–
MRCR v2 (8-needle)Long context performance · 1M (pointwise)26.3%26.3%––––
View Google DeepMind's original charts
Google DeepMind: Gemini 3.1 Pro launch benchmark comparison.
Original chart published by Google DeepMind, shown for reference.
  • Bold marks the best reported score in each row.
  • Exact scores from the Google DeepMind model card. Gemini models use high thinking, Claude models use max thinking and GPT models use xhigh thinking.
  • Terminal-Bench results using the Terminus-2 harness and the best self-reported Codex harness are separate rows. Do not combine them as one evaluation.
  • Historical competitors are benchmark comparisons only. A dash means no published result or unsupported context length.
Source: Gemini 3.1 Pro model card (Google DeepMind)