OpenAI
o3 combined reasoning with agentic tool use in ChatGPT. It can decide when to combine web search, Python, or image generation. It can also incorporate images into its reasoning rather than only describing them before it starts thinking.
OpenAI reports fewer major errors on difficult practical tasks than with o1. o4-mini launched alongside it as a smaller, cheaper tier. o3-pro adds more compute for longer thinking.
The index averages rank percentiles from 10 documented comparison cohorts.
Page 1 of 11
Overall
225 code-editing tasks from Exercism across C++, Go, Java, JavaScript, Python, and Rust. Aider, the edit format, reasoning budget, and two attempts are part of the result.
higher is betterAs of September 2, 2026Scale: 0% - 100%
Accuracy
65 scientific coding problems with 338 subproblems across 16 research fields. Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
65 scientific coding problems with 338 subproblems across 16 research fields. Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
65 scientific coding problems with 338 subproblems across 16 research fields. Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
102 expert-validated tasks for data-driven scientific work. The scaffold, tools, and code execution are part of the result. HAL reports both 42 and 44 underlying publications in different places.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Accuracy
A random, fully human-validated subset of 50 GitHub issues from 12 Python repositories. The small subset is cheaper but statistically less stable than the full 500 tasks. SWE-Agent and HAL Generalist remain separate.
Note (o3): HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
307 algorithmic programming tasks from Bronze through Platinum with exhaustive tests. The benchmark measures competitive programming, not repository navigation, maintainability, or product experience.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Accuracy
33 automatically verifiable HAL tasks drawn from 214 realistic, time-consuming web research tasks. The published HAL subset is smaller than the full dataset. Reported costs do not account for caching benefits.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
300 verified tasks across 136 dynamic live websites. SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
300 verified tasks across 136 dynamic live websites. SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Human task duration at which a model-agent system reaches a 50% or 80% predicted success probability. Minutes refer to human task duration, not model runtime. METR considers estimates above 16 hours unreliable.
| Sub-task | o3 (This model) | Gemini 3.1 Pro Preview | GPT-5.3-Codex | GPT-5 |
|---|---|---|---|---|
| 50% time horizon | 119.73 | 384.15 (best value in this row) | 349.53 | 203.01 |
| 80% time horizon | 29.98 | 89.8 (best value in this row) | 54.74 | 38.31 |
Unit: minuteshigher is betterAs of August 26, 2026Source: METRbars scaled within each row, not across rows
Page 1 of 11
Each test shows the latest attempt, with no subsequent fixes to the generated code. Earlier runs remain available in the test details. These results do not contribute to an overall score.
A community classic for free-form SVG drawing.
o3
Reasoning (requested): high
Visually reviewed. This individual test does not establish overall model quality.
Generate an SVG of a pelican riding a bicycle
Token limit including reasoning: 8,192 tokens
Task origin (Simon Willison)22.1 / 100
| Specification | Value and source |
|---|---|
| Weights | ProprietarySource |
| Access | APISource |
| Context window | 200,000 tokensSource |
| Knowledge cutoff | June 1, 2024Source |
| Notes | Reasoning model, parameter count not disclosedSource |
| API model ID | o3Source |
| Maximum output | 100,000 tokensSource |
| Knowledge cutoff | 2024-06-01Source |
| Input | Text, ImageSource |
| Output | TextSource |
| API interfaces | Responses API, Chat Completions APISource |
| Reasoning mode | low, medium, high, xhighSource |
| Structured output | YesSource |
| Release | 2025-04-16Source |
