o3
Unavailable in this round
OpenAI
o3 combined reasoning with agentic tool use in ChatGPT. It can decide when to combine web search, Python, or image generation. It can also incorporate images into its reasoning rather than only describing them before it starts thinking.
OpenAI reports fewer major errors on difficult practical tasks than with o1. o4-mini launched alongside it as a smaller, cheaper tier. o3-pro adds more compute for longer thinking.
The LLM comparison hub's selected-panel index averages rank percentiles from 65 documented comparison cohorts. Benchmark meta-scores are shown in the benchmark section.
Benchmark meta-analysis
These Gradually indexes place selected published measurements on a 0 to 100 scale. They are not accuracy percentages and do not guarantee results in your application.
Retrieval dates of the sources used 2026-08-05 - 2026-10-19
Index 4.3 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 5.0 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 2.7 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 1.2 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 4.4 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 4.6 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 6.3 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 7.7 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 13.1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 9.7 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 2.0 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 2.1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Index 3.0 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
Method version: benchmark-meta-score-v3
The index includes active measurements from reviewed primary sources when at least five models are compared in the same documented comparison slice across dataset, harness, agent, tools, and trial count. Each benchmark series and evaluator contributes one shared setup slice from the newest capture. When one source and series include several explicitly documented reasoning tiers, the highest documented tier (none through max) is selected, never the best result value. The explicit source label “Non-reasoning” is treated as none. Unknown or ambiguous tiers exclude the model from that series. Within each series, percentages and rates are logit-transformed, other units stay unchanged, and all values are then standardized. A one-factor model then estimates one shared strength per model and, per series, how sharply it separates the models. Area scores move away from that strength only as far as the model's own results in the area support. The overall score weights the core areas at 75% and finance, law and health at 25%. It requires at least eight test series from two sources and three areas. The 0 to 100 scale refers to a fixed reference group of 29 language models whose average sits at 50.
Published model-specific parameters can differ. This is not a controlled experiment; comparisons with such differences are marked provisional.
Unmeasured series are not entered as zero. Where a model has no own results in a core area, its overall strength stands in, and the note “estimated without own results in …” names the area. Fewer test series lead to a larger standard error.
The 95% interval follows from the standard error of 200 recalculations with randomly redrawn test families. It describes how precisely the recorded test series determine the value. It is no prediction for future tasks or for later recalculations with other participants.
The source count is not a count of independent experiments.
OECD handbook on composite indicatorsAll 28 test families and 12 of 29 raw measurements are listed here. The complete calculation for every model is available in the JSON export.
Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 50 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 2.68 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 0 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 2.08 % · Coverage of this test family: 100 %
Published configurations differ within this cohort.
Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 50 %
Family weight in the fixed test panel (informative): 0 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.08 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.78 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.78 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.78 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 3.57 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 0 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.68 % · Coverage of this test family: 100 %
Family weight in the fixed test panel (informative): 2.68 % · Coverage of this test family: 100 %
Each evaluator measures different capabilities under its own conditions. Test series, composite indices and their components stay separate. The detailed results below retain their scales, configurations and source dates.
| Evaluator | Test series | Composite indices | Recorded active measurements |
|---|---|---|---|
| Artificial Analysis | 3 | 0 | 10 |
| Vals AI | 15 | 0 | 69 |
| METR | 1 | 0 | 2 |
| Princeton HAL | 6 | 0 | 9 |
| Scale AI Evals | 7 | 0 | 10 |
Coverage is not a quality ranking. Overlapping tests and multiple settings do not count as independent experiments. Vendor claims, historical and disputed results are excluded from this overview and remain labelled in the detailed evidence.
6 measurements from these sources still lack a verified test-family identity and do not increase the test-series count.
Page 1 of 11
Overall
225 code-editing tasks from Exercism across C++, Go, Java, JavaScript, Python, and Rust. Aider, the edit format, reasoning budget, and two attempts are part of the result.
higher is betterAs of September 2, 2026Scale: 0% - 100%
Accuracy
65 scientific coding problems with 338 subproblems across 16 research fields. Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
65 scientific coding problems with 338 subproblems across 16 research fields. Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
65 scientific coding problems with 338 subproblems across 16 research fields. Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
102 expert-validated tasks for data-driven scientific work. The scaffold, tools, and code execution are part of the result. HAL reports both 42 and 44 underlying publications in different places.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Accuracy
A random, fully human-validated subset of 50 GitHub issues from 12 Python repositories. The small subset is cheaper but statistically less stable than the full 500 tasks. SWE-Agent and HAL Generalist remain separate.
Note (o3): HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
307 algorithmic programming tasks from Bronze through Platinum with exhaustive tests. The benchmark measures competitive programming, not repository navigation, maintainability, or product experience.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Accuracy
33 automatically verifiable HAL tasks drawn from 214 realistic, time-consuming web research tasks. The published HAL subset is smaller than the full dataset. Reported costs do not account for caching benefits.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
300 verified tasks across 136 dynamic live websites. SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Accuracy
300 verified tasks across 136 dynamic live websites. SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Human task duration at which a model-agent system reaches a 50% or 80% predicted success probability. Minutes refer to human task duration, not model runtime. METR considers estimates above 16 hours unreliable.
| Sub-task | o3 (This model) | Gemini 3.1 Pro Preview | GPT-5.3-Codex | GPT-5 |
|---|---|---|---|---|
| 50% time horizon | 119.73 | 384.15 (best value in this row) | 349.53 | 203.01 |
| 80% time horizon | 29.98 | 89.8 (best value in this row) | 54.74 | 38.31 |
Unit: minuteshigher is betterAs of August 26, 2026Source: METRbars scaled within each row, not across rows
Page 1 of 11
One SVG prompt and fixed output settings. Captures are shown only with 0 reported reasoning tokens or documented lack of reasoning controls. Identical test conditions do not make the models' internal compute identical.
Unavailable in this round
A community classic for free-form SVG drawing.
o3

Reasoning (requested): high
Visually reviewed. This individual test does not establish overall model quality.
Generate an SVG of a pelican riding a bicycle
Token limit including reasoning: 8,192 tokens
Task origin (Simon Willison)22.2 / 100
| Specification | Value and source |
|---|---|
| Weights | ProprietarySource |
| Access | APISource |
| Context window | 200,000 tokensSource |
| Knowledge cutoff | June 1, 2024Source |
| Notes | Reasoning model, parameter count not disclosedSource |
| API model ID | o3Source |
| Maximum output | 100,000 tokensSource |
| Knowledge cutoff | 2024-06-01Source |
| Input | Text, ImageSource |
| Output | TextSource |
| API interfaces | Responses API, Chat Completions APISource |
| Reasoning mode | low, medium, high, xhighSource |
| Structured output | YesSource |
| Release | 2025-04-16Source |
OpenAI presents o3, released April 16, 2025, as the successor generation to o1, its prior full-size reasoning model (OpenAI skipped the name "o2" for trademark reasons).
OpenAI o3 and o4-mini system card / announcement (2026-09-19)