Language modelActive

o3

OpenAI

Released: April 16, 2025
Data date: September 8, 2026

Intelligence: Not ranked
Overall score
Speed: 178.4 tokens/s
Output throughput
Price: $2 · $8
Input · Output / 1M tokens
Input: Text, Image
Output: Text

No published category-wide ranking is available for this model.

Entries 1-12 of 124. Page 1 of 11.

Published benchmarks

Each chart compares models in the same documented test and highlights the current model.

Coding

7 measurements

Aider Polyglot

Overall

What this benchmark measures: 225 code-editing tasks from Exercism across C++, Go, Java, JavaScript, Python, and Rust.

Context: Aider, the edit format, reasoning budget, and two attempts are part of the result.

Higher is better4 modelso3: Rank 3 of 4Data date: September 2, 2026

1.o3-pro84.9%

2.Gemini 2.5 Pro83.1%

3.o381.3%

4.DeepSeek-R171.4%

Scale: 0% - 100%

Source: Aider

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: aider-polyglot-pass2-diff
Metric: percent correct after two attempts
Scale: 0-100
Participants: 4
Methodology and context: 225 Exercism tasks across six languages using Aider and a documented edit format.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Aider. Dataset type: public. Leaderboard date: 2025-06-25. Attempts: pass 2. Reasoning effort: high. Latency in seconds: 197.3. Cost per task in USD: 0.09433733333333331. Source ID: aider. Evidence capture ID: curated-aider-2026-09-02-2c407c1b0871. Source payload hash: 2c407c1b08711911ae3339b8d2b4a107fea7ea85865b411dda839aa972c4195a. Parser version: curated-v1. Imported at: 2026-09-04

Source: Aider

SciCode

Accuracy

What this benchmark measures: 65 scientific coding problems with 338 subproblems across 16 research fields.

Context: Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.

Higher is better2 modelso3: Rank 1 of 2Data date: August 5, 2026

1.o33.08%

2.GPT-4.11.54%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-scicode-hal-generalist-overall
Metric: accuracy
Scale: 0-100
Participants: 2
Methodology and context: HAL Generalist on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Generalist. Agent: HAL Generalist. Reasoning effort: medium. Cost per task in USD: 1.0304615384615383. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-0dae933d0c9e. Source payload hash: 0dae933d0c9e77627c932b50291c4be7cd13136af178d02a9b2fad27a4c037e8. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

SciCode

Accuracy

What this benchmark measures: 65 scientific coding problems with 338 subproblems across 16 research fields.

Context: Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.

Higher is better4 modelso3: Rank 1 of 4Data date: August 5, 2026

1.o39.23%

2.GPT-56.15%

3.Claude Sonnet 4.53.08%

4.GPT-4.11.54%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-scicode-scicode-tool-calling-overall
Metric: accuracy
Scale: 0-100
Participants: 4
Methodology and context: SciCode Tool Calling on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SciCode Tool Calling. Agent: SciCode Tool Calling. Reasoning effort: medium. Cost per task in USD: 1.7093846153846153. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-3b73275854bb. Source payload hash: 3b73275854bb2de797dd9ed67237c8ad83c065e9283768a925ef571bd04c9156. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

SciCode

Accuracy

What this benchmark measures: 65 scientific coding problems with 338 subproblems across 16 research fields.

Context: Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.

Higher is better3 modelso3: Rank 2 of 3Data date: August 5, 2026

1.GPT-4.16.15%

2.o34.62%

3.DeepSeek-V33.08%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-scicode-scicode-zero-shot-overall
Metric: accuracy
Scale: 0-100
Participants: 3
Methodology and context: SciCode Zero Shot on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SciCode Zero Shot. Agent: SciCode Zero Shot. Reasoning effort: medium. Cost per task in USD: 0.09276923076923076. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-df59398e4579. Source payload hash: df59398e457968e8e4222c4fd0fae3ecb1a86262fd0a3b12ddce6534ca487f62. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

ScienceAgentBench

Accuracy

What this benchmark measures: 102 expert-validated tasks for data-driven scientific work.

Context: The scaffold, tools, and code execution are part of the result. HAL reports both 42 and 44 underlying publications in different places.

Higher is better5 modelso3: Rank 1 of 5Data date: August 6, 2026

1.o333.33%

2.GPT-530.39%

3.GPT-4.124.51%

4.DeepSeek-R123.53%

5.DeepSeek-V315.69%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-scienceagentbench-sab-self-debug-overall
Metric: accuracy
Scale: 0-100
Participants: 5
Methodology and context: SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SAB Self-Debug. Agent: SAB Self-Debug. Reasoning effort: medium. Cost per task in USD: 0.1146078431372549. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-9ade5dc68557. Source payload hash: 9ade5dc685576c1832cf9d48efadbc1874421ff2f41acf31b26b010ad5d90e6c. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

SWE-bench Verified Mini

Accuracy

What this benchmark measures: A random, fully human-validated subset of 50 GitHub issues from 12 Python repositories.

Context: The small subset is cheaper but statistically less stable than the full 500 tasks. SWE-Agent and HAL Generalist remain separate.

Note (o3): HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

Higher is better4 modelso3: Rank 1 of 4Data date: August 5, 2026

1.GPT-546%

Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

1.o346%

3.GPT-4.144%

Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

4.DeepSeek-V324%

Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-swebench-verified-mini-swe-agent-overall
Metric: accuracy
Scale: 0-100
Participants: 4
Methodology and context: SWE-Agent on the same random 50-task subset. Total cost is divided by 50.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SWE-Agent. Agent: SWE-Agent. Reasoning effort: medium. Cost per task in USD: 9.6686. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-dfc1141bc4df. Source payload hash: dfc1141bc4df34f7ad53b05ff9088f1baa14b509e1ab3d6f61197895235f081c. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

USACO

Accuracy

What this benchmark measures: 307 algorithmic programming tasks from Bronze through Platinum with exhaustive tests.

Context: The benchmark measures competitive programming, not repository navigation, maintainability, or product experience.

Higher is better5 modelso3: Rank 2 of 5Data date: August 6, 2026

1.GPT-569.06%

2.o346.25%

3.GPT-4.144.95%

4.DeepSeek-V339.09%

5.DeepSeek-R138.11%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-usaco-usaco-episodic-semantic-overall
Metric: accuracy
Scale: 0-100
Participants: 5
Methodology and context: USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic. Reasoning effort: medium. Cost per task in USD: 0.18664495114006513. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-a6c14f678280. Source payload hash: a6c14f67828082f2e362b304d2937f0008fbe7325b9f9537343e362916bda5c2. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

Agent tasks

5 measurements

AssistantBench

Accuracy

What this benchmark measures: 33 automatically verifiable HAL tasks drawn from 214 realistic, time-consuming web research tasks.

Context: The published HAL subset is smaller than the full dataset. Reported costs do not account for caching benefits.

Higher is better4 modelso3: Rank 1 of 4Data date: August 5, 2026

1.o338.81%

2.GPT-535.23%

3.GPT-4.117.39%

4.DeepSeek-V32.03%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-assistantbench-browser-use-overall
Metric: accuracy
Scale: 0-100
Participants: 4
Methodology and context: One shared Browser-Use scaffold solves 33 verified HAL tasks. Total cost is divided by 33.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Reasoning effort: medium. Cost per task in USD: 0.4590909090909091. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-841883e41588. Source payload hash: 841883e4158871aba9a9d77489fe6a278e31e51c19da95a103f354da8eef2ce1. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

Online Mind2Web

Accuracy

What this benchmark measures: 300 verified tasks across 136 dynamic live websites.

Context: SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.

Higher is better5 modelso3: Rank 4 of 5Data date: August 5, 2026

1.GPT-4.136.33%

2.DeepSeek-V332.33%

3.GPT-532%

4.o329%

5.DeepSeek-R125.33%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-online-mind2web-browser-use-overall
Metric: accuracy
Scale: 0-100
Participants: 5
Methodology and context: The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Reasoning effort: medium. Cost per task in USD: 1.2386333333333333. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-77ccfede6f14. Source payload hash: 77ccfede6f14eea25d1a9500c263a7724e8796b6e0eadad09ca22ede439c8eee. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

Online Mind2Web

Accuracy

What this benchmark measures: 300 verified tasks across 136 dynamic live websites.

Context: SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.

Higher is better3 modelso3: Rank 2 of 3Data date: August 5, 2026

1.GPT-542.33%

2.o339%

3.GPT-4.130.33%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-online-mind2web-seeact-overall
Metric: accuracy
Scale: 0-100
Participants: 3
Methodology and context: The SeeAct scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SeeAct. Agent: SeeAct. Reasoning effort: medium. Cost per task in USD: 0.8624666666666667. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-a37c8133bcb5. Source payload hash: a37c8133bcb5512d66e07429ceef993f88084374775f340990b09ed7555b4e1b. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

METR Time Horizon

50% time horizon

What this benchmark measures: Human task duration at which a model-agent system reaches a 50% or 80% predicted success probability.

Context: Minutes refer to human task duration, not model runtime. METR considers estimates above 16 hours unreliable.

Higher is better4 modelso3: Rank 4 of 4Data date: August 26, 2026

1.Gemini 3.1 Pro Preview384.15 minutes

2.GPT-5.3-Codex349.53 minutes

3.GPT-5203.01 minutes

4.o3119.73 minutes

Scale: 0 minutes - 384.15 minutes

Source: METR

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: metr-th1-1-p50
Metric: 50% time horizon
Scale: No upper bound is documented. The chart spans the recorded values and zero.
Participants: 4
Methodology and context: Triframe agent with six independent runs per task. Time is estimated human completion duration.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Model snapshot: o3-2025-04-16. Evaluator: METR. Dataset type: public. Leaderboard date: 2026-05-08. Harness: Triframe. Agent: Triframe. Attempts: 6 independent runs per task. Score display: 2 h (95% confidence interval 1.2 h to 3.2 h). Source ID: metr. Evidence capture ID: curated-metr-2026-08-26-ed07fab486b1. Source payload hash: ed07fab486b18c5739fa21d045fe99cbdcf441b72457a5e6ebf4c0dbd8cec361. Parser version: curated-v1. Imported at: 2026-09-04

Source: METR

METR Time Horizon

80% time horizon

What this benchmark measures: Human task duration at which a model-agent system reaches a 50% or 80% predicted success probability.

Context: Minutes refer to human task duration, not model runtime. METR considers estimates above 16 hours unreliable.

Higher is better4 modelso3: Rank 4 of 4Data date: August 26, 2026

1.Gemini 3.1 Pro Preview89.8 minutes

2.GPT-5.3-Codex54.74 minutes

3.GPT-538.31 minutes

4.o329.98 minutes

Scale: 0 minutes - 89.8 minutes

Source: METR

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: metr-th1-1-p80
Metric: 80% time horizon
Scale: No upper bound is documented. The chart spans the recorded values and zero.
Participants: 4
Methodology and context: Triframe agent with six independent runs per task. Time is estimated human completion duration.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Model snapshot: o3-2025-04-16. Evaluator: METR. Dataset type: public. Leaderboard date: 2026-05-08. Harness: Triframe. Agent: Triframe. Attempts: 6 independent runs per task. Score display: 30 min (95% confidence interval 15.1 min to 57.8 min). Source ID: metr. Evidence capture ID: curated-metr-2026-08-26-a746ff39dd06. Source payload hash: a746ff39dd06992cab61063345954aed9bfa44921409b3070db3620aed664011. Parser version: curated-v1. Imported at: 2026-09-04

Source: METR

Recorded measurement details

Every observation retains its source value and published test conditions.

Aider Polyglot: Overall (Show measurement, test conditions, and source)

Source value: 81.3
Score: 81.3
Metric: percent correct after two attempts
Unit: %
Benchmark version: 225-case polyglot suite
Category: coding
Task: overall
Task label: Overall
Direction: Higher is better
Comparison cohort: aider-polyglot-pass2-diff
Source type: official-leaderboard
Evaluator: Aider
Status: active
Retrieved at: 2026-09-02
Methodology: 225 Exercism tasks across six languages using Aider and a documented edit format.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Aider. Dataset type: public. Leaderboard date: 2025-06-25. Attempts: pass 2. Reasoning effort: high. Latency in seconds: 197.3. Cost per task in USD: 0.09433733333333331. Source ID: aider. Evidence capture ID: curated-aider-2026-09-02-2c407c1b0871. Source payload hash: 2c407c1b08711911ae3339b8d2b4a107fea7ea85865b411dda839aa972c4195a. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: 225 Exercism tasks across six languages using Aider and a documented edit format.

Source Aider

AssistantBench: Accuracy (Show measurement, test conditions, and source)

Source value: 38.81
Score: 38.81
Metric: accuracy
Unit: %
Benchmark version: HAL 33-task public set
Category: agentic
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-assistantbench-browser-use-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: One shared Browser-Use scaffold solves 33 verified HAL tasks. Total cost is divided by 33.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Reasoning effort: medium. Cost per task in USD: 0.4590909090909091. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-841883e41588. Source payload hash: 841883e4158871aba9a9d77489fe6a278e31e51c19da95a103f354da8eef2ce1. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: One shared Browser-Use scaffold solves 33 verified HAL tasks. Total cost is divided by 33.

Source Princeton HAL

Online Mind2Web: Accuracy (Show measurement, test conditions, and source)

Source value: 29
Score: 29
Metric: accuracy
Unit: %
Benchmark version: 300-task live set
Category: agentic
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-online-mind2web-browser-use-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Reasoning effort: medium. Cost per task in USD: 1.2386333333333333. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-77ccfede6f14. Source payload hash: 77ccfede6f14eea25d1a9500c263a7724e8796b6e0eadad09ca22ede439c8eee. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.

Source Princeton HAL

Online Mind2Web: Accuracy (Show measurement, test conditions, and source)

Source value: 39
Score: 39
Metric: accuracy
Unit: %
Benchmark version: 300-task live set
Category: agentic
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-online-mind2web-seeact-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: The SeeAct scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SeeAct. Agent: SeeAct. Reasoning effort: medium. Cost per task in USD: 0.8624666666666667. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-a37c8133bcb5. Source payload hash: a37c8133bcb5512d66e07429ceef993f88084374775f340990b09ed7555b4e1b. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: The SeeAct scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.

Source Princeton HAL

SciCode: Accuracy (Show measurement, test conditions, and source)

Source value: 3.08
Score: 3.08
Metric: accuracy
Unit: %
Benchmark version: 65-problem main set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-scicode-hal-generalist-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: HAL Generalist on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Generalist. Agent: HAL Generalist. Reasoning effort: medium. Cost per task in USD: 1.0304615384615383. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-0dae933d0c9e. Source payload hash: 0dae933d0c9e77627c932b50291c4be7cd13136af178d02a9b2fad27a4c037e8. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: HAL Generalist on 65 main scientific problems. Total cost is divided by 65.

Source Princeton HAL

SciCode: Accuracy (Show measurement, test conditions, and source)

Source value: 9.23
Score: 9.23
Metric: accuracy
Unit: %
Benchmark version: 65-problem main set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-scicode-scicode-tool-calling-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: SciCode Tool Calling on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SciCode Tool Calling. Agent: SciCode Tool Calling. Reasoning effort: medium. Cost per task in USD: 1.7093846153846153. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-3b73275854bb. Source payload hash: 3b73275854bb2de797dd9ed67237c8ad83c065e9283768a925ef571bd04c9156. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: SciCode Tool Calling on 65 main scientific problems. Total cost is divided by 65.

Source Princeton HAL

SciCode: Accuracy (Show measurement, test conditions, and source)

Source value: 4.62
Score: 4.62
Metric: accuracy
Unit: %
Benchmark version: 65-problem main set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-scicode-scicode-zero-shot-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: SciCode Zero Shot on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SciCode Zero Shot. Agent: SciCode Zero Shot. Reasoning effort: medium. Cost per task in USD: 0.09276923076923076. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-df59398e4579. Source payload hash: df59398e457968e8e4222c4fd0fae3ecb1a86262fd0a3b12ddce6534ca487f62. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: SciCode Zero Shot on 65 main scientific problems. Total cost is divided by 65.

Source Princeton HAL

ScienceAgentBench: Accuracy (Show measurement, test conditions, and source)

Source value: 33.33
Score: 33.33
Metric: accuracy
Unit: %
Benchmark version: verified 102-task set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-scienceagentbench-sab-self-debug-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-06
Methodology: SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SAB Self-Debug. Agent: SAB Self-Debug. Reasoning effort: medium. Cost per task in USD: 0.1146078431372549. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-9ade5dc68557. Source payload hash: 9ade5dc685576c1832cf9d48efadbc1874421ff2f41acf31b26b010ad5d90e6c. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.

Source Princeton HAL

SWE-bench Verified Mini: Accuracy (Show measurement, test conditions, and source)

Source value: 46
Score: 46
Metric: accuracy
Unit: %
Benchmark version: 50-task random verified subset
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-swebench-verified-mini-swe-agent-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: SWE-Agent on the same random 50-task subset. Total cost is divided by 50.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: SWE-Agent. Agent: SWE-Agent. Reasoning effort: medium. Cost per task in USD: 9.6686. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-dfc1141bc4df. Source payload hash: dfc1141bc4df34f7ad53b05ff9088f1baa14b509e1ab3d6f61197895235f081c. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: SWE-Agent on the same random 50-task subset. Total cost is divided by 50.
Context: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

Source Princeton HAL

USACO: Accuracy (Show measurement, test conditions, and source)

Source value: 46.25
Score: 46.25
Metric: accuracy
Unit: %
Benchmark version: HAL 307-problem set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-usaco-usaco-episodic-semantic-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-06
Methodology: USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic. Reasoning effort: medium. Cost per task in USD: 0.18664495114006513. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-a6c14f678280. Source payload hash: a6c14f67828082f2e362b304d2937f0008fbe7325b9f9537343e362916bda5c2. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.

Source Princeton HAL

METR Time Horizon: 50% time horizon (Show measurement, test conditions, and source)

Source value: 119.732634
Score: 119.732634
Metric: 50% time horizon
Unit: minutes
Benchmark version: TH 1.1
Category: agentic
Task: p50
Task label: 50% time horizon
Direction: Higher is better
Comparison cohort: metr-th1-1-p50
Source type: independent-evaluator
Evaluator: METR
Status: active
Retrieved at: 2026-08-26
Methodology: Triframe agent with six independent runs per task. Time is estimated human completion duration.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Model snapshot: o3-2025-04-16. Evaluator: METR. Dataset type: public. Leaderboard date: 2026-05-08. Harness: Triframe. Agent: Triframe. Attempts: 6 independent runs per task. Score display: 2 h (95% confidence interval 1.2 h to 3.2 h). Source ID: metr. Evidence capture ID: curated-metr-2026-08-26-ed07fab486b1. Source payload hash: ed07fab486b18c5739fa21d045fe99cbdcf441b72457a5e6ebf4c0dbd8cec361. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: Triframe agent with six independent runs per task. Time is estimated human completion duration.

Source METR

METR Time Horizon: 80% time horizon (Show measurement, test conditions, and source)

Source value: 29.981603
Score: 29.981603
Metric: 80% time horizon
Unit: minutes
Benchmark version: TH 1.1
Category: agentic
Task: p80
Task label: 80% time horizon
Direction: Higher is better
Comparison cohort: metr-th1-1-p80
Source type: independent-evaluator
Evaluator: METR
Status: active
Retrieved at: 2026-08-26
Methodology: Triframe agent with six independent runs per task. Time is estimated human completion duration.. Source model ID: curated/o3. Source model name: o3. Provider reported by source: OpenAI. Model snapshot: o3-2025-04-16. Evaluator: METR. Dataset type: public. Leaderboard date: 2026-05-08. Harness: Triframe. Agent: Triframe. Attempts: 6 independent runs per task. Score display: 30 min (95% confidence interval 15.1 min to 57.8 min). Source ID: metr. Evidence capture ID: curated-metr-2026-08-26-a746ff39dd06. Source payload hash: a746ff39dd06992cab61063345954aed9bfa44921409b3070db3620aed664011. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: Triframe agent with six independent runs per task. Time is estimated human completion duration.

Source METR

Our model tests

Community tests you can inspect

Compare how the models respond to the same prompt. Each test shows the first attempt, with no subsequent fixes to the generated code. These results do not contribute to an overall score.

Pelican on a bicycle

A community classic for free-form SVG drawing.

Not tested yet

Prompt and test conditions

Generate an SVG of a pelican riding a bicycle

Token limit including reasoning: 8,192 tokens

Task origin (Simon Willison)

Evidence

Sources and data date

Every statement links to its underlying documentation or leaderboard.

Speed: OpenAI. Output speed after the first streaming chunk, not total response time. Retrieved September 8, 2026.
Artificial Analysis Methodology
Price: Base API prices without caching or batch discounts. Higher context tiers and other rates are listed under Costs.
Pricing
Input: Documented modalities. Retrieved September 8, 2026.
Model documentation
Output: Documented modalities. Retrieved September 8, 2026.
Model documentation

Research date September 8, 2026. 4 source URLs checked. This documents the inspected sources, not an exhaustive inventory of every publication.

Open evidence questions and research notes (original wording)

Not found in the inspected sources

parameters-billions. OpenAI API and release pages do not publish total or active parameter counts.

Not found in the inspected sources

independent-benchmarks. This pass used the exact OpenAI model-column tables; no separate independent evaluator record was added.

Model metadata (retrieved September 8, 2026)Source
API pricing (retrieved September 8, 2026)Source
Aider (retrieved September 2, 2026)Source
Princeton HAL (retrieved August 5, 2026)Source
Princeton HAL (retrieved August 5, 2026)Source
Princeton HAL (retrieved August 5, 2026)Source
Princeton HAL (retrieved August 6, 2026)Source
Princeton HAL (retrieved August 5, 2026)Source
Princeton HAL (retrieved August 6, 2026)Source
METR (retrieved August 26, 2026)Source
Scale AI (retrieved August 26, 2026)Source
Scale AI (retrieved August 26, 2026)Source
Scale AI (retrieved August 26, 2026)Source
Scale AI (retrieved August 26, 2026)Source
Scale AI (retrieved August 26, 2026)Source
Scale AI (retrieved August 26, 2026)Source
Scale AI (retrieved August 26, 2026)Source
Kaggle (retrieved August 9, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved September 4, 2026)Source
Vals AI (retrieved September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
Vals AI (retrieved August 11, 2026; September 4, 2026)Source
OpenAI (retrieved September 8, 2026)Source
Artificial Analysis (retrieved September 8, 2026)Source
Speed measurement methodologySource

o3

Published benchmarks

Coding

Aider Polyglot

SciCode

SciCode

SciCode

ScienceAgentBench

SWE-bench Verified Mini

USACO

Agent tasks

AssistantBench

Online Mind2Web

Online Mind2Web

METR Time Horizon

METR Time Horizon

Recorded measurement details

Community tests you can inspect

Pelican on a bicycle

Specifications and access

Additional source observations

Published prices

100 API requests

Compare this model

Sources and data date