o1
Unavailable in this round
Language modelDeprecated
o1
OpenAI
- Released
- December 2024
- Data date
- September 8, 2026
- Successor
- o3
- Benchmark-Meta-Score
- Standard error ± 0.1
- Price
- Historical rates under Costs
- Input
- Text, Image
- Output
- Text
With o1, OpenAI introduced a model that uses additional thinking time before answering. Reinforcement learning trained it to check different approaches and recognize mistakes while reasoning. That distinguished the o-series from GPT models that answered directly.
o3 later extended the reasoning line with agentic tool use in ChatGPT, including combinations of web search and Python.
- Position
- Rank 67 of 91
- Index score
- 0.1 / 100
Methodology and source
The LLM comparison hub's selected-panel index averages rank percentiles from 65 documented comparison cohorts. Benchmark meta-scores are shown in the benchmark section.
Benchmark meta-analysis
Overall and area scores
These Gradually indexes place selected published measurements on a 0 to 100 scale. They are not accuracy percentages and do not guarantee results in your application.
Retrieval dates of the sources used 2026-08-05 - 2026-10-19
Overall index
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.1
- 95% interval
- 0.0 to 1.2
- Test series
- 12
- Test families
- 12 / 65
- Sources
- 3
- Share of the fixed test panel (informative)
- 21 %
Coding
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.0
- Test series
- 1
- Test families
- 1 / 15
- Sources
- 1
- Share of the fixed test panel (informative)
- 7 %
Reasoning
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.3
- Test series
- 1
- Test families
- 1 / 7
- Sources
- 1
- Share of the fixed test panel (informative)
- 7 %
Mathematics
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.1
- Test series
- 2
- Test families
- 2 / 5
- Sources
- 1
- Share of the fixed test panel (informative)
- 40 %
Knowledge
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.2
- 95% interval
- 0.0 to 1.3
- Test series
- 1
- Test families
- 1 / 3
- Sources
- 1
- Share of the fixed test panel (informative)
- 33 %
Agents
Index below 1 / 100
provisional
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 1.2
- Test series
- 2
- Test families
- 2 / 12
- Sources
- 1
- Share of the fixed test panel (informative)
- 17 %
General
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.7
- 95% interval
- 0.1 to 3.8
- Test series
- 1
- Test families
- 1 / 5
- Sources
- 1
- Share of the fixed test panel (informative)
- 20 %
Finance
Index 1.8 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 1.8
- 95% interval
- 0.2 to 9.5
- Test series
- 1
- Test families
- 1 / 7
- Sources
- 1
- Share of the fixed test panel (informative)
- 14 %
Legal
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.2
- 95% interval
- 0.0 to 1.2
- Test series
- 1
- Test families
- 1 / 4
- Sources
- 1
- Share of the fixed test panel (informative)
- 25 %
Health
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.7
- 95% interval
- 0.2 to 3.9
- Test series
- 1
- Test families
- 1 / 3
- Sources
- 1
- Share of the fixed test panel (informative)
- 33 %
Multimodal
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.1
- 95% interval
- 0.0 to 0.7
- Test series
- 1
- Test families
- 1 / 4
- Sources
- 1
- Share of the fixed test panel (informative)
- 13 %
Documented method · Included test families and measurements
Method version: benchmark-meta-score-v3
The index includes active measurements from reviewed primary sources when at least five models are compared in the same documented comparison slice across dataset, harness, agent, tools, and trial count. Each benchmark series and evaluator contributes one shared setup slice from the newest capture. When one source and series include several explicitly documented reasoning tiers, the highest documented tier (none through max) is selected, never the best result value. The explicit source label “Non-reasoning” is treated as none. Unknown or ambiguous tiers exclude the model from that series. Within each series, percentages and rates are logit-transformed, other units stay unchanged, and all values are then standardized. A one-factor model then estimates one shared strength per model and, per series, how sharply it separates the models. Area scores move away from that strength only as far as the model's own results in the area support. The overall score weights the core areas at 75% and finance, law and health at 25%. It requires at least eight test series from two sources and three areas. The 0 to 100 scale refers to a fixed reference group of 29 language models whose average sits at 50.
Published model-specific parameters can differ. This is not a controlled experiment; comparisons with such differences are marked provisional.
Unmeasured series are not entered as zero. Where a model has no own results in a core area, its overall strength stands in, and the note “estimated without own results in …” names the area. Fewer test series lead to a larger standard error.
The 95% interval follows from the standard error of 200 recalculations with randomly redrawn test families. It describes how precisely the recorded test series determine the value. It is no prediction for future tasks or for later recalculations with other participants.
The source count is not a count of independent experiments.
OECD handbook on composite indicatorsAll 12 test families and 12 of 12 raw measurements are listed here. The complete calculation for every model is available in the JSON export.
Test family: IFBench
General · Mean rank percentile of the family: 68.1 / 100Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-ifbench
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.14 %
- Raw value:
- 70.3401360544218 %
- Metric:
- IFBench prompt-level loose accuracy (pass@1)
- Version:
- AA test set: 294 questions
- Participating models:
- 70
- Rank percentile within the cohort:
- 68.1 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-ifbench:AA test set: 294 questions:standalone:reasoning:tokens=0:suffix=unspecified
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-ifbench","AA test set: 294 questions","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Tool policy:
- Single-turn answer evaluated with the official IFBench source code
- Attempts:
- pass@1, 5 repeats per question
- Setup and model parameters
- Exact AA configuration: o1 (release o1, row o1, UUID 5ad2f60f-ee05-49fd-85a0-cef69aa7cb7b, reasoning=true, source reasoning-token configuration budget=0, suffix=not specified). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled
Exact AA configuration: o1 (release o1, row o1, UUID 5ad2f60f-ee05-49fd-85a0-cef69aa7cb7b, reasoning=true, source reasoning-token configuration budget=0, suffix=not specified). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Source model ID: artificial-analysis/5ad2f60f-ee05-49fd-85a0-cef69aa7cb7b/o1/o1. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1. Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled. Score display: 70.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: GPQA Diamond
Reasoning · Mean rank percentile of the family: 22.4 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 50 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-gpqa
- Series weight within the test family:
- 50 %
- Nominal contribution to the overall panel:
- 0.77 %
- Raw value:
- 73.232 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 77
- Rank percentile within the cohort:
- 22.4 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-gpqa:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-gpqa:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 100000
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Reasoning effort: high. Maximum output tokens: 100000. Latency in seconds: 40.176. Standard error: 2.231. Cost per task in USD: 0.252632. Source ID: vals-ai. Evidence capture ID: vals-gpqa-2026-10-15-951eb5224097. Source payload hash: 951eb5224097733a9d09d06c14908ae2980e2e640e539f2c284877c5092d7077. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: GAIA Reliability
Agents · Mean rank percentile of the family: 12.5 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Princeton HAL
- Measurement:
- hal-gaia-reliability
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 34.4 %
- Metric:
- exact-match accuracy
- Version:
- final leak-cleaned public validation set
- Participating models:
- 9
- Rank percentile within the cohort:
- 12.5 %
- Retrieved:
- 2026-08-06
- Comparison cohort:
- hal-gaia-reliability-hal-reliability-overall
- Documented comparison setup
- Publication cohort:
- hal-gaia-reliability-hal-reliability-overall
- Dataset type:
- public
- Harness:
- HAL Reliability
- Agent:
- HAL Reliability
- Setup and model parameters
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Harness: HAL Reliability. Agent: HAL Reliability
165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-0b27954353a7. Source payload hash: 0b27954353a7c19a0cf4d633cf03db86774a4f1731e1998e0c8fb557f2d2a1e2. Parser version: curated-v1. Imported at: 2026-10-15
- Princeton HAL
Test family: τ-bench Airline Clean
Agents · Mean rank percentile of the family: 25.0 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Princeton HAL
- Measurement:
- hal-tau-airline-clean
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 66.2 %
- Metric:
- task success rate
- Version:
- 26-task cleaned subset
- Participating models:
- 9
- Rank percentile within the cohort:
- 25.0 %
- Retrieved:
- 2026-08-05
- Comparison cohort:
- hal-tau-airline-clean-hal-reliability-overall
- Documented comparison setup
- Publication cohort:
- hal-tau-airline-clean-hal-reliability-overall
- Dataset type:
- public
- Harness:
- HAL Reliability
- Agent:
- HAL Reliability
- Setup and model parameters
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Harness: HAL Reliability. Agent: HAL Reliability
26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-5c14c4dac3c3. Source payload hash: 5c14c4dac3c3d877041cbb52e3bbb6b35af52ee20551afb0f3638c0c305ace9a. Parser version: curated-v1. Imported at: 2026-10-15
- Princeton HAL
Test family: MMMU-Pro
Multimodal · Mean rank percentile of the family: 21.3 / 100Family weight in the fixed test panel (informative): 2.68 % · Coverage of this test family: 50 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-mmmu
- Series weight within the test family:
- 50 %
- Nominal contribution to the overall panel:
- 1.34 %
- Raw value:
- 77.412 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 48
- Rank percentile within the cohort:
- 21.3 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-mmmu:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-mmmu:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 16384
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Reasoning effort: high. Maximum output tokens: 16384. Latency in seconds: 26.415. Standard error: 1.005. Cost per task in USD: 0.14586. Source ID: vals-ai. Evidence capture ID: vals-mmmu-2026-10-15-29395186eddd. Source payload hash: 29395186eddd2c70601228c584a14b3c1d8f9dba4b68256561b46b0dd03c635a. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: LiveCodeBench v6
Coding · Mean rank percentile of the family: 10.4 / 100Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-lcb
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.71 %
- Raw value:
- 50.264 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 78
- Rank percentile within the cohort:
- 10.4 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-lcb:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-lcb:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 16384
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Reasoning effort: high. Maximum output tokens: 16384. Latency in seconds: 92.082. Standard error: 1.393. Cost per task in USD: 0.328067. Source ID: vals-ai. Evidence capture ID: vals-lcb-2026-10-15-1a7c7dabc9d9. Source payload hash: 1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: LegalBench
Legal · Mean rank percentile of the family: 35.9 / 100Family weight in the fixed test panel (informative): 2.08 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-legal-bench
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.08 %
- Raw value:
- 80.393 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 79
- Rank percentile within the cohort:
- 35.9 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-legal-bench:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-legal-bench:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 2048
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-10-13. Reasoning effort: high. Maximum output tokens: 2048. Latency in seconds: 8.338. Standard error: 0.578. Source ID: vals-ai. Evidence capture ID: vals-legal_bench-2026-10-15-c8486ba756ba. Source payload hash: c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MATH 500
Mathematics · Mean rank percentile of the family: 50.0 / 100Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-math500
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.14 %
- Raw value:
- 90.4 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 23
- Rank percentile within the cohort:
- 50.0 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-math500:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-math500:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 32768
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-01-09. Reasoning effort: high. Maximum output tokens: 32768. Latency in seconds: 23.537. Standard error: 2.591. Cost per task in USD: 0.095269. Source ID: vals-ai. Evidence capture ID: vals-math500-2026-10-15-435ec4da1112. Source payload hash: 435ec4da1112362c0a6bb4afeab53bf7b0faf2000e1ddbe1a878735dca1fe35b. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MedQA
Health · Mean rank percentile of the family: 100.0 / 100Family weight in the fixed test panel (informative): 2.78 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-medqa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.78 %
- Raw value:
- 96.517 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 40
- Rank percentile within the cohort:
- 100.0 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-medqa:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-medqa:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 4096
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-04-16. Reasoning effort: high. Maximum output tokens: 4096. Latency in seconds: 11.152. Standard error: 0.33. Cost per task in USD: 0.05282. Source ID: vals-ai. Evidence capture ID: vals-medqa-2026-10-15-d67905978d93. Source payload hash: d67905978d932f64698ebe9e230d9519e5bae1c9d7176dda8379dc3874b6f90f. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MGSM
Mathematics · Mean rank percentile of the family: 43.1 / 100Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-mgsm
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.14 %
- Raw value:
- 89.309 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 30
- Rank percentile within the cohort:
- 43.1 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-mgsm:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-mgsm:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 16384
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-01-09. Reasoning effort: high. Maximum output tokens: 16384. Latency in seconds: 11.207. Standard error: 1.779. Cost per task in USD: 0.057248. Source ID: vals-ai. Evidence capture ID: vals-mgsm-2026-10-15-1b87f953e52c. Source payload hash: 1b87f953e52c00bf4bdd90c3add26165f644eeb7a74a8382d2a0324ff0cc1040. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MMLU-Pro
Knowledge · Mean rank percentile of the family: 40.3 / 100Family weight in the fixed test panel (informative): 3.57 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-mmlu-pro
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 3.57 %
- Raw value:
- 83.488 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 78
- Rank percentile within the cohort:
- 40.3 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-mmlu-pro:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-mmlu-pro:1:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 16384
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Reasoning effort: high. Maximum output tokens: 16384. Latency in seconds: 26.867. Standard error: 0.364. Cost per task in USD: 0.116223. Source ID: vals-ai. Evidence capture ID: vals-mmlu_pro-2026-10-15-dab025507f46. Source payload hash: dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: TaxEval v2
Finance · Mean rank percentile of the family: 64.5 / 100Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-tax-eval-v2
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.19 %
- Raw value:
- 74.284 %
- Metric:
- accuracy
- Version:
- 2
- Participating models:
- 77
- Rank percentile within the cohort:
- 64.5 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-tax-eval-v2:2:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-tax-eval-v2:2:overall
- Dataset type:
- private
- Setup and model parameters
- Reasoning effort: high. Maximum output tokens: 16384
Source model ID: openai/o1-2024-12-17. Source model name: o1. Provider reported by source: OpenAI. Model snapshot: o1-2024-12-17. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-09-01. Reasoning effort: high. Maximum output tokens: 16384. Latency in seconds: 19.477. Standard error: 0.861. Cost per task in USD: 0.12173. Source ID: vals-ai. Evidence capture ID: vals-tax_eval_v2-2026-10-15-8b354a6ae848. Source payload hash: 8b354a6ae848f5bb25a2e8b061b5f576d5b9e0c11b941dc1e6bed078e6529cd6. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Benchmark sources at a glance
Each evaluator measures different capabilities under its own conditions. Test series, composite indices and their components stay separate. The detailed results below retain their scales, configurations and source dates.
| Evaluator | Test series | Composite indices | Recorded active measurements |
|---|---|---|---|
| Artificial Analysis | 2 | 0 | 3 |
| Vals AI | 10 | 0 | 55 |
| Princeton HAL | 2 | 0 | 12 |
Coverage is not a quality ranking. Overlapping tests and multiple settings do not count as independent experiments. Vendor claims, historical and disputed results are excluded from this overview and remain labelled in the detailed evidence.
Page 1 of 9
Published benchmarks
Agent tasks
GAIA Reliability
Reliability evaluation on 165 GAIA tasks with repetitions and perturbations. Accuracy, reliability, consistency, predictability, robustness, and safety are separate metrics.
| Sub-task | o1 (This model) | Claude Opus 4.5 | Claude Opus 4.7 | Gemini 3.1 Pro Preview | Gemini 3.5 Flash | GPT-4o mini | Gemini 2.5 Pro | Gemini 2.5 Flash | GPT-5.5 |
|---|---|---|---|---|---|---|---|---|---|
| Consistency | 0.68 | 0.81 (best value in this row) | 0.81 (best value in this row) | 0.76 | 0.75 | 0.73 | 0.72 | 0.7 | 0.61 |
| Predictability | 0.68 | 0.84 (best value in this row) | 0.73 | 0.78 | 0.8 | 0.58 | 0.71 | 0.64 | 0.8 |
| Reliability | 0.79 | 0.85 (best value in this row) | 0.84 | 0.82 | 0.84 | 0.76 | 0.78 | 0.74 | 0.79 |
| Robustness | 1 (best value in this row) | 0.91 | 0.99 | 0.94 | 0.96 | 0.97 | 0.91 | 0.87 | 0.95 |
| Safety | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) |
Unit: ratinghigher is betterAs of August 6, 2026Source: Princeton HALbars scaled within each row, not across rows
GAIA Reliability
Accuracy
Reliability evaluation on 165 GAIA tasks with repetitions and perturbations. Accuracy, reliability, consistency, predictability, robustness, and safety are separate metrics.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- hal-gaia-reliability-hal-reliability-overall
- Metric
- exact-match accuracy
- Scale
- 0-100
- Participants
- 9
- Methodology and context
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-0b27954353a7. Source payload hash: 0b27954353a7c19a0cf4d633cf03db86774a4f1731e1998e0c8fb557f2d2a1e2. Parser version: curated-v1. Imported at: 2026-10-15
τ-bench Airline Clean
26 cleaned customer-service tasks involving dialogue, policy compliance, and mutating backend tools. The cleaned subset removes 24 broken or ambiguous tasks. Accuracy and reliability dimensions measure different properties.
| Sub-task | o1 (This model) | Claude Opus 4.7 | Claude Opus 4.5 | GPT-5.5 | Gemini 3.1 Pro Preview | Gemini 3.5 Flash | Gemini 2.5 Flash | Gemini 2.5 Pro | GPT-4o mini |
|---|---|---|---|---|---|---|---|---|---|
| Consistency | 0.71 | 0.88 (best value in this row) | 0.84 | 0.84 | 0.81 | 0.78 | 0.75 | 0.74 | 0.74 |
| Predictability | 0.74 | 0.87 (best value in this row) | 0.85 | 0.84 | 0.83 | 0.82 | 0.6 | 0.78 | 0.35 |
| Reliability | 0.81 | 0.89 (best value in this row) | 0.88 | 0.89 (best value in this row) | 0.86 | 0.86 | 0.72 | 0.81 | 0.67 |
| Robustness | 0.98 (best value in this row) | 0.91 | 0.97 | 0.97 | 0.94 | 0.97 | 0.82 | 0.93 | 0.93 |
| Safety | 0.91 | 0.94 | 0.98 | 0.96 | 0.97 | 0.99 (best value in this row) | 0.9 | 0.92 | 0.85 |
Unit: ratinghigher is betterAs of August 5, 2026Source: Princeton HALbars scaled within each row, not across rows
τ-bench Airline Clean
Accuracy
26 cleaned customer-service tasks involving dialogue, policy compliance, and mutating backend tools. The cleaned subset removes 24 broken or ambiguous tasks. Accuracy and reliability dimensions measure different properties.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-overall
- Metric
- task success rate
- Scale
- 0-100
- Participants
- 9
- Methodology and context
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-5c14c4dac3c3. Source payload hash: 5c14c4dac3c3d877041cbb52e3bbb6b35af52ee20551afb0f3638c0c305ace9a. Parser version: curated-v1. Imported at: 2026-10-15
Recorded measurement details
GAIA Reliability: Consistency (Show measurement, test conditions, and source)
- Source value
- 0.68
- Score
- 0.68
- Metric
- aggregate consistency
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- consistency
- Task label
- Consistency
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-consistency
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-27db66c4b150. Source payload hash: 27db66c4b1505bf60fa380eecd40fee7eda73c877f0803d008198df8b45f3791. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Predictability (Show measurement, test conditions, and source)
- Source value
- 0.68
- Score
- 0.68
- Metric
- aggregate predictability
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- predictability
- Task label
- Predictability
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-predictability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-700f4d20dd9d. Source payload hash: 700f4d20dd9d307edd3b70ea63bea10521b9663f054fbd5396515c5e6d58a23e. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Reliability (Show measurement, test conditions, and source)
- Source value
- 0.79
- Score
- 0.79
- Metric
- aggregate reliability
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- reliability
- Task label
- Reliability
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-reliability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-c89e5f0e01ee. Source payload hash: c89e5f0e01ee0e1799681553dc22a1859d3956fc995c42b953c9e9ca454623de. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Robustness (Show measurement, test conditions, and source)
- Source value
- 1
- Score
- 1
- Metric
- aggregate robustness
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- robustness
- Task label
- Robustness
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-robustness
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-9be6ee834442. Source payload hash: 9be6ee83444284a66ed293c950e953648c71aa6c5f5a6948dfe582d11f5de1a7. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Safety (Show measurement, test conditions, and source)
- Source value
- 1
- Score
- 1
- Metric
- aggregate safety
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- safety
- Task label
- Safety
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-safety
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-7b08c2b52623. Source payload hash: 7b08c2b5262365cef8eb694f50ae89c6c0f11eb7984429f024285528558e9d60. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Accuracy (Show measurement, test conditions, and source)
- Source value
- 34.4
- Score
- 34.4
- Metric
- exact-match accuracy
- Unit
- %
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- overall
- Task label
- Accuracy
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-overall
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-0b27954353a7. Source payload hash: 0b27954353a7c19a0cf4d633cf03db86774a4f1731e1998e0c8fb557f2d2a1e2. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
τ-bench Airline Clean: Consistency (Show measurement, test conditions, and source)
- Source value
- 0.71
- Score
- 0.71
- Metric
- aggregate consistency
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- consistency
- Task label
- Consistency
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-consistency
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-6bfdff20ad21. Source payload hash: 6bfdff20ad21b3817ab0b539ad53e6a383f542222890c8da5d58512d9c1fb6de. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Predictability (Show measurement, test conditions, and source)
- Source value
- 0.74
- Score
- 0.74
- Metric
- aggregate predictability
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- predictability
- Task label
- Predictability
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-predictability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-b8057f91c3e8. Source payload hash: b8057f91c3e8f20eb208ea8d8f905d219b30748861becf6a8e887a5aaffb89c4. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Reliability (Show measurement, test conditions, and source)
- Source value
- 0.81
- Score
- 0.81
- Metric
- aggregate reliability
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- reliability
- Task label
- Reliability
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-reliability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-48cde879a7c5. Source payload hash: 48cde879a7c52057c3803e0d18835b0ea4e6df4b46597d59a6a52eb88436f488. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Robustness (Show measurement, test conditions, and source)
- Source value
- 0.98
- Score
- 0.98
- Metric
- aggregate robustness
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- robustness
- Task label
- Robustness
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-robustness
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-7b9ebb788fcf. Source payload hash: 7b9ebb788fcf11648139fae639ecce250a38592c83eefa9a7102bb675706babb. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Safety (Show measurement, test conditions, and source)
- Source value
- 0.91
- Score
- 0.91
- Metric
- aggregate safety
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- safety
- Task label
- Safety
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-safety
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-d485db5c9de6. Source payload hash: d485db5c9de6349be1692c7b6233d809eed3d2e75925a6fefa9a9e2191debf88. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Accuracy (Show measurement, test conditions, and source)
- Source value
- 66.2
- Score
- 66.2
- Metric
- task success rate
- Unit
- %
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- overall
- Task label
- Accuracy
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-overall
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/o1. Source model name: o1. Provider reported by source: OpenAI. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-5c14c4dac3c3. Source payload hash: 5c14c4dac3c3d877041cbb52e3bbb6b35af52ee20551afb0f3638c0c305ace9a. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
Page 1 of 9
Our tests
SVG short test
One SVG prompt and fixed output settings. Captures are shown only with 0 reported reasoning tokens or documented lack of reasoning controls. Identical test conditions do not make the models' internal compute identical.
Earlier tests
Pelican on a bicycle
A community classic for free-form SVG drawing.
o1
Reasoning (requested): high
Current attempt
- Run date
- Oct 2, 2026
- API provider
- openrouter
- Requested API model
- openai/o1
- API model reported in response
- openai/o1
- Reasoning setting
- high
- Pinned endpoint
- OpenAI | openai/o1-2024-12-17
- Temperature
- Provider default
- Token limit including reasoning
- 8,192 tokens
- Test protocol
- community-visual-tests-v1
- Technical checks
- Passed, not a quality rating
- Response time
- 15.4 s
Visually reviewed. This individual test does not establish overall model quality.
Earlier attempts (1)
- Oct 1, 2026Output withheld for safetyReasoning setting: high
Prompt and test conditions
Generate an SVG of a pelican riding a bicycle
Token limit including reasoning: 8,192 tokens
Task origin (Simon Willison)Popularity
24.9 / 100
Specifications and access
| Specification | Value and source |
|---|---|
| Weights | ProprietarySource not specified |
| Context window | 200,000 tokensSource |
| Knowledge cutoff | October 1, 2023Source |
| Notes | Reasoning model, parameter count not disclosedSource not specified |
| API model ID | o1Source |
| Maximum output | 100,000 tokensSource |
| Knowledge cutoff | 2023-10-01Source |
| Input | Text, ImageSource |
| Output | TextSource |
| Reasoning mode | highSource |
| API interfaces | Responses API, Chat Completions APISource |
| Tool calling | YesSource |
| Structured output | YesSource |
Predecessor and successor
Sources and data date
| Type | Evidence and data date |
|---|---|
| Input | Documented modalities. Retrieved September 8, 2026. |
| Output | Documented modalities. Retrieved September 8, 2026. |
| Research status | Research date September 8, 2026. 3 source URLs checked. This documents the inspected sources, not an exhaustive inventory of every publication. |
| Not found in the inspected sources | parameters-billions Original wording OpenAI API and release pages do not publish parameter counts. |
| Not found in the inspected sources | independent-benchmarks Original wording Only exact OpenAI appendix columns were used in this artifact. |
| Unresolved | release-date Original wording The o1 family launched in September 2024, while the stable API snapshot was released in December 2024; this record does not choose between those distinct dates. |
| Not found in the inspected sources | AA benchmark rows Original wording No exact-model row in the selected public JSON-LD datasets (some pages only expose N/A or unknown metric summaries). |
| Additional source | Context window data · Introducing OpenAI o1 · Editorial description reviewed October 4, 2026 |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Princeton HAL (retrieved August 6, 2026) |
| Additional source | Princeton HAL (retrieved August 5, 2026) |
| Additional source | LongBench (retrieved September 2, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | OpenAI (retrieved September 8, 2026) |
| Additional source | Learning to reason with LLMs · Editorial description reviewed October 4, 2026 |
| Additional source | OpenRouter: Popularity index (derived), CC BY 4.0 |
| Additional source | Vercel AI Gateway: Popularity index (derived), CC BY 4.0 |
| Additional source | Ollama: cumulative downloads for the popularity index |
| Additional source | Replicate: cumulative runs for the popularity index |
| Additional source | Google Trending Now: limited feed selection for Germany and the US |
| Additional source | Creative Commons Attribution 4.0 |
| Additional source | OpenAI o3 and o4-mini system card / announcement (2026-09-19) |