DeepSeek-R1
Unavailable in this round
Language modelActive
DeepSeek-R1
DeepSeek
- Released
- January 2025
- Data date
- September 8, 2026
- Successor
- DeepSeek-R1 0528
- Benchmark-Meta-Score
- Standard error ± 0
- Price
- Input · Output / 1M tokens
DeepSeek-R1 was the open reasoning model that established the R1 family. It is based on DeepSeek-V3-Base and combines cold-start data with multiple reinforcement-learning and supervised fine-tuning stages. The mixture-of-experts checkpoint has 671 billion total parameters, 37 billion active parameters, and a 128,000-token context window.
R1 remains separate from the general V3 chat mode. Its model card documents reasoning and local use but no native tool interface. The later DeepSeek-R1 0528 revision expands the context to 163,840 tokens but adds no general non-thinking mode.
- Position
- Rank 73 of 91
- Index score
- 0 / 100
Methodology and source
The LLM comparison hub's selected-panel index averages rank percentiles from 65 documented comparison cohorts. Benchmark meta-scores are shown in the benchmark section.
Benchmark meta-analysis
Overall and area scores
These Gradually indexes place selected published measurements on a 0 to 100 scale. They are not accuracy percentages and do not guarantee results in your application.
Retrieval dates of the sources used 2026-08-05 - 2026-10-15
Overall index
Index below 1 / 100
provisional
Estimated without own results in: General
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.2
- Test series
- 12
- Test families
- 12 / 65
- Sources
- 3
- Share of the fixed test panel (informative)
- 20 %
Coding
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.2
- Test series
- 3
- Test families
- 3 / 15
- Sources
- 2
- Share of the fixed test panel (informative)
- 20 %
Reasoning
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.2
- Test series
- 1
- Test families
- 1 / 7
- Sources
- 1
- Share of the fixed test panel (informative)
- 14 %
Mathematics
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.1
- 95% interval
- 0.0 to 0.9
- Test series
- 2
- Test families
- 2 / 5
- Sources
- 1
- Share of the fixed test panel (informative)
- 40 %
Knowledge
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.1
- 95% interval
- 0.0 to 0.6
- Test series
- 1
- Test families
- 1 / 3
- Sources
- 1
- Share of the fixed test panel (informative)
- 33 %
Agents
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.2
- Test series
- 1
- Test families
- 1 / 12
- Sources
- 1
- Share of the fixed test panel (informative)
- 8 %
Finance
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.1
- 95% interval
- 0.0 to 1.7
- Test series
- 2
- Test families
- 2 / 7
- Sources
- 1
- Share of the fixed test panel (informative)
- 29 %
Legal
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.0
- Test series
- 1
- Test families
- 1 / 4
- Sources
- 1
- Share of the fixed test panel (informative)
- 25 %
Health
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.2
- Test series
- 1
- Test families
- 1 / 3
- Sources
- 1
- Share of the fixed test panel (informative)
- 33 %
Documented method · Included test families and measurements
Method version: benchmark-meta-score-v3
The index includes active measurements from reviewed primary sources when at least five models are compared in the same documented comparison slice across dataset, harness, agent, tools, and trial count. Each benchmark series and evaluator contributes one shared setup slice from the newest capture. When one source and series include several explicitly documented reasoning tiers, the highest documented tier (none through max) is selected, never the best result value. The explicit source label “Non-reasoning” is treated as none. Unknown or ambiguous tiers exclude the model from that series. Within each series, percentages and rates are logit-transformed, other units stay unchanged, and all values are then standardized. A one-factor model then estimates one shared strength per model and, per series, how sharply it separates the models. Area scores move away from that strength only as far as the model's own results in the area support. The overall score weights the core areas at 75% and finance, law and health at 25%. It requires at least eight test series from two sources and three areas. The 0 to 100 scale refers to a fixed reference group of 29 language models whose average sits at 50.
Published model-specific parameters can differ. This is not a controlled experiment; comparisons with such differences are marked provisional.
Unmeasured series are not entered as zero. Where a model has no own results in a core area, its overall strength stands in, and the note “estimated without own results in …” names the area. Fewer test series lead to a larger standard error.
The 95% interval follows from the standard error of 200 recalculations with randomly redrawn test families. It describes how precisely the recorded test series determine the value. It is no prediction for future tasks or for later recalculations with other participants.
The source count is not a count of independent experiments.
OECD handbook on composite indicatorsAll 12 test families and 12 of 12 raw measurements are listed here. The complete calculation for every model is available in the JSON export.
Test family: Online Mind2Web
Agents · Mean rank percentile of the family: 20.0 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Princeton HAL
Published configurations differ within this cohort.
- Measurement:
- hal-online-mind2web
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 25.33 %
- Metric:
- accuracy
- Version:
- 300-task live set
- Participating models:
- 6
- Rank percentile within the cohort:
- 20.0 %
- Retrieved:
- 2026-08-05
- Comparison cohort:
- hal-online-mind2web-browser-use-overall
- Documented comparison setup
- Publication cohort:
- hal-online-mind2web-browser-use-overall
- Dataset type:
- public
- Harness:
- Browser-Use
- Agent:
- Browser-Use
- Setup and model parameters
- The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Harness: Browser-Use. Agent: Browser-Use
The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Cost per task in USD: 0.9364333333333333. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-664f99d510f5. Source payload hash: 664f99d510f5321c20560c70c760680b5d40263821b082cc494f3bdfae6cc0cb. Parser version: curated-v1. Imported at: 2026-10-15
- Princeton HAL
Test family: ScienceAgentBench
Coding · Mean rank percentile of the family: 20.0 / 100Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
- Princeton HAL
Published configurations differ within this cohort.
- Measurement:
- hal-scienceagentbench
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.71 %
- Raw value:
- 23.53 %
- Metric:
- accuracy
- Version:
- verified 102-task set
- Participating models:
- 6
- Rank percentile within the cohort:
- 20.0 %
- Retrieved:
- 2026-08-06
- Comparison cohort:
- hal-scienceagentbench-sab-self-debug-overall
- Documented comparison setup
- Publication cohort:
- hal-scienceagentbench-sab-self-debug-overall
- Dataset type:
- public
- Harness:
- SAB Self-Debug
- Agent:
- SAB Self-Debug
- Setup and model parameters
- SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Harness: SAB Self-Debug. Agent: SAB Self-Debug
SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SAB Self-Debug. Agent: SAB Self-Debug. Cost per task in USD: 0.1788235294117647. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-6fc49efd47b7. Source payload hash: 6fc49efd47b70abe914ca04758101a25125a9e8a20c5a6c90019906650d6beee. Parser version: curated-v1. Imported at: 2026-10-15
- Princeton HAL
Test family: USACO
Coding · Mean rank percentile of the family: 0.0 / 100Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
- Princeton HAL
Published configurations differ within this cohort.
- Measurement:
- hal-usaco
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.71 %
- Raw value:
- 38.11 %
- Metric:
- accuracy
- Version:
- HAL 307-problem set
- Participating models:
- 6
- Rank percentile within the cohort:
- 0.0 %
- Retrieved:
- 2026-08-06
- Comparison cohort:
- hal-usaco-usaco-episodic-semantic-overall
- Documented comparison setup
- Publication cohort:
- hal-usaco-usaco-episodic-semantic-overall
- Dataset type:
- public
- Harness:
- USACO Episodic + Semantic
- Agent:
- USACO Episodic + Semantic
- Setup and model parameters
- USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic
USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic. Cost per task in USD: 0.2607166123778502. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-b17824331ef0. Source payload hash: b17824331ef027e156721f99110a57b278ee78be01b8959a892bacfb3502ff67. Parser version: curated-v1. Imported at: 2026-10-15
- Princeton HAL
Test family: MultiNRC
Reasoning · Mean rank percentile of the family: 18.2 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 100 %
- Scale AI
Published configurations differ within this cohort.
- Measurement:
- multinrc
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.53 %
- Raw value:
- 27.58 %
- Metric:
- accuracy
- Version:
- French, Spanish, and Chinese release
- Participating models:
- 12
- Rank percentile within the cohort:
- 18.2 %
- Retrieved:
- 2026-08-26
- Comparison cohort:
- scale-multinrc-native-languages
- Documented comparison setup
- Publication cohort:
- scale-multinrc-native-languages
- Dataset type:
- public
- Attempts:
- pass@1
- Setup and model parameters
- Native, non-translated reasoning tasks with automatic evaluation.. Attempts: pass@1
Native, non-translated reasoning tasks with automatic evaluation.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-0528. Evaluator: Scale AI. Dataset type: public. Attempts: pass@1. Score display: 27.58% ± 2.7. Source ID: scale-ai. Evidence capture ID: curated-scale-ai-2026-08-26-ee6d5a59e0c6. Source payload hash: ee6d5a59e0c65ff18a7d24b1e80e2265bd6fdefe8e24bd84ffb982ddcf754154. Parser version: curated-v1. Imported at: 2026-10-15
- Scale AI
Test family: CorpFin v2
Finance · Mean rank percentile of the family: 16.2 / 100Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-corp-fin-v2
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.19 %
- Raw value:
- 54.118 %
- Metric:
- accuracy
- Version:
- 2
- Participating models:
- 75
- Rank percentile within the cohort:
- 16.2 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-corp-fin-v2:2:overall
- Documented comparison setup
- Publication cohort:
- vals-corp-fin-v2:2:overall
- Dataset type:
- private
- Setup and model parameters
- Temperature: 1. Maximum output tokens: 16384
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-08-12. Temperature: 1. Maximum output tokens: 16384. Latency in seconds: 46.774. Standard error: 0.979. Cost per task in USD: 0.086732. Source ID: vals-ai. Evidence capture ID: vals-corp_fin_v2-2026-10-15-7cf2217ab372. Source payload hash: 7cf2217ab3728d5a84bffce82aab9fc52d7f810b0cbe8c65b387df47f3d31c9b. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: LiveCodeBench v6
Coding · Mean rank percentile of the family: 23.4 / 100Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-lcb
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.71 %
- Raw value:
- 70.221 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 78
- Rank percentile within the cohort:
- 23.4 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-lcb:1:overall
- Documented comparison setup
- Publication cohort:
- vals-lcb:1:overall
- Dataset type:
- public
- Setup and model parameters
- Temperature: 1. Maximum output tokens: 16384
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Temperature: 1. Maximum output tokens: 16384. Latency in seconds: 86.074. Standard error: 1.15. Cost per task in USD: 0.040788. Source ID: vals-ai. Evidence capture ID: vals-lcb-2026-10-15-1a7c7dabc9d9. Source payload hash: 1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: LegalBench
Legal · Mean rank percentile of the family: 5.1 / 100Family weight in the fixed test panel (informative): 2.08 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-legal-bench
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.08 %
- Raw value:
- 67.323 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 79
- Rank percentile within the cohort:
- 5.1 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-legal-bench:1:overall
- Documented comparison setup
- Publication cohort:
- vals-legal-bench:1:overall
- Dataset type:
- public
- Setup and model parameters
- Temperature: 1. Maximum output tokens: 2048
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-10-13. Temperature: 1. Maximum output tokens: 2048. Latency in seconds: 18.549. Standard error: 0.535. Source ID: vals-ai. Evidence capture ID: vals-legal_bench-2026-10-15-c8486ba756ba. Source payload hash: c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MATH 500
Mathematics · Mean rank percentile of the family: 59.1 / 100Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-math500
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.14 %
- Raw value:
- 92.2 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 23
- Rank percentile within the cohort:
- 59.1 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-math500:1:overall
- Documented comparison setup
- Publication cohort:
- vals-math500:1:overall
- Dataset type:
- public
- Setup and model parameters
- Temperature: 0. Maximum output tokens: 32768
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-01-09. Temperature: 0. Maximum output tokens: 32768. Latency in seconds: 156.474. Standard error: 2.364. Cost per task in USD: 0.010886. Source ID: vals-ai. Evidence capture ID: vals-math500-2026-10-15-435ec4da1112. Source payload hash: 435ec4da1112362c0a6bb4afeab53bf7b0faf2000e1ddbe1a878735dca1fe35b. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MedQA
Health · Mean rank percentile of the family: 43.6 / 100Family weight in the fixed test panel (informative): 2.78 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-medqa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.78 %
- Raw value:
- 90.8 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 40
- Rank percentile within the cohort:
- 43.6 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-medqa:1:overall
- Documented comparison setup
- Publication cohort:
- vals-medqa:1:overall
- Dataset type:
- public
- Setup and model parameters
- Temperature: 1. Maximum output tokens: 4096
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-04-16. Temperature: 1. Maximum output tokens: 4096. Latency in seconds: 41.569. Standard error: 0.518. Cost per task in USD: 0.005903. Source ID: vals-ai. Evidence capture ID: vals-medqa-2026-10-15-d67905978d93. Source payload hash: d67905978d932f64698ebe9e230d9519e5bae1c9d7176dda8379dc3874b6f90f. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MGSM
Mathematics · Mean rank percentile of the family: 82.8 / 100Family weight in the fixed test panel (informative): 2.14 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-mgsm
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.14 %
- Raw value:
- 92.254 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 30
- Rank percentile within the cohort:
- 82.8 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-mgsm:1:overall
- Documented comparison setup
- Publication cohort:
- vals-mgsm:1:overall
- Dataset type:
- public
- Setup and model parameters
- Temperature: 1. Maximum output tokens: 16384
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-01-09. Temperature: 1. Maximum output tokens: 16384. Latency in seconds: 10.167. Standard error: 1.655. Cost per task in USD: 0.004123. Source ID: vals-ai. Evidence capture ID: vals-mgsm-2026-10-15-1b87f953e52c. Source payload hash: 1b87f953e52c00bf4bdd90c3add26165f644eeb7a74a8382d2a0324ff0cc1040. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: MMLU-Pro
Knowledge · Mean rank percentile of the family: 39.0 / 100Family weight in the fixed test panel (informative): 3.57 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-mmlu-pro
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 3.57 %
- Raw value:
- 83.184 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 78
- Rank percentile within the cohort:
- 39.0 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-mmlu-pro:1:overall
- Documented comparison setup
- Publication cohort:
- vals-mmlu-pro:1:overall
- Dataset type:
- public
- Setup and model parameters
- Temperature: 1. Maximum output tokens: 16384
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Temperature: 1. Maximum output tokens: 16384. Latency in seconds: 27.278. Standard error: 0.444. Cost per task in USD: 0.012339. Source ID: vals-ai. Evidence capture ID: vals-mmlu_pro-2026-10-15-dab025507f46. Source payload hash: dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: TaxEval v2
Finance · Mean rank percentile of the family: 44.7 / 100Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-tax-eval-v2
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.19 %
- Raw value:
- 72.281 %
- Metric:
- accuracy
- Version:
- 2
- Participating models:
- 77
- Rank percentile within the cohort:
- 44.7 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-tax-eval-v2:2:overall
- Documented comparison setup
- Publication cohort:
- vals-tax-eval-v2:2:overall
- Dataset type:
- private
- Setup and model parameters
- Temperature: 1. Maximum output tokens: 16384
Source model ID: fireworks/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: Fireworks AI. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-09-01. Temperature: 1. Maximum output tokens: 16384. Latency in seconds: 152.958. Standard error: 0.874. Cost per task in USD: 0.012239. Source ID: vals-ai. Evidence capture ID: vals-tax_eval_v2-2026-10-15-8b354a6ae848. Source payload hash: 8b354a6ae848f5bb25a2e8b061b5f576d5b9e0c11b941dc1e6bed078e6529cd6. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Benchmark sources at a glance
Each evaluator measures different capabilities under its own conditions. Test series, composite indices and their components stay separate. The detailed results below retain their scales, configurations and source dates.
| Evaluator | Test series | Composite indices | Recorded active measurements |
|---|---|---|---|
| Vals AI | 9 | 0 | 55 |
| Princeton HAL | 4 | 0 | 5 |
| Scale AI Evals | 1 | 0 | 1 |
Coverage is not a quality ranking. Overlapping tests and multiple settings do not count as independent experiments. Vendor claims, historical and disputed results are excluded from this overview and remain labelled in the detailed evidence.
Page 1 of 8
Published benchmarks
Coding
Aider Polyglot
Overall
225 code-editing tasks from Exercism across C++, Go, Java, JavaScript, Python, and Rust. Aider, the edit format, reasoning budget, and two attempts are part of the result.
higher is betterAs of September 2, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- aider-polyglot-pass2-diff
- Metric
- percent correct after two attempts
- Scale
- 0-100
- Participants
- 4
- Methodology and context
- 225 Exercism tasks across six languages using Aider and a documented edit format.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-0528. Evaluator: Aider. Dataset type: public. Leaderboard date: 2025-06-06. Attempts: pass 2. Latency in seconds: 716.6. Cost per task in USD: 0.021340444444444443. Source ID: aider. Evidence capture ID: curated-aider-2026-09-02-83be0654f25d. Source payload hash: 83be0654f25dcee536c478783a0547fbe586876fabb5c3f977893c0de30a9531. Parser version: curated-v1. Imported at: 2026-10-15
ScienceAgentBench
Accuracy
102 expert-validated tasks for data-driven scientific work. The scaffold, tools, and code execution are part of the result. HAL reports both 42 and 44 underlying publications in different places.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- hal-scienceagentbench-sab-self-debug-overall
- Metric
- accuracy
- Scale
- 0-100
- Participants
- 5
- Methodology and context
- SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SAB Self-Debug. Agent: SAB Self-Debug. Cost per task in USD: 0.1788235294117647. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-6fc49efd47b7. Source payload hash: 6fc49efd47b70abe914ca04758101a25125a9e8a20c5a6c90019906650d6beee. Parser version: curated-v1. Imported at: 2026-10-15
USACO
Accuracy
307 algorithmic programming tasks from Bronze through Platinum with exhaustive tests. The benchmark measures competitive programming, not repository navigation, maintainability, or product experience.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- hal-usaco-usaco-episodic-semantic-overall
- Metric
- accuracy
- Scale
- 0-100
- Participants
- 5
- Methodology and context
- USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic. Cost per task in USD: 0.2607166123778502. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-b17824331ef0. Source payload hash: b17824331ef027e156721f99110a57b278ee78be01b8959a892bacfb3502ff67. Parser version: curated-v1. Imported at: 2026-10-15
Online Mind2Web
Accuracy
300 verified tasks across 136 dynamic live websites. SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- hal-online-mind2web-browser-use-overall
- Metric
- accuracy
- Scale
- 0-100
- Participants
- 5
- Methodology and context
- The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Cost per task in USD: 0.9364333333333333. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-664f99d510f5. Source payload hash: 664f99d510f5321c20560c70c760680b5d40263821b082cc494f3bdfae6cc0cb. Parser version: curated-v1. Imported at: 2026-10-15
Reasoning
LongBench v2
503 tasks with contexts from 8,000 to 2 million words cover documents, dialogue, code repositories, and structured data. Scores with chain of thought are not interchangeable with direct-answer results without reasoning.
| Sub-task | DeepSeek-R1 (This model) | Gemini 2.5 Pro | Gemini 2.5 Flash | o1 | GPT-4o |
|---|---|---|---|---|---|
| Easy | 66.1% | 75% (best value in this row) | 72.3% | 66.8% | 54.2% |
| Hard | 53.4% | 56.1% (best value in this row) | 55.8% | 52.1% | 49.7% |
| Long, 128,000 to 2 million words | 59.3% | 71% (best value in this row) | 55.7% | 58.1% | 43.5% |
| Medium, 32,000 to 128,000 words | 54.4% | 56.3% | 60% (best value in this row) | 53.5% | 48.6% |
| Overall | 58.3% | 63.3% (best value in this row) | 62.1% | 57.7% | 51.4% |
| Short, up to 32,000 words | 62.2% | 67.2% | 68.3% (best value in this row) | 62.6% | 59.6% |
higher is betterAs of September 2, 2026Source: LongBenchbars scaled within each row, not across rows
Methodology and context
- Metric
- accuracy with chain of thought
Knowledge
MMLU-ProX
11,829 parallel questions per language compare knowledge and reasoning across 29 languages. Results use five-shot chain of thought and are not interchangeable with MMLU-Pro.
| Sub-task | DeepSeek-R1 (This model) | GPT-4.1 | DeepSeek-V3 | o4-mini |
|---|---|---|---|---|
| Average across 29 languages | 75.5% (best value in this row) | 72.7% | 70.5% | 69.3% |
| German | 76.7% (best value in this row) | 76.4% | 75.1% | 73.5% |
higher is betterAs of August 9, 2026Source: MMLU-ProXbars scaled within each row, not across rows
Methodology and context
- Metric
- accuracy
Recorded measurement details
Aider Polyglot: Overall (Show measurement, test conditions, and source)
- Source value
- 71.4
- Score
- 71.4
- Metric
- percent correct after two attempts
- Unit
- %
- Benchmark version
- 225-case polyglot suite
- Category
- coding
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- aider-polyglot-pass2-diff
- Source type
- official-leaderboard
- Evaluator
- Aider
- Status
- active
- Retrieved at
- 2026-09-02
- Methodology
- 225 Exercism tasks across six languages using Aider and a documented edit format.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-0528. Evaluator: Aider. Dataset type: public. Leaderboard date: 2025-06-06. Attempts: pass 2. Latency in seconds: 716.6. Cost per task in USD: 0.021340444444444443. Source ID: aider. Evidence capture ID: curated-aider-2026-09-02-83be0654f25d. Source payload hash: 83be0654f25dcee536c478783a0547fbe586876fabb5c3f977893c0de30a9531. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 225 Exercism tasks across six languages using Aider and a documented edit format.
Online Mind2Web: Accuracy (Show measurement, test conditions, and source)
- Source value
- 25.33
- Score
- 25.33
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 300-task live set
- Category
- agentic
- Task
- overall
- Task label
- Accuracy
- Direction
- Higher is better
- Comparison cohort
- hal-online-mind2web-browser-use-overall
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Cost per task in USD: 0.9364333333333333. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-664f99d510f5. Source payload hash: 664f99d510f5321c20560c70c760680b5d40263821b082cc494f3bdfae6cc0cb. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.
ScienceAgentBench: Accuracy (Show measurement, test conditions, and source)
- Source value
- 23.53
- Score
- 23.53
- Metric
- accuracy
- Unit
- %
- Benchmark version
- verified 102-task set
- Category
- coding
- Task
- overall
- Task label
- Accuracy
- Direction
- Higher is better
- Comparison cohort
- hal-scienceagentbench-sab-self-debug-overall
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SAB Self-Debug. Agent: SAB Self-Debug. Cost per task in USD: 0.1788235294117647. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-6fc49efd47b7. Source payload hash: 6fc49efd47b70abe914ca04758101a25125a9e8a20c5a6c90019906650d6beee. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.
USACO: Accuracy (Show measurement, test conditions, and source)
- Source value
- 38.11
- Score
- 38.11
- Metric
- accuracy
- Unit
- %
- Benchmark version
- HAL 307-problem set
- Category
- coding
- Task
- overall
- Task label
- Accuracy
- Direction
- Higher is better
- Comparison cohort
- hal-usaco-usaco-episodic-semantic-overall
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic. Cost per task in USD: 0.2607166123778502. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-b17824331ef0. Source payload hash: b17824331ef027e156721f99110a57b278ee78be01b8959a892bacfb3502ff67. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.
LongBench v2: Easy (Show measurement, test conditions, and source)
- Source value
- 66.1
- Score
- 66.1
- Metric
- accuracy with chain of thought
- Unit
- %
- Benchmark version
- v2
- Category
- reasoning
- Task
- easy
- Task label
- Easy
- Direction
- Higher is better
- Comparison cohort
- longbench-v2-cot-easy
- Source type
- official-leaderboard
- Evaluator
- LongBench
- Status
- active
- Retrieved at
- 2026-09-02
- Methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-2025-01-20. Evaluator: LongBench. Dataset type: public. Leaderboard date: 2025-01-20. Attempts: pass@1. Reasoning effort: chain of thought. Source ID: longbench. Evidence capture ID: curated-longbench-2026-09-02-0a5a05f9e399. Source payload hash: 0a5a05f9e3998728b5a5b827562b52b9ca09a1367d7119c88a6029d109ea6ba1. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.
LongBench v2: Hard (Show measurement, test conditions, and source)
- Source value
- 53.4
- Score
- 53.4
- Metric
- accuracy with chain of thought
- Unit
- %
- Benchmark version
- v2
- Category
- reasoning
- Task
- hard
- Task label
- Hard
- Direction
- Higher is better
- Comparison cohort
- longbench-v2-cot-hard
- Source type
- official-leaderboard
- Evaluator
- LongBench
- Status
- active
- Retrieved at
- 2026-09-02
- Methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-2025-01-20. Evaluator: LongBench. Dataset type: public. Leaderboard date: 2025-01-20. Attempts: pass@1. Reasoning effort: chain of thought. Source ID: longbench. Evidence capture ID: curated-longbench-2026-09-02-4bbc7fc01da6. Source payload hash: 4bbc7fc01da6ef1b5e372f2f17fe0849fdde69eabd7caecb2a887466f25e1590. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.
LongBench v2: Long, 128,000 to 2 million words (Show measurement, test conditions, and source)
- Source value
- 59.3
- Score
- 59.3
- Metric
- accuracy with chain of thought
- Unit
- %
- Benchmark version
- v2
- Category
- reasoning
- Task
- long
- Task label
- Long, 128,000 to 2 million words
- Direction
- Higher is better
- Comparison cohort
- longbench-v2-cot-long
- Source type
- official-leaderboard
- Evaluator
- LongBench
- Status
- active
- Retrieved at
- 2026-09-02
- Methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-2025-01-20. Evaluator: LongBench. Dataset type: public. Leaderboard date: 2025-01-20. Attempts: pass@1. Reasoning effort: chain of thought. Source ID: longbench. Evidence capture ID: curated-longbench-2026-09-02-82dc59c97071. Source payload hash: 82dc59c970714785cd17275ff52c361f3d2b8a5993aa36a66c75fdeeb5ae73f7. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.
LongBench v2: Medium, 32,000 to 128,000 words (Show measurement, test conditions, and source)
- Source value
- 54.4
- Score
- 54.4
- Metric
- accuracy with chain of thought
- Unit
- %
- Benchmark version
- v2
- Category
- reasoning
- Task
- medium
- Task label
- Medium, 32,000 to 128,000 words
- Direction
- Higher is better
- Comparison cohort
- longbench-v2-cot-medium
- Source type
- official-leaderboard
- Evaluator
- LongBench
- Status
- active
- Retrieved at
- 2026-09-02
- Methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-2025-01-20. Evaluator: LongBench. Dataset type: public. Leaderboard date: 2025-01-20. Attempts: pass@1. Reasoning effort: chain of thought. Source ID: longbench. Evidence capture ID: curated-longbench-2026-09-02-fb7d385c6728. Source payload hash: fb7d385c672839bb9d1ed3dc00b8c30983a1ce5c55a3eb9d36472f825cd289e0. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.
LongBench v2: Overall (Show measurement, test conditions, and source)
- Source value
- 58.3
- Score
- 58.3
- Metric
- accuracy with chain of thought
- Unit
- %
- Benchmark version
- v2
- Category
- reasoning
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- longbench-v2-cot-overall
- Source type
- official-leaderboard
- Evaluator
- LongBench
- Status
- active
- Retrieved at
- 2026-09-02
- Methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-2025-01-20. Evaluator: LongBench. Dataset type: public. Leaderboard date: 2025-01-20. Attempts: pass@1. Reasoning effort: chain of thought. Source ID: longbench. Evidence capture ID: curated-longbench-2026-09-02-6ffc97a26a0d. Source payload hash: 6ffc97a26a0d168714d6ad2c17b12e71f02f861a991e218b4e26bc83961982ac. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.
LongBench v2: Short, up to 32,000 words (Show measurement, test conditions, and source)
- Source value
- 62.2
- Score
- 62.2
- Metric
- accuracy with chain of thought
- Unit
- %
- Benchmark version
- v2
- Category
- reasoning
- Task
- short
- Task label
- Short, up to 32,000 words
- Direction
- Higher is better
- Comparison cohort
- longbench-v2-cot-short
- Source type
- official-leaderboard
- Evaluator
- LongBench
- Status
- active
- Retrieved at
- 2026-09-02
- Methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Model snapshot: deepseek-r1-2025-01-20. Evaluator: LongBench. Dataset type: public. Leaderboard date: 2025-01-20. Attempts: pass@1. Reasoning effort: chain of thought. Source ID: longbench. Evidence capture ID: curated-longbench-2026-09-02-95504bd97ff3. Source payload hash: 95504bd97ff39c57c15186af68e6f7d6c9ec26f875b04b56acd8b82b88cb1b28. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 503 multiple-choice tasks with the prescribed reasoning prompt.
MMLU-ProX: Average across 29 languages (Show measurement, test conditions, and source)
- Source value
- 75.5
- Score
- 75.5
- Metric
- accuracy
- Unit
- %
- Benchmark version
- EMNLP 2025 full release
- Category
- knowledge
- Task
- avg29
- Task label
- Average across 29 languages
- Direction
- Higher is better
- Comparison cohort
- mmlu-prox:official:avg29
- Source type
- official-leaderboard
- Evaluator
- MMLU-ProX
- Status
- active
- Retrieved at
- 2026-08-09
- Methodology
- Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: MMLU-ProX. Dataset type: public. Source ID: mmlu-prox. Evidence capture ID: curated-mmlu-prox-2026-08-09-8f9f86134c2e. Source payload hash: 8f9f86134c2ebc8925a6d4ff3dbb89da6047f972657ea0db5d5844f2ec5914a1. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.
MMLU-ProX: German (Show measurement, test conditions, and source)
- Source value
- 76.7
- Score
- 76.7
- Metric
- accuracy
- Unit
- %
- Benchmark version
- EMNLP 2025 full release
- Category
- knowledge
- Task
- de
- Task label
- German
- Direction
- Higher is better
- Comparison cohort
- mmlu-prox:official:de
- Source type
- official-leaderboard
- Evaluator
- MMLU-ProX
- Status
- active
- Retrieved at
- 2026-08-09
- Methodology
- Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.. Source model ID: curated/deepseek-r1. Source model name: DeepSeek-R1. Provider reported by source: DeepSeek. Evaluator: MMLU-ProX. Dataset type: public. Source ID: mmlu-prox. Evidence capture ID: curated-mmlu-prox-2026-08-09-e0ead6d94ec9. Source payload hash: e0ead6d94ec90867a9587837a0fe3c52850e102d710bae5ca6e97bf6b86470e1. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.
Page 1 of 8
Our tests
SVG short test
One SVG prompt and fixed output settings. Captures are shown only with 0 reported reasoning tokens or documented lack of reasoning controls. Identical test conditions do not make the models' internal compute identical.
Earlier tests
Pelican on a bicycle
A community classic for free-form SVG drawing.
DeepSeek-R1
Output withheld for safety
Reasoning (requested): high
Current attempt
- Run date
- Oct 5, 2026
- API provider
- openrouter
- Requested API model
- deepseek/deepseek-r1
- API model reported in response
- deepseek/deepseek-r1
- Reasoning setting
- high
- Pinned endpoint
- Novita | deepseek/deepseek-r1
- Temperature
- Provider default
- Token limit including reasoning
- 8,192 tokens
- Test protocol
- community-visual-tests-v1
- Technical checks
- Failed
- Response time
- 293.5 s
Not yet visually reviewed. A successful recording does not confirm correct physics or full compliance with the prompt.
Earlier attempts (1)
- Oct 2, 2026API request failedReasoning setting: high
Prompt and test conditions
Generate an SVG of a pelican riding a bicycle
Token limit including reasoning: 8,192 tokens
Task origin (Simon Willison)Popularity
57.5 / 100
Specifications and access
| Specification | Value and source |
|---|---|
| Weights | Open weightsSource not specified |
| Access | API, LocalSource |
| Parameters | 671BSource not specified |
| Active parameters | 37BSource not specified |
| Context window | 128,000 tokensSource |
| Notes | Reasoning model based on V3 architectureSource not specified |
| License | MITSource |
| Parameters | 671 BSource |
| Active parameters | 37 BSource |
| Architecture | MoE reasoning modelSource |
| Reasoning mode | long chain-of-thought reasoningSource |
| Tool calling | NoSource |