Language modelActive

DeepSeek-V3

DeepSeek

Released: December 26, 2024
Data date: September 8, 2026

Intelligence: Not ranked
Overall score
Price: $0.2574 · $1.0287
Input · Output / 1M tokens
Input: Text
Output: Text

No published category-wide ranking is available for this model.

Entries 1-12 of 121. Page 1 of 11.

Published benchmarks

Each chart compares models in the same documented test and highlights the current model.

Agent tasks

2 measurements

AssistantBench

Accuracy

What this benchmark measures: 33 automatically verifiable HAL tasks drawn from 214 realistic, time-consuming web research tasks.

Context: The published HAL subset is smaller than the full dataset. Reported costs do not account for caching benefits.

Higher is better4 modelsDeepSeek-V3: Rank 4 of 4Data date: August 5, 2026

1.o338.81%

2.GPT-535.23%

3.GPT-4.117.39%

4.DeepSeek-V32.03%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-assistantbench-browser-use-overall
Metric: accuracy
Scale: 0-100
Participants: 4
Methodology and context: One shared Browser-Use scaffold solves 33 verified HAL tasks. Total cost is divided by 33.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Model snapshot: deepseek-v3-0324. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Cost per task in USD: 0.38363636363636366. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-9a5202122ebd. Source payload hash: 9a5202122ebda67b0616c9e02bb7338b3a7efbd02c0281f38afbc539486363dc. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

Online Mind2Web

Accuracy

What this benchmark measures: 300 verified tasks across 136 dynamic live websites.

Context: SeeAct and Browser-Use are different agent scaffolds and therefore remain separate comparison cohorts.

Higher is better5 modelsDeepSeek-V3: Rank 2 of 5Data date: August 5, 2026

1.GPT-4.136.33%

2.DeepSeek-V332.33%

3.GPT-532%

4.o329%

5.DeepSeek-R125.33%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-online-mind2web-browser-use-overall
Metric: accuracy
Scale: 0-100
Participants: 5
Methodology and context: The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Cost per task in USD: 0.7158. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-4f73f352634c. Source payload hash: 4f73f352634c61b68118cfe631a33093434e04a6de3ca7be5e55ee7a038f4074. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

Coding

4 measurements

SciCode

Accuracy

What this benchmark measures: 65 scientific coding problems with 338 subproblems across 16 research fields.

Context: Zero Shot, Tool Calling, and HAL Generalist are not directly comparable scaffolds.

Higher is better3 modelsDeepSeek-V3: Rank 3 of 3Data date: August 5, 2026

1.GPT-4.16.15%

2.o34.62%

3.DeepSeek-V33.08%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-scicode-scicode-zero-shot-overall
Metric: accuracy
Scale: 0-100
Participants: 3
Methodology and context: SciCode Zero Shot on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SciCode Zero Shot. Agent: SciCode Zero Shot. Cost per task in USD: 0.012153846153846154. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-8ba24e65935f. Source payload hash: 8ba24e65935f745e28c39d827922a1504e0dd3c542bbbf7f52073bf2ed1c60e2. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

ScienceAgentBench

Accuracy

What this benchmark measures: 102 expert-validated tasks for data-driven scientific work.

Context: The scaffold, tools, and code execution are part of the result. HAL reports both 42 and 44 underlying publications in different places.

Higher is better5 modelsDeepSeek-V3: Rank 5 of 5Data date: August 6, 2026

1.o333.33%

2.GPT-530.39%

3.GPT-4.124.51%

4.DeepSeek-R123.53%

5.DeepSeek-V315.69%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-scienceagentbench-sab-self-debug-overall
Metric: accuracy
Scale: 0-100
Participants: 5
Methodology and context: SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SAB Self-Debug. Agent: SAB Self-Debug. Cost per task in USD: 0.02049019607843137. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-205da614cb5f. Source payload hash: 205da614cb5f85706a2bf9b0e8c27e8a75cf8c503ee510834f6ee27bdc15e656. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

SWE-bench Verified Mini

Accuracy

What this benchmark measures: A random, fully human-validated subset of 50 GitHub issues from 12 Python repositories.

Context: The small subset is cheaper but statistically less stable than the full 500 tasks. SWE-Agent and HAL Generalist remain separate.

Note (DeepSeek-V3): HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

Higher is better4 modelsDeepSeek-V3: Rank 4 of 4Data date: August 5, 2026

1.GPT-546%

Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

1.o346%

Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

3.GPT-4.144%

Note: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

4.DeepSeek-V324%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-swebench-verified-mini-swe-agent-overall
Metric: accuracy
Scale: 0-100
Participants: 4
Methodology and context: SWE-Agent on the same random 50-task subset. Total cost is divided by 50.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SWE-Agent. Agent: SWE-Agent. Cost per task in USD: 0.2354. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-69f60128a276. Source payload hash: 69f60128a276569b958f9a6412d4d7437b18e233dd872ac2800cfd8194e19711. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

USACO

Accuracy

What this benchmark measures: 307 algorithmic programming tasks from Bronze through Platinum with exhaustive tests.

Context: The benchmark measures competitive programming, not repository navigation, maintainability, or product experience.

Higher is better5 modelsDeepSeek-V3: Rank 4 of 5Data date: August 6, 2026

1.GPT-569.06%

2.o346.25%

3.GPT-4.144.95%

4.DeepSeek-V339.09%

5.DeepSeek-R138.11%

Scale: 0% - 100%

Source: Princeton HAL

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: hal-usaco-usaco-episodic-semantic-overall
Metric: accuracy
Scale: 0-100
Participants: 5
Methodology and context: USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic. Cost per task in USD: 0.039348534201954395. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-d7390cf317e3. Source payload hash: d7390cf317e3a69cc8a9318b4278c5077dfef4f126088a7f33bd3bde371e78f9. Parser version: curated-v1. Imported at: 2026-09-04

Source: Princeton HAL

Knowledge

2 measurements

MMLU-ProX

Average across 29 languages

What this benchmark measures: 11,829 parallel questions per language compare knowledge and reasoning across 29 languages.

Context: Results use five-shot chain of thought and are not interchangeable with MMLU-Pro.

Higher is better4 modelsDeepSeek-V3: Rank 3 of 4Data date: August 9, 2026

1.DeepSeek-R175.5%

2.GPT-4.172.7%

3.DeepSeek-V370.5%

4.o4-mini69.3%

Scale: 0% - 100%

Source: MMLU-ProX

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: mmlu-prox:official:avg29
Metric: accuracy
Scale: 0-100
Participants: 4
Methodology and context: Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: MMLU-ProX. Dataset type: public. Source ID: mmlu-prox. Evidence capture ID: curated-mmlu-prox-2026-08-09-ca9ab5dfeb4b. Source payload hash: ca9ab5dfeb4ba849dd5a0c1c49be73ac5229e5feeafb72eee1c7feacf57d680b. Parser version: curated-v1. Imported at: 2026-09-04

Source: MMLU-ProX

MMLU-ProX

German

What this benchmark measures: 11,829 parallel questions per language compare knowledge and reasoning across 29 languages.

Context: Results use five-shot chain of thought and are not interchangeable with MMLU-Pro.

Higher is better4 modelsDeepSeek-V3: Rank 3 of 4Data date: August 9, 2026

1.DeepSeek-R176.7%

2.GPT-4.176.4%

3.DeepSeek-V375.1%

4.o4-mini73.5%

Scale: 0% - 100%

Source: MMLU-ProX

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: mmlu-prox:official:de
Metric: accuracy
Scale: 0-100
Participants: 4
Methodology and context: Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: MMLU-ProX. Dataset type: public. Source ID: mmlu-prox. Evidence capture ID: curated-mmlu-prox-2026-08-09-7c8587b85110. Source payload hash: 7c8587b85110113c94fb5098e12ebbbe37dd97d735b9599d3af4f4bbd1d492ef. Parser version: curated-v1. Imported at: 2026-09-04

Source: MMLU-ProX

CorpFin v2

Overall

Higher is better50 modelsDeepSeek-V3: Rank 46 of 50Data date: September 4, 2026

1.Claude Opus 573.19%

2.Claude Fable 571.83%

3.Kimi K371.56%

4.Inkling Small69.62%

5.Inkling68.57%

6.Grok 4.368.53%

7.GPT-5.568.42%

8.Kimi K2.568.26%

9.Grok 4.567.41%

10.Claude Sonnet 566.98%

11.Kimi K2.666.74%

12.Gemini 3 Flash66.43%

13.Grok 4.666.16%

14.GLM-5.266.12%

15.GPT-5.6 Terra65.31%

16.GPT-5.465.27%

17.Gemini 3.5 Flash64.69%

18.Gemini 3.1 Pro Preview64.49%

19.GLM-5.164.45%

20.GPT-5.6 Sol64.38%

21.GPT-5.6 Luna64.22%

22.Gemini 3.6 Flash63.33%

23.GPT-4.163.05%

24.GLM-562.9%

25.MiMo-V2.5-Pro61.42%

26.DeepSeek-V4-Pro61.38%

27.GPT-5.4 nano61.19%

28.GPT-561.07%

29.GLM-4.560.96%

30.GPT-5.4 mini60.92%

31.Gemini 2.5 Pro60.8%

32.Gemini 3.5 Flash-Lite60.68%

33.Kimi K2 Thinking60.57%

34.GPT-5 mini60.18%

35.MiMo-V2.559.91%

36.o359.71%

37.Gemini 3.1 Flash-Lite59.36%

38.o4-mini58.97%

39.GPT OSS 120B58.24%

40.GPT-4.1 mini57.93%

41.GLM-4.656.84%

42.DeepSeek-V3 032454.74%

43.Trinity Large Thinking54.66%

44.DeepSeek-R154.12%

45.GPT OSS 20B53.15%

46.DeepSeek-V352.49%

47.GLM-4.746.39%

48.GPT-4o mini45.46%

49.o3-mini45.3%

50.GPT-4.1 nano42.08%

Scale: 0% - 100%

Source: Vals AI

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: vals-corp-fin-v2:2:overall
Metric: accuracy
Scale: 0-100
Participants: 50
Methodology and context: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-08-12. Latency in seconds: 28.322. Cost per task in USD: 0.052026. Source ID: vals-ai. Evidence capture ID: vals-corp_fin_v2-2026-09-04-46dad6684551. Source payload hash: 46dad668455190ee4239b7c743d00d81c6fd3127e0d87123c518ec107c13b44d. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source: Vals AI

GPQA Diamond

Overall

Higher is better54 modelsDeepSeek-V3: Rank 51 of 54Data date: September 4, 2026

1.Gemini 3.1 Pro Preview95.45%

2.GPT-5.6 Sol95.2%

3.Grok 4.694.7%

4.Gemini 3.8 Flash94.44%

5.Gemini 3.7 Flash93.94%

6.Claude Opus 593.43%

6.Gemini 3.6 Flash93.43%

8.Claude Fable 593.18%

8.GPT-5.593.18%

10.Grok 4.592.93%

10.Kimi K392.93%

12.Gemini 3.5 Flash92.68%

13.GPT-5.491.67%

13.GPT-5.6 Luna91.67%

15.Grok 4.391.41%

16.GPT-5.6 Terra90.91%

17.DeepSeek-V4-Pro89.39%

18.Kimi K2.689.14%

19.Claude Sonnet 588.89%

20.GLM-5.388.13%

21.Gemini 3 Flash87.88%

22.Inkling87.12%

23.GLM-5.3-Flash86.36%

24.GLM-5.285.61%

24.GPT-585.61%

26.GLM-5.184.52%

27.Kimi K2.584.09%

27.o384.09%

29.Gemini 3.5 Flash-Lite83.84%

30.Inkling Small83.59%

31.GLM-583.33%

32.GPT-5.4 mini83.08%

33.MiMo-V2.5-Pro82.58%

34.MiMo-V2.581.57%

35.Gemini 3.1 Flash-Lite81.06%

36.GPT-5 mini80.3%

37.GLM-4.780.05%

38.GPT OSS 120B78.54%

38.Kimi K2 Thinking78.54%

40.GPT-5.4 nano77.53%

41.o3-mini75.51%

42.GLM-4.674.5%

42.o4-mini74.5%

44.o173.23%

45.GLM-4.572.22%

46.GPT OSS 20B68.94%

47.GPT-4.1 mini67.93%

48.GPT-4.165.4%

49.GPT-5 nano63.38%

50.DeepSeek-V3 032461.62%

51.DeepSeek-V354.55%

52.GPT-4.1 nano50.76%

53.GPT-4o mini44.19%

54.Command R+31.06%

Scale: 0% - 100%

Source: Vals AI

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: vals-gpqa:1:overall
Metric: accuracy
Scale: 0-100
Participants: 54
Methodology and context: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Latency in seconds: 21.282. Cost per task in USD: 0.001468. Source ID: vals-ai. Evidence capture ID: vals-gpqa-2026-09-04-fc438726f0d8. Source payload hash: fc438726f0d8523d320f02d75ae372a3fa38b28b36f70fe318d119f55153560d. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source: Vals AI

LegalBench

Overall

Higher is better53 modelsDeepSeek-V3: Rank 34 of 53Data date: September 4, 2026

1.Claude Fable 588.56%

2.Gemini 3.1 Pro Preview87.4%

3.Gemini 3.7 Flash87.26%

4.Gemini 3.8 Flash86.99%

5.Claude Opus 586.97%

6.GPT-5.6 Sol86.97%

7.Gemini 3 Flash86.86%

8.Gemini 3.6 Flash86.71%

9.GPT-5.586.52%

10.Grok 4.686.31%

11.GPT-5.486.04%

12.GPT-586.02%

13.Kimi K386.02%

14.Grok 4.585.97%

15.GPT-5.6 Terra85.11%

16.GLM-5.384.84%

17.Kimi K2.684.74%

18.Grok 4.384.46%

19.GLM-5.184.39%

20.Gemini 3.5 Flash-Lite84.07%

20.GLM-5.284.07%

22.GLM-584.06%

23.GPT-5.6 Luna84.03%

24.GLM-5.3-Flash83.93%

25.Claude Sonnet 583.92%

26.Gemini 3.1 Flash-Lite83.76%

27.o383.76%

28.Gemini 3.5 Flash83.6%

29.GLM-4.783.36%

30.GPT-4.183.1%

31.Inkling Small82.99%

32.Inkling82.95%

33.GPT-5 mini81.77%

34.DeepSeek-V380.76%

35.o180.39%

36.DeepSeek-V4-Pro80.32%

37.Kimi K2 Thinking80.2%

38.GLM-4.679.61%

39.o4-mini79.19%

40.MiMo-V2.578.89%

41.GPT-4.1 mini78.04%

42.GPT-5.4 nano77.92%

43.DeepSeek-V3 032477.73%

44.MiMo-V2.5-Pro77.13%

45.GPT OSS 120B75.94%

46.GLM-4.575.63%

47.o3-mini71.54%

48.GPT OSS 20B70.85%

49.Command R+68.29%

50.DeepSeek-R167.32%

51.GPT-4.1 nano61.06%

52.GPT-5 nano50.13%

53.Command R32.97%

Scale: 0% - 100%

Source: Vals AI

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: vals-legal-bench:1:overall
Metric: accuracy
Scale: 0-100
Participants: 53
Methodology and context: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Latency in seconds: 4.395. Source ID: vals-ai. Evidence capture ID: vals-legal_bench-2026-09-04-ff81649923fb. Source payload hash: ff81649923fb684ff37d5f557fd320ba5ab433927831065ed781a94548f480ef. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source: Vals AI

MATH 500

Overall

Higher is better17 modelsDeepSeek-V3: Rank 15 of 17Data date: September 4, 2026

1.GPT-596%

2.GPT OSS 120B94.8%

2.GPT-5 mini94.8%

4.o394.6%

5.GPT OSS 20B94.2%

5.o4-mini94.2%

7.GLM-4.594%

8.GPT-5 nano93.8%

9.DeepSeek-R192.2%

10.o3-mini91.8%

11.o190.4%

12.DeepSeek-V3 032488.6%

13.GPT-4.1 mini88%

14.GPT-4.187.2%

15.DeepSeek-V380.4%

16.GPT-4.1 nano80.2%

17.GPT-4o mini72.6%

Scale: 0% - 100%

Source: Vals AI

CC BY 4.0

gradually.ai

Methodology and context

Comparison cohort: vals-math500:1:overall
Metric: accuracy
Scale: 0-100
Participants: 17
Methodology and context: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-01-09. Latency in seconds: 7.94. Cost per task in USD: 0.000563. Source ID: vals-ai. Evidence capture ID: vals-math500-2026-09-04-a87fdb25617d. Source payload hash: a87fdb25617d5588f5c63d5774177861d27be3a848b9eb67a7f417d2b97cb127. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source: Vals AI

Recorded measurement details

Every observation retains its source value and published test conditions.

AssistantBench: Accuracy (Show measurement, test conditions, and source)

Source value: 2.03
Score: 2.03
Metric: accuracy
Unit: %
Benchmark version: HAL 33-task public set
Category: agentic
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-assistantbench-browser-use-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: One shared Browser-Use scaffold solves 33 verified HAL tasks. Total cost is divided by 33.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Model snapshot: deepseek-v3-0324. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Cost per task in USD: 0.38363636363636366. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-9a5202122ebd. Source payload hash: 9a5202122ebda67b0616c9e02bb7338b3a7efbd02c0281f38afbc539486363dc. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: One shared Browser-Use scaffold solves 33 verified HAL tasks. Total cost is divided by 33.

Source Princeton HAL

Online Mind2Web: Accuracy (Show measurement, test conditions, and source)

Source value: 32.33
Score: 32.33
Metric: accuracy
Unit: %
Benchmark version: 300-task live set
Category: agentic
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-online-mind2web-browser-use-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: Browser-Use. Agent: Browser-Use. Cost per task in USD: 0.7158. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-4f73f352634c. Source payload hash: 4f73f352634c61b68118cfe631a33093434e04a6de3ca7be5e55ee7a038f4074. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: The Browser-Use scaffold handles the same 300 tasks across 136 live websites. Total cost is divided by 300.

Source Princeton HAL

SciCode: Accuracy (Show measurement, test conditions, and source)

Source value: 3.08
Score: 3.08
Metric: accuracy
Unit: %
Benchmark version: 65-problem main set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-scicode-scicode-zero-shot-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: SciCode Zero Shot on 65 main scientific problems. Total cost is divided by 65.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SciCode Zero Shot. Agent: SciCode Zero Shot. Cost per task in USD: 0.012153846153846154. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-8ba24e65935f. Source payload hash: 8ba24e65935f745e28c39d827922a1504e0dd3c542bbbf7f52073bf2ed1c60e2. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: SciCode Zero Shot on 65 main scientific problems. Total cost is divided by 65.

Source Princeton HAL

ScienceAgentBench: Accuracy (Show measurement, test conditions, and source)

Source value: 15.69
Score: 15.69
Metric: accuracy
Unit: %
Benchmark version: verified 102-task set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-scienceagentbench-sab-self-debug-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-06
Methodology: SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SAB Self-Debug. Agent: SAB Self-Debug. Cost per task in USD: 0.02049019607843137. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-205da614cb5f. Source payload hash: 205da614cb5f85706a2bf9b0e8c27e8a75cf8c503ee510834f6ee27bdc15e656. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: SAB Self-Debug on 102 scientific tasks. Total cost is divided by 102 and excludes caching benefits according to HAL.

Source Princeton HAL

SWE-bench Verified Mini: Accuracy (Show measurement, test conditions, and source)

Source value: 24
Score: 24
Metric: accuracy
Unit: %
Benchmark version: 50-task random verified subset
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-swebench-verified-mini-swe-agent-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-05
Methodology: SWE-Agent on the same random 50-task subset. Total cost is divided by 50.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: SWE-Agent. Agent: SWE-Agent. Cost per task in USD: 0.2354. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-69f60128a276. Source payload hash: 69f60128a276569b958f9a6412d4d7437b18e233dd872ac2800cfd8194e19711. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: SWE-Agent on the same random 50-task subset. Total cost is divided by 50.
Context: HAL reports min-max ranges for repeated runs. The central value is the reported mean, not a 95% confidence interval.

Source Princeton HAL

USACO: Accuracy (Show measurement, test conditions, and source)

Source value: 39.09
Score: 39.09
Metric: accuracy
Unit: %
Benchmark version: HAL 307-problem set
Category: coding
Task: overall
Task label: Accuracy
Direction: Higher is better
Comparison cohort: hal-usaco-usaco-episodic-semantic-overall
Source type: independent-evaluator
Evaluator: Princeton HAL
Status: active
Retrieved at: 2026-08-06
Methodology: USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: Princeton HAL. Dataset type: public. Harness: USACO Episodic + Semantic. Agent: USACO Episodic + Semantic. Cost per task in USD: 0.039348534201954395. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-d7390cf317e3. Source payload hash: d7390cf317e3a69cc8a9318b4278c5077dfef4f126088a7f33bd3bde371e78f9. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: USACO Episodic + Semantic on the same 307-problem split. Total cost is divided by 307 and excludes caching benefits.

Source Princeton HAL

MMLU-ProX: Average across 29 languages (Show measurement, test conditions, and source)

Source value: 70.5
Score: 70.5
Metric: accuracy
Unit: %
Benchmark version: EMNLP 2025 full release
Category: knowledge
Task: avg29
Task label: Average across 29 languages
Direction: Higher is better
Comparison cohort: mmlu-prox:official:avg29
Source type: official-leaderboard
Evaluator: MMLU-ProX
Status: active
Retrieved at: 2026-08-09
Methodology: Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: MMLU-ProX. Dataset type: public. Source ID: mmlu-prox. Evidence capture ID: curated-mmlu-prox-2026-08-09-ca9ab5dfeb4b. Source payload hash: ca9ab5dfeb4ba849dd5a0c1c49be73ac5229e5feeafb72eee1c7feacf57d680b. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.

Source MMLU-ProX

MMLU-ProX: German (Show measurement, test conditions, and source)

Source value: 75.1
Score: 75.1
Metric: accuracy
Unit: %
Benchmark version: EMNLP 2025 full release
Category: knowledge
Task: de
Task label: German
Direction: Higher is better
Comparison cohort: mmlu-prox:official:de
Source type: official-leaderboard
Evaluator: MMLU-ProX
Status: active
Retrieved at: 2026-08-09
Methodology: Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.. Source model ID: curated/deepseek-v3. Source model name: DeepSeek-V3. Provider reported by source: DeepSeek. Evaluator: MMLU-ProX. Dataset type: public. Source ID: mmlu-prox. Evidence capture ID: curated-mmlu-prox-2026-08-09-7c8587b85110. Source payload hash: 7c8587b85110113c94fb5098e12ebbbe37dd97d735b9599d3af4f4bbd1d492ef. Parser version: curated-v1. Imported at: 2026-09-04
Comparison methodology: Full 29-language dataset with five-shot chain of thought. The source states August 2025 without a verifiable publication day.

Source MMLU-ProX

CorpFin v2: Overall (Show measurement, test conditions, and source)

Source value: 52.486
Score: 52.486
Metric: accuracy
Unit: %
Benchmark version: 2
Category: finance
Task: overall
Task label: Overall
Direction: Higher is better
Comparison cohort: vals-corp-fin-v2:2:overall
Source type: independent-evaluator
Evaluator: Vals AI
Status: active
Retrieved at: 2026-09-04
Methodology: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-08-12. Latency in seconds: 28.322. Cost per task in USD: 0.052026. Source ID: vals-ai. Evidence capture ID: vals-corp_fin_v2-2026-09-04-46dad6684551. Source payload hash: 46dad668455190ee4239b7c743d00d81c6fd3127e0d87123c518ec107c13b44d. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source Vals AI

GPQA Diamond: Overall (Show measurement, test conditions, and source)

Source value: 54.546
Score: 54.546
Metric: accuracy
Unit: %
Benchmark version: 1
Category: reasoning
Task: overall
Task label: Overall
Direction: Higher is better
Comparison cohort: vals-gpqa:1:overall
Source type: independent-evaluator
Evaluator: Vals AI
Status: active
Retrieved at: 2026-09-04
Methodology: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Latency in seconds: 21.282. Cost per task in USD: 0.001468. Source ID: vals-ai. Evidence capture ID: vals-gpqa-2026-09-04-fc438726f0d8. Source payload hash: fc438726f0d8523d320f02d75ae372a3fa38b28b36f70fe318d119f55153560d. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source Vals AI

LegalBench: Overall (Show measurement, test conditions, and source)

Source value: 80.762
Score: 80.762
Metric: accuracy
Unit: %
Benchmark version: 1
Category: legal
Task: overall
Task label: Overall
Direction: Higher is better
Comparison cohort: vals-legal-bench:1:overall
Source type: independent-evaluator
Evaluator: Vals AI
Status: active
Retrieved at: 2026-09-04
Methodology: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-09-01. Latency in seconds: 4.395. Source ID: vals-ai. Evidence capture ID: vals-legal_bench-2026-09-04-ff81649923fb. Source payload hash: ff81649923fb684ff37d5f557fd320ba5ab433927831065ed781a94548f480ef. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source Vals AI

MATH 500: Overall (Show measurement, test conditions, and source)

Source value: 80.4
Score: 80.4
Metric: accuracy
Unit: %
Benchmark version: 1
Category: math
Task: overall
Task label: Overall
Direction: Higher is better
Comparison cohort: vals-math500:1:overall
Source type: independent-evaluator
Evaluator: Vals AI
Status: active
Retrieved at: 2026-09-04
Methodology: Source model ID: fireworks/deepseek-v3. Source model name: DeepSeek-V3. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-01-09. Latency in seconds: 7.94. Cost per task in USD: 0.000563. Source ID: vals-ai. Evidence capture ID: vals-math500-2026-09-04-a87fdb25617d. Source payload hash: a87fdb25617d5588f5c63d5774177861d27be3a848b9eb67a7f417d2b97cb127. Parser version: vals-astro-v2. Imported at: 2026-09-04

Source Vals AI

Our model tests

Community tests you can inspect

Compare how the models respond to the same prompt. Each test shows the first attempt, with no subsequent fixes to the generated code. These results do not contribute to an overall score.

Pelican on a bicycle

A community classic for free-form SVG drawing.

DeepSeek-V3

Not tested yet

Prompt and test conditions

Generate an SVG of a pelican riding a bicycle

Token limit including reasoning: 8,192 tokens

Task origin (Simon Willison)

DeepSeek-V3

Published benchmarks

Agent tasks

AssistantBench

Online Mind2Web

Coding

SciCode

ScienceAgentBench

SWE-bench Verified Mini

USACO

Knowledge

MMLU-ProX

MMLU-ProX

CorpFin v2

GPQA Diamond

LegalBench

MATH 500

Recorded measurement details

Community tests you can inspect

Pelican on a bicycle

Specifications and access

Additional source observations

Published prices

100 API requests

Compare this model

Sources and data date