Gemma 4 31B
Language modelActive
Gemma 4 31B
- Released
- March 31, 2026
- Data date
- September 8, 2026
- Predecessor
- Gemma 3 27B
- Benchmark-Meta-Score
- Standard error ± 0.5
- Speed
- 35.3 tokens/sOutput throughput
- Price
- Input · Output / 1M tokens
- Input
- Text, Image
- Output
- Text
Gemma 4 31B is distinguished by its dense architecture, 30.7 billion parameters, and hybrid attention across local and global windows. It is the direct open counterpart to the sparse MoE checkpoint Gemma 4 26B A4B.
Google releases the weights under Apache 2.0. The model card documents text and image input plus reasoning, but no audio. Google defines no hosted Gemini API model ID for the weights.
- Position
- Rank 57 of 91
- Index score
- 0.3 / 100
Methodology and source
The LLM comparison hub's selected-panel index averages rank percentiles from 65 documented comparison cohorts. Benchmark meta-scores are shown in the benchmark section.
Benchmark meta-analysis
Overall and area scores
These Gradually indexes place selected published measurements on a 0 to 100 scale. They are not accuracy percentages and do not guarantee results in your application.
Retrieval dates of the sources used 2026-10-15 - 2026-10-19
Overall index
Index below 1 / 100
provisional
Estimated without own results in: Mathematics and General
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.5
- 95% interval
- 0.0 to 4.0
- Test series
- 12
- Test families
- 10 / 65
- Sources
- 2
- Share of the fixed test panel (informative)
- 13 %
Coding
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.2
- 95% interval
- 0.1 to 0.9
- Test series
- 2
- Test families
- 1 / 15
- Sources
- 2
- Share of the fixed test panel (informative)
- 4 %
Reasoning
Index 1.1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.6
- 95% interval
- 0.3 to 2.8
- Test series
- 3
- Test families
- 3 / 7
- Sources
- 1
- Share of the fixed test panel (informative)
- 36 %
Knowledge
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.0
- 95% interval
- 0.0 to 0.3
- Test series
- 1
- Test families
- 1 / 3
- Sources
- 1
- Share of the fixed test panel (informative)
- 33 %
Agents
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 1.1
- 95% interval
- 0.0 to 12.4
- Test series
- 4
- Test families
- 4 / 12
- Sources
- 1
- Share of the fixed test panel (informative)
- 33 %
Finance
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.4
- 95% interval
- 0.0 to 2.6
- Test series
- 1
- Test families
- 1 / 7
- Sources
- 1
- Share of the fixed test panel (informative)
- 14 %
Education
Index 21.7 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 19.2
- 95% interval
- 2.0 to 69.1
- Test series
- 1
- Test families
- 1 / 2
- Sources
- 1
- Share of the fixed test panel (informative)
- 50 %
Documented method · Included test families and measurements
Method version: benchmark-meta-score-v3
The index includes active measurements from reviewed primary sources when at least five models are compared in the same documented comparison slice across dataset, harness, agent, tools, and trial count. Each benchmark series and evaluator contributes one shared setup slice from the newest capture. When one source and series include several explicitly documented reasoning tiers, the highest documented tier (none through max) is selected, never the best result value. The explicit source label “Non-reasoning” is treated as none. Unknown or ambiguous tiers exclude the model from that series. Within each series, percentages and rates are logit-transformed, other units stay unchanged, and all values are then standardized. A one-factor model then estimates one shared strength per model and, per series, how sharply it separates the models. Area scores move away from that strength only as far as the model's own results in the area support. The overall score weights the core areas at 75% and finance, law and health at 25%. It requires at least eight test series from two sources and three areas. The 0 to 100 scale refers to a fixed reference group of 29 language models whose average sits at 50.
Published model-specific parameters can differ. This is not a controlled experiment; comparisons with such differences are marked provisional.
Unmeasured series are not entered as zero. Where a model has no own results in a core area, its overall strength stands in, and the note “estimated without own results in …” names the area. Fewer test series lead to a larger standard error.
The 95% interval follows from the standard error of 200 recalculations with randomly redrawn test families. It describes how precisely the recorded test series determine the value. It is no prediction for future tasks or for later recalculations with other participants.
The source count is not a count of independent experiments.
OECD handbook on composite indicatorsAll 11 test families and 12 of 12 raw measurements are listed here. The complete calculation for every model is available in the JSON export.
Test family: AA-Briefcase
Agents · Mean rank percentile of the family: 23.2 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-aa-briefcase
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 367.87 Elo
- Metric:
- AA-Briefcase Elo
- Version:
- v1.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 23.2 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-aa-briefcase:v1.1:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-aa-briefcase","v1.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AA-LCR
Reasoning · Mean rank percentile of the family: 25.6 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-aa-lcr
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.53 %
- Raw value:
- 69.6666666666667 %
- Metric:
- Pass rate
- Version:
- v1.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 25.6 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-aa-lcr:v1.1:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-aa-lcr","v1.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 69.7%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AutomationBench-AA
Agents · Mean rank percentile of the family: 31.7 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-automationbench-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 4.933063448681232 %
- Metric:
- Objectives completed without guardrail violations
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 31.7 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-automationbench-aa:unspecified:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-automationbench-aa","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 4.9%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: GDP.pdf
Reasoning · Mean rank percentile of the family: 31.1 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-gdp-pdf
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.53 %
- Raw value:
- 6 %
- Metric:
- All-pass rate
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 31.1 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-gdp-pdf:unspecified:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-gdp-pdf","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 6.0%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: GDPval-AA
Agents · Mean rank percentile of the family: 24.4 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-gdpval-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 619.79 Elo
- Metric:
- GDPval-AA Elo
- Version:
- v2.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 24.4 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-gdpval-aa:v2.1:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-gdpval-aa","v2.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: ITBench-AA (SRE)
Agents · Mean rank percentile of the family: 37.1 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-itbench-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 37.288135593220304 %
- Metric:
- Average precision at full recall
- Version:
- AA cohort: 59 tasks
- Participating models:
- 36
- Rank percentile within the cohort:
- 37.1 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-itbench-aa:AA cohort: 59 tasks:standalone:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-itbench-aa","AA cohort: 59 tasks","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Harness:
- Artificial Analysis Stirrup
- Agent:
- Stirrup agent
- Tool policy:
- Shell access to an offline Kubernetes incident snapshot
- Attempts:
- Average precision at full recall, 3 repeats per task
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 37.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AA-Omniscience
Knowledge · Mean rank percentile of the family: 13.4 / 100Family weight in the fixed test panel (informative): 3.57 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-omniscience
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 3.57 %
- Raw value:
- -47.93333333333333 points
- Metric:
- AA-Omniscience Index
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 13.4 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-omniscience:unspecified:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-omniscience","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: Humanity’s Last Exam
Reasoning · Mean rank percentile of the family: 32.9 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 50 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-humanitys-last-exam
- Series weight within the test family:
- 50 %
- Nominal contribution to the overall panel:
- 0.77 %
- Raw value:
- 23.6329935125116 %
- Metric:
- Text-only pass@1 accuracy
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 32.9 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-humanitys-last-exam:unspecified:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-humanitys-last-exam","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 23.6%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: Terminal-Bench
Coding · Mean rank percentile of the family: 24.0 / 100Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 67 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-terminal-bench-4
- Series weight within the test family:
- 50 %
- Nominal contribution to the overall panel:
- 0.36 %
- Raw value:
- 0 %
- Metric:
- Task success
- Version:
- 4.0
- Participating models:
- 83
- Rank percentile within the cohort:
- 17.7 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-terminal-bench-4:4.0:v4.3.2:reasoning:tokens=1736:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-terminal-bench-4","4.0","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Gemma 4 31B (Reasoning) (release gemma-4-31b, row gemma-4-31b, UUID cd26a386-4873-46ff-b853-d239050025a2, reasoning=true, source reasoning-token configuration budget=1736, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/cd26a386-4873-46ff-b853-d239050025a2/gemma-4-31b/gemma-4-31b. Source model name: Gemma 4 31B. Provider reported by source: Google. Model snapshot: Gemma 4 31B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 0.0%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19 - Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-terminal-bench-2
- Series weight within the test family:
- 16.67 %
- Nominal contribution to the overall panel:
- 0.12 %
- Raw value:
- 39.326 %
- Metric:
- accuracy
- Version:
- 2.0
- Participating models:
- 43
- Rank percentile within the cohort:
- 42.9 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-terminal-bench-2:2.0:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-terminal-bench-2:2.0:overall
- Dataset type:
- public
- Setup and model parameters
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-06-04. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 855.298. Standard error: 5.207. Source ID: vals-ai. Evidence capture ID: vals-terminal-bench-2-2026-10-15-38e640d99525. Source payload hash: 38e640d995259778e531f048b2f2c7b3b281cb05b56be643ddc9dc53eb85adf7. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Artificial Analysis
Test family: MortgageTax
Finance · Mean rank percentile of the family: 21.2 / 100Family weight in the fixed test panel (informative): 1.19 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-mortgage-tax
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.19 %
- Raw value:
- 61.368 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 53
- Rank percentile within the cohort:
- 21.2 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-mortgage-tax:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-mortgage-tax:1:overall
- Dataset type:
- private
- Setup and model parameters
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-09-01. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 41.928. Standard error: 0.967. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-mortgage_tax-2026-10-15-e5c844561f97. Source payload hash: e5c844561f97d119d66d70269e726ecfb68be79eae39acf14ead3e95422e648c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Test family: SAGE
Education · Mean rank percentile of the family: 98.3 / 100Family weight in the fixed test panel (informative): 0 % · Coverage of this test family: 100 %
- Vals AI
Published configurations differ within this cohort.
- Measurement:
- vals-sage
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0 %
- Raw value:
- 55.034 %
- Metric:
- accuracy
- Version:
- 1
- Participating models:
- 59
- Rank percentile within the cohort:
- 98.3 %
- Retrieved:
- 2026-10-15
- Comparison cohort:
- vals-sage:1:overall
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- vals-sage:1:overall
- Dataset type:
- private
- Setup and model parameters
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000
Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-10-08. Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000. Latency in seconds: 93.058. Standard error: 3.411. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-sage-2026-10-15-cc5fb3fecdb0. Source payload hash: cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Vals AI
Benchmark sources at a glance
Each evaluator measures different capabilities under its own conditions. Test series, composite indices and their components stay separate. The detailed results below retain their scales, configurations and source dates.
| Evaluator | Test series | Composite indices | Recorded active measurements |
|---|---|---|---|
| Artificial Analysis | 11 | 1 | 27 |
| Vals AI | 4 | 0 | 13 |
Coverage is not a quality ranking. Overlapping tests and multiple settings do not count as independent experiments. Vendor claims, historical and disputed results are excluded from this overview and remain labelled in the detailed evidence.
6 measurements from these sources still lack a verified test-family identity and do not increase the test-series count.
Page 1 of 5
Published benchmarks
Finance
MortgageTax
| Sub-task | Gemma 4 31B (This model) | GPT-4.1 | GPT-4.1 mini | GPT-4.1 nano |
|---|---|---|---|---|
| Numerical Extraction | 65.66% | 76.23% (best value in this row) | 72.34% | 54.85% |
| Overall | 61.37% | 65.94% (best value in this row) | 65.5% | 52.82% |
| Semantic Extraction | 57.08% | 55.64% | 58.67% (best value in this row) | 50.8% |
higher is betterAs of October 15, 2026Source: Vals AIbars scaled within each row, not across rows
Methodology and context
- Metric
- accuracy
Education
SAGE
| Sub-task | Gemma 4 31B (This model) | Gemini 3 Flash Preview | Gemini 3.1 Flash-Lite | Gemini 3.5 Flash | Gemini 3.7 Flash | Gemini 3.6 Flash | Gemini 3.8 Flash | Gemini 3.1 Pro Preview | Gemini 3.5 Flash-Lite |
|---|---|---|---|---|---|---|---|---|---|
| Automata | 41.67% | 43.25% (best value in this row) | 42.06% | 38.49% | 35.32% | 34.92% | 33.33% | 32.14% | 28.97% |
| Linear Algebra | 72.64% | 71.02% | 58.94% | 68.31% | 69.52% | 69.87% | 41.7% | 74.21% (best value in this row) | 63.74% |
| Overall | 55.03% (best value in this row) | 51.85% | 49.54% | 49.89% | 49.23% | 48.69% | 35.06% | 48.68% | 47.3% |
| Partial Differential Equations | 50.79% (best value in this row) | 41.27% | 47.62% | 42.86% | 42.86% | 41.27% | 30.16% | 39.68% | 49.21% |
higher is betterAs of October 15, 2026Source: Vals AIbars scaled within each row, not across rows
Methodology and context
- Metric
- accuracy
SAGE
Calculus
higher is betterAs of October 15, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- vals-sage:1:calculus
- Metric
- accuracy
- Scale
- 0-100
- Participants
- 8
- Methodology and context
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-10-08. Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000. Latency in seconds: 88.767. Standard error: 6.315. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-sage-2026-10-15-cc5fb3fecdb0. Source payload hash: cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c. Parser version: vals-astro-v3. Imported at: 2026-10-15
Agent tasks
Terminal-Bench 2.0
| Sub-task | Gemma 4 31B (This model) | GPT-4.1 |
|---|---|---|
| Easy | 75% (best value in this row) | 50% |
| Hard | 23.33% (best value in this row) | 3.33% |
| Medium | 45.46% (best value in this row) | 18.18% |
| Overall | 39.33% (best value in this row) | 14.61% |
higher is betterAs of October 15, 2026Source: Vals AIbars scaled within each row, not across rows
Methodology and context
- Metric
- accuracy
Recorded measurement details
MortgageTax: Numerical Extraction (Show measurement, test conditions, and source)
- Source value
- 65.66
- Score
- 65.66
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- finance
- Task
- numerical_extraction
- Task label
- Numerical Extraction
- Direction
- Higher is better
- Comparison cohort
- vals-mortgage-tax:1:numerical_extraction
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-09-01. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 55.497. Standard error: 1.339. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-mortgage_tax-2026-10-15-e5c844561f97. Source payload hash: e5c844561f97d119d66d70269e726ecfb68be79eae39acf14ead3e95422e648c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
MortgageTax: Overall (Show measurement, test conditions, and source)
- Source value
- 61.368
- Score
- 61.368
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- finance
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- vals-mortgage-tax:1:overall
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-09-01. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 41.928. Standard error: 0.967. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-mortgage_tax-2026-10-15-e5c844561f97. Source payload hash: e5c844561f97d119d66d70269e726ecfb68be79eae39acf14ead3e95422e648c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
MortgageTax: Semantic Extraction (Show measurement, test conditions, and source)
- Source value
- 57.075
- Score
- 57.075
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- finance
- Task
- semantic_extraction
- Task label
- Semantic Extraction
- Direction
- Higher is better
- Comparison cohort
- vals-mortgage-tax:1:semantic_extraction
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-09-01. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 28.359. Standard error: 1.396. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-mortgage_tax-2026-10-15-e5c844561f97. Source payload hash: e5c844561f97d119d66d70269e726ecfb68be79eae39acf14ead3e95422e648c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
SAGE: Automata (Show measurement, test conditions, and source)
- Source value
- 41.666
- Score
- 41.666
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- education
- Task
- automata
- Task label
- Automata
- Direction
- Higher is better
- Comparison cohort
- vals-sage:1:automata
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-10-08. Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000. Latency in seconds: 77.76. Standard error: 6.317. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-sage-2026-10-15-cc5fb3fecdb0. Source payload hash: cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000
SAGE: Linear Algebra (Show measurement, test conditions, and source)
- Source value
- 72.643
- Score
- 72.643
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- education
- Task
- linear_algebra
- Task label
- Linear Algebra
- Direction
- Higher is better
- Comparison cohort
- vals-sage:1:linear_algebra
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-10-08. Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000. Latency in seconds: 94.494. Standard error: 5.046. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-sage-2026-10-15-cc5fb3fecdb0. Source payload hash: cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000
SAGE: Overall (Show measurement, test conditions, and source)
- Source value
- 55.034
- Score
- 55.034
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- education
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- vals-sage:1:overall
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-10-08. Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000. Latency in seconds: 93.058. Standard error: 3.411. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-sage-2026-10-15-cc5fb3fecdb0. Source payload hash: cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000
SAGE: Partial Differential Equations (Show measurement, test conditions, and source)
- Source value
- 50.794
- Score
- 50.794
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- education
- Task
- PDEs
- Task label
- Partial Differential Equations
- Direction
- Higher is better
- Comparison cohort
- vals-sage:1:PDEs
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-10-08. Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000. Latency in seconds: 106.921. Standard error: 6.273. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-sage-2026-10-15-cc5fb3fecdb0. Source payload hash: cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000
SAGE: Calculus (Show measurement, test conditions, and source)
- Source value
- 59.259
- Score
- 59.259
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 1
- Category
- education
- Task
- calculus
- Task label
- Calculus
- Direction
- Higher is better
- Comparison cohort
- vals-sage:1:calculus
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: private. Leaderboard date: 2026-10-08. Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000. Latency in seconds: 88.767. Standard error: 6.315. Cost per task in USD: 0. Source ID: vals-ai. Evidence capture ID: vals-sage-2026-10-15-cc5fb3fecdb0. Source payload hash: cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 30000
Terminal-Bench 2.0: Easy (Show measurement, test conditions, and source)
- Source value
- 75
- Score
- 75
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 2.0
- Category
- agentic
- Task
- easy
- Task label
- Easy
- Direction
- Higher is better
- Comparison cohort
- vals-terminal-bench-2:2.0:easy
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-06-04. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 19030.377. Standard error: 25. Source ID: vals-ai. Evidence capture ID: vals-terminal-bench-2-2026-10-15-38e640d99525. Source payload hash: 38e640d995259778e531f048b2f2c7b3b281cb05b56be643ddc9dc53eb85adf7. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
Terminal-Bench 2.0: Hard (Show measurement, test conditions, and source)
- Source value
- 23.333
- Score
- 23.333
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 2.0
- Category
- agentic
- Task
- hard
- Task label
- Hard
- Direction
- Higher is better
- Comparison cohort
- vals-terminal-bench-2:2.0:hard
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-06-04. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 2537.384. Standard error: 7.854. Source ID: vals-ai. Evidence capture ID: vals-terminal-bench-2-2026-10-15-38e640d99525. Source payload hash: 38e640d995259778e531f048b2f2c7b3b281cb05b56be643ddc9dc53eb85adf7. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
Terminal-Bench 2.0: Medium (Show measurement, test conditions, and source)
- Source value
- 45.455
- Score
- 45.455
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 2.0
- Category
- agentic
- Task
- medium
- Task label
- Medium
- Direction
- Higher is better
- Comparison cohort
- vals-terminal-bench-2:2.0:medium
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-06-04. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 1384.027. Standard error: 6.776. Source ID: vals-ai. Evidence capture ID: vals-terminal-bench-2-2026-10-15-38e640d99525. Source payload hash: 38e640d995259778e531f048b2f2c7b3b281cb05b56be643ddc9dc53eb85adf7. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
Terminal-Bench 2.0: Overall (Show measurement, test conditions, and source)
- Source value
- 39.326
- Score
- 39.326
- Metric
- accuracy
- Unit
- %
- Benchmark version
- 2.0
- Category
- agentic
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- vals-terminal-bench-2:2.0:overall
- Source type
- independent-evaluator
- Evaluator
- Vals AI
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Source model ID: google/gemma-4-31b-it. Source model name: Gemma 4 31B. Provider reported by source: Google. Evaluator: Vals AI. Dataset type: public. Leaderboard date: 2026-06-04. Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768. Latency in seconds: 855.298. Standard error: 5.207. Source ID: vals-ai. Evidence capture ID: vals-terminal-bench-2-2026-10-15-38e640d99525. Source payload hash: 38e640d995259778e531f048b2f2c7b3b281cb05b56be643ddc9dc53eb85adf7. Parser version: vals-astro-v3. Imported at: 2026-10-15
- Comparison methodology
- Reasoning effort: high. Temperature: 1. Maximum output tokens: 32768
Page 1 of 5
Our tests
SVG short test
One SVG prompt and fixed output settings. Captures are shown only with 0 reported reasoning tokens or documented lack of reasoning controls. Identical test conditions do not make the models' internal compute identical.
Earlier tests
Pelican on a bicycle
A community classic for free-form SVG drawing.
Gemma 4 31B
Reasoning (requested): high
Current attempt
- Run date
- Oct 1, 2026
- API provider
- openrouter
- Requested API model
- google/gemma-4-31b-it
- API model reported in response
- google/gemma-4-31b-it
- Reasoning setting
- high
- Pinned endpoint
- DeepInfra | google/gemma-4-31b-it-20260402
- Temperature
- Provider default
- Token limit including reasoning
- 8,192 tokens
- Test protocol
- community-visual-tests-v1
- Technical checks
- Passed, not a quality rating
- Response time
- 70.6 s
Visually reviewed. This individual test does not establish overall model quality.
Prompt and test conditions
Generate an SVG of a pelican riding a bicycle
Token limit including reasoning: 8,192 tokens
Task origin (Simon Willison)Popularity
53.3 / 100
Specifications and access
| Specification | Value and source |
|---|---|
| Weights | Open weightsSource |
| Access | API, LocalSource |
| Parameters | 30.7BSource |
| Context window | 262,144 tokensSource |
| Knowledge cutoff | January 2025Source |
| Release | 2026-03-31Source |
| Parameters | 30.7 BSource |
| Input | Text, ImageSource |
| Output | TextSource |
| Architecture | Dense transformer, 30.7B total parameters; hybrid sliding/global attentionSource |
| License | Apache 2.0Source |
| Knowledge cutoff | 2025-01Source |
| Languages | 140+Source |
| Tool calling | YesSource |
| Announcement | 2026-04-02Source |
Predecessor and successor
Sources and data date
| Type | Evidence and data date |
|---|---|
| Speed | Google. Output speed after the first streaming chunk, not total response time. Retrieved September 8, 2026. |
| Price | Base API prices without caching or batch discounts. Higher context tiers and other rates are listed under Costs. |
| Input | Documented modalities. Retrieved September 8, 2026. |
| Output | Documented modalities. Retrieved September 8, 2026. |
| Research status | Research date September 8, 2026. 4 source URLs checked. This documents the inspected sources, not an exhaustive inventory of every publication. |
| Not found in the inspected sources | active-parameters-billions Original wording Active parameter count is not separately published for this dense model. |
| Not found in the inspected sources | api-model-id Original wording Gemma is distributed as open weights; the card does not define a hosted Gemini API model ID. |
| Not found in the inspected sources | training-tokens Original wording The Gemma 4 model card describes the dataset but does not publish a token-count total. |
| Additional source | Model metadata · Gemma 4 31B instruction-tuned weights · Editorial description reviewed October 2, 2026 |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Artificial Analysis (retrieved October 19, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Vals AI (retrieved October 15, 2026) |
| Additional source | Model metadata (retrieved September 8, 2026) |
| Additional source | Model metadata (retrieved September 8, 2026) |
| Additional source | OpenRouter: Popularity index (derived), CC BY 4.0 |
| Additional source | Vercel AI Gateway: Popularity index (derived), CC BY 4.0 |
| Additional source | Ollama: cumulative downloads for the popularity index |
| Additional source | Replicate: cumulative runs for the popularity index |
| Additional source | Google Trending Now: limited feed selection for Germany and the US |
| Additional source | Creative Commons Attribution 4.0 |