Gemini 3.5 Flash
Unavailable in this round
Language modelActive
Gemini 3.5 Flash
- Released
- May 19, 2026
- Data date
- September 8, 2026
- Predecessor
- Gemini 3 Flash Preview
- Successor
- Gemini 3.6 Flash
- Benchmark-Meta-Score
- Provisional, 55.5 % panel coverage
- Speed
- 209 tokens/sOutput throughput
- Price
- Input · Output / 1M tokens
- Input
- Text, Image, Audio, Video
- Output
- Text
What was new in Gemini 3.5 Flash was its stable GA release in a line whose direct predecessor is Gemini 3 Flash Preview and which, according to the model card, underpins Gemini 3.6 Flash. The GA label separates it from the neighboring preview.
Google documents a 1,048,576-token context, 65,536 output tokens, and multimodal input. It discloses neither a parameter count nor a knowledge cutoff.
- Position
- Rank 20 of 24
- Index score
- 28.8 / 100
Methodology and source
The LLM comparison hub's selected-panel index averages rank percentiles from 10 documented comparison cohorts. Benchmark meta-scores are shown in the benchmark section.
Benchmark meta-analysis
Overall and area scores
These Gradually indexes place selected published measurements on a 0 to 100 scale. They are not accuracy percentages and do not guarantee results in your application.
Retrieval dates of the sources used 2026-08-05 - 2026-10-19
Overall index
Index 60.3 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 56 %
- Test families
- 46 / 71
- Sources
- 6
- Mathematical bounds with missing values
- 33.5-77.9 / 100
Coding
Not enough coverage for a score yet
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 47 %
- Test families
- 8 / 15
- Sources
- 3
- Mathematical bounds with missing values
- 23.0-76.4 / 100
Reasoning
Index 50.5 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 79 %
- Test families
- 6 / 7
- Sources
- 4
- Mathematical bounds with missing values
- 39.7-61.1 / 100
Mathematics
Index 27.0 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 60 %
- Test families
- 3 / 5
- Sources
- 2
- Mathematical bounds with missing values
- 16.2-56.2 / 100
Knowledge
Index 79.4 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 100 %
- Test families
- 3 / 3
- Sources
- 3
- Mathematical bounds with missing values
- 79.4-79.4 / 100
Agents
Index 68.0 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 75 %
- Test families
- 9 / 12
- Sources
- 3
- Mathematical bounds with missing values
- 51.0-76.0 / 100
General
Not enough coverage for a score yet
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 40 %
- Test families
- 2 / 5
- Sources
- 2
- Mathematical bounds with missing values
- 30.5-90.5 / 100
Finance
Index 66.6 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 86 %
- Test families
- 6 / 7
- Sources
- 1
- Mathematical bounds with missing values
- 57.1-71.4 / 100
Legal
Index 49.5 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 75 %
- Test families
- 3 / 4
- Sources
- 1
- Mathematical bounds with missing values
- 37.1-62.1 / 100
Health
Index 61.7 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 67 %
- Test families
- 2 / 3
- Sources
- 1
- Mathematical bounds with missing values
- 41.1-74.5 / 100
Multimodal
Index 77.1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 50 %
- Test families
- 2 / 4
- Sources
- 3
- Mathematical bounds with missing values
- 38.6-88.6 / 100
Productivity
Not enough coverage for a score yet
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 50 %
- Test families
- 1 / 2
- Sources
- 1
- Mathematical bounds with missing values
- 17.1-67.1 / 100
Education
Not enough coverage for a score yet
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Panel-weight coverage
- 50 %
- Test families
- 1 / 2
- Sources
- 1
- Mathematical bounds with missing values
- 37.9-87.9 / 100
Documented method · Included test families and measurements
Method version: benchmark-meta-score-v1
The index includes active measurements from reviewed primary sources when at least five models are compared in the same documented comparison slice across dataset, harness, agent, tools, and trial count. Each benchmark series and evaluator contributes one shared setup slice from the newest capture; when dates match, the wider slice is used. When one source and series include several explicitly documented reasoning tiers, the highest documented tier (none through max) is selected, never the best result value. The explicit source label “Non-reasoning” is treated as none. When selecting between multiple results, unknown or ambiguous tiers exclude the model from that series. Unambiguous single configurations do not require a tier ordering. The best individual result for each model is not selected. Equal raw values share the same rank percentile. Multiple series from the same evaluator within a family receive equal weight; evaluators within a family, families within an area, and areas in the overall index then receive equal weight. Unmeasured series remain visible in coverage and bounds. An overall score requires at least four areas, six test families, and two sources. An area score requires at least two test families. Every overall and area score also requires at least 50% of its panel weight.
Published model-specific parameters can differ. This is not a controlled experiment; comparisons with such differences are marked provisional.
Unmeasured portions are not entered as zero. They reduce coverage and contribute only to the displayed bounds. Scores with different coverage are therefore not equally complete.
The bounds hold observed percentiles fixed and allow missing contributions anywhere from 0 to 100. They are not confidence intervals or predictions of future recalculations.
The source count is not a count of independent experiments.
OECD handbook on composite indicatorsAll 46 test families and 12 of 52 raw measurements are listed here. The complete calculation for every model is available in the JSON export.
Test family: AIME
Mathematics · Family index: 13.6 / 100Nominal family weight in the overall panel: 1.43 % · Coverage of this test family: 100 %
- Epoch AI benchmark run dataset snapshot
Published configurations differ within this cohort.
- Measurement:
- epoch-otis-mock-aime-2024-2025
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.43 %
- Raw value:
- 0.8888888888888888 ratio
- Metric:
- model_extracted_exact_match
- Version:
- 1.1.6
- Participating models:
- 23
- Rank percentile within the cohort:
- 13.6 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- epoch-score-design-v1:["OTIS Mock AIME 2024-2025","OTIS Mock AIME 2024-2025","1.1.6","model_extracted_exact_match"]
- Reasoning tier:
- low
- Documented comparison setup
- Publication cohort:
- epoch-score-design-v1:["OTIS Mock AIME 2024-2025","OTIS Mock AIME 2024-2025","1.1.6","model_extracted_exact_match"]
- Dataset type:
- not-reported-in-captured-run-csv
- Setup and model parameters
- Published reasoning tier: low. The capture does not report other inference parameters or a compute budget.
Independent evaluation. Epoch run alias: gemini-3.5-flash_low; Epoch model label: Gemini 3.5 Flash; Display label field: Gemini 3.5 Flash; Reasoning setting encoded in the alias: low. Task ID: OTIS Mock AIME 2024-2025. Task version: 1.1.6. Run ID: ig7gQ23WRms9SQLLNPtbTi. Started at 2026-08-06T22:34:49.000Z. Original task label: OTIS Mock AIME 2024-2025. The published row reports mean_score=0.8888888888888888 and stderr=0.047377936967913425. The CSV has no sample-count column, so none is inferred. Standard error is not presented as a confidence interval. Meta-score selection: Per model, the highest documented reasoning tier is selected within the same Epoch source test design, original task, version, and scorer, independently of the measured value. This derived score copy does not claim matched compute budgets; the capture publishes neither a common budget nor all inference parameters. The raw model-profile observation remains unchanged. Selected reasoning tier: low.
- Epoch AI benchmark run dataset snapshot
Test family: AA-AnalystAgent
Agents · Family index: 60.5 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-aa-analyst-agent
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.6 %
- Raw value:
- 45 %
- Metric:
- pass^5 reliability
- Version:
- AA cohort: 80 questions
- Participating models:
- 39
- Rank percentile within the cohort:
- 60.5 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-aa-analyst-agent:AA cohort: 80 questions:standalone:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-aa-analyst-agent","AA cohort: 80 questions","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Harness:
- Artificial Analysis Stirrup
- Agent:
- Stirrup analyst agent
- Tool policy:
- 100-turn cap; Python sandbox with source files, URL fetch, image viewing, and answer submission
- Attempts:
- pass^5, 5 independent attempts per question
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. 80 quantitative questions across 14 domains are run five independent times. pass^5 counts only questions answered correctly in all five attempts.. Harness: Artificial Analysis Stirrup. Agent: Stirrup analyst agent. Tool policy: 100-turn cap; Python sandbox with source files, URL fetch, image viewing, and answer submission. Attempts: pass^5, 5 independent attempts per question. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. 80 quantitative questions across 14 domains are run five independent times. pass^5 counts only questions answered correctly in all five attempts.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup analyst agent. Tool policy: 100-turn cap; Python sandbox with source files, URL fetch, image viewing, and answer submission. Attempts: pass^5, 5 independent attempts per question. Reasoning: reasoning enabled. Reasoning effort: High. Score display: 45.0%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AA-Briefcase
Agents · Family index: 50.0 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-aa-briefcase
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.6 %
- Raw value:
- 874.32 Elo
- Metric:
- AA-Briefcase Elo
- Version:
- v1.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 50.0 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-aa-briefcase:v1.1:v4.3.2:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-aa-briefcase","v1.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: High. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AA-LCR
Reasoning · Family index: 36.6 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-aa-lcr
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.02 %
- Raw value:
- 73.3333333333333 %
- Metric:
- Pass rate
- Version:
- v1.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 36.6 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-aa-lcr:v1.1:v4.3.2:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-aa-lcr","v1.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: High. Score display: 73.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: APEX-Agents-AA
Agents · Family index: 100.0 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-apex-agents-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.6 %
- Raw value:
- 47.0501474926254 %
- Metric:
- Pass@1 success rate
- Version:
- AA public cohort: 452 tasks
- Participating models:
- 27
- Rank percentile within the cohort:
- 100.0 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-apex-agents-aa:AA public cohort: 452 tasks:standalone:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-apex-agents-aa","AA public cohort: 452 tasks","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Harness:
- Artificial Analysis Stirrup
- Agent:
- Stirrup agent
- Tool policy:
- 200-turn cap; MCP-backed workplace tools in Archipelago
- Attempts:
- pass@1, averaged over 3 repeats per task
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. 452 tasks from the public 480-task release, with three repeats per task. A run passes only if every rubric criterion is satisfied; the leaderboard averages pass@1 across repeats.. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: 200-turn cap; MCP-backed workplace tools in Archipelago. Attempts: pass@1, averaged over 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. 452 tasks from the public 480-task release, with three repeats per task. A run passes only if every rubric criterion is satisfied; the leaderboard averages pass@1 across repeats.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: 200-turn cap; MCP-backed workplace tools in Archipelago. Attempts: pass@1, averaged over 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: High. Score display: 47.1%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AutomationBench-AA
Agents · Family index: 63.4 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-automationbench-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.6 %
- Raw value:
- 42.07973565652745 %
- Metric:
- Objectives completed without guardrail violations
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 63.4 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-automationbench-aa:unspecified:v4.3.2:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-automationbench-aa","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: High. Score display: 42.1%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: GDP.pdf
Reasoning · Family index: 75.6 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-gdp-pdf
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.02 %
- Raw value:
- 19.8 %
- Metric:
- All-pass rate
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 75.6 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-gdp-pdf:unspecified:v4.3.2:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-gdp-pdf","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: High. Score display: 19.8%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: GDPval-AA
Agents · Family index: 57.3 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-gdpval-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.6 %
- Raw value:
- 1,202.58 Elo
- Metric:
- GDPval-AA Elo
- Version:
- v2.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 57.3 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-gdpval-aa:v2.1:v4.3.2:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-gdpval-aa","v2.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: High. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: IFBench
General · Family index: 91.3 / 100Nominal family weight in the overall panel: 1.43 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-ifbench
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.43 %
- Raw value:
- 76.32653061224491 %
- Metric:
- IFBench prompt-level loose accuracy (pass@1)
- Version:
- AA test set: 294 questions
- Participating models:
- 70
- Rank percentile within the cohort:
- 91.3 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-ifbench:AA test set: 294 questions:standalone:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-ifbench","AA test set: 294 questions","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Tool policy:
- Single-turn answer evaluated with the official IFBench source code
- Attempts:
- pass@1, 5 repeats per question
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled. Reasoning effort: High. Score display: 76.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: ITBench-AA (SRE)
Agents · Family index: 57.1 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-itbench-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.6 %
- Raw value:
- 40.3483992467043 %
- Metric:
- Average precision at full recall
- Version:
- AA cohort: 59 tasks
- Participating models:
- 36
- Rank percentile within the cohort:
- 57.1 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-itbench-aa:AA cohort: 59 tasks:standalone:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-itbench-aa","AA cohort: 59 tasks","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Harness:
- Artificial Analysis Stirrup
- Agent:
- Stirrup agent
- Tool policy:
- Shell access to an offline Kubernetes incident snapshot
- Attempts:
- Average precision at full recall, 3 repeats per task
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: High. Score display: 40.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AA-Omniscience
Knowledge · Family index: 79.3 / 100Nominal family weight in the overall panel: 2.38 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-omniscience
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 2.38 %
- Raw value:
- 21.183333333333334 points
- Metric:
- AA-Omniscience Index
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 79.3 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-omniscience:unspecified:v4.3.2:reasoning:tokens=0:suffix=High
- Reasoning tier:
- High
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-omniscience","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: High
Exact AA configuration: Gemini 3.5 Flash (High) (release gemini-3-5-flash, row gemini-3-5-flash, UUID 0097ebf5-124f-42f6-9463-33b00e711f03, reasoning=true, source reasoning-token configuration budget=0, suffix=High). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0097ebf5-124f-42f6-9463-33b00e711f03/gemini-3-5-flash/gemini-3-5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: Gemini 3.5 Flash (High). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: High. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: EnigmaEval
Reasoning · Family index: 25.0 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
- Scale AI
Published configurations differ within this cohort.
- Measurement:
- enigma-eval
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.02 %
- Raw value:
- 25.41 %
- Metric:
- pass@1 accuracy
- Version:
- open-source release 2026-07-23
- Participating models:
- 5
- Rank percentile within the cohort:
- 25.0 %
- Retrieved:
- 2026-08-26
- Comparison cohort:
- scale-enigma-eval-2026-07-23
- Reasoning tier:
- high
- Documented comparison setup
- Publication cohort:
- scale-enigma-eval-2026-07-23
- Dataset type:
- public
- Attempts:
- pass@1
- Setup and model parameters
- Complex multimodal puzzles with exact answer matching and one attempt.. Attempts: pass@1. Reasoning effort: high
Complex multimodal puzzles with exact answer matching and one attempt.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Model snapshot: gemini-3.5-flash-high. Evaluator: Scale AI. Dataset type: public. Leaderboard date: 2026-07-23. Attempts: pass@1. Reasoning effort: high. Score display: 25.41% ± 2.45. Source ID: scale-ai. Evidence capture ID: curated-scale-ai-2026-08-26-42f1970146b9. Source payload hash: 42f1970146b9e570d91fd2312f41b9bc5b47ac0633cc41fcf2201ead2e816f1f. Parser version: curated-v1. Imported at: 2026-10-15
- Scale AI
Test family: Chess Puzzles
Reasoning · Family index: 54.8 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: SimpleQA Verified
Knowledge · Family index: 69.4 / 100Nominal family weight in the overall panel: 2.38 % · Coverage of this test family: 100 %
Test family: FrontierMath
Mathematics · Family index: 30.3 / 100Nominal family weight in the overall panel: 1.43 % · Coverage of this test family: 100 %
Test family: GPQA Diamond
Reasoning · Family index: 48.3 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: GAIA Reliability
Agents · Family index: 100.0 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
Test family: τ-bench Airline Clean
Agents · Family index: 68.8 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
Test family: Humanity’s Last Exam
Reasoning · Family index: 75.0 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 50 %
Test family: LMArena Code Arena WebDev
Coding · Family index: 27.8 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 100 %
Test family: LMArena Text, Style Control
General · Family index: 61.1 / 100Nominal family weight in the overall panel: 1.43 % · Coverage of this test family: 100 %
Test family: LMArena Vision, Style Control
Multimodal · Family index: 66.7 / 100Nominal family weight in the overall panel: 1.79 % · Coverage of this test family: 100 %
Test family: MMMU-Pro
Multimodal · Family index: 87.6 / 100Nominal family weight in the overall panel: 1.79 % · Coverage of this test family: 100 %
Test family: SWE-bench Verified
Coding · Family index: 63.6 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 50 %
Test family: Terminal-Bench
Coding · Family index: 61.1 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 100 %
Test family: Terminal-Bench Science
Coding · Family index: 38.7 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 50 %
Test family: Code Migration
Coding · Family index: 46.8 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 100 %
Test family: CorpFin v2
Finance · Family index: 70.3 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: EMB
Finance · Family index: 61.0 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: Finance Agent (v2)
Finance · Family index: 86.7 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: LiveCodeBench v6
Coding · Family index: 87.0 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 100 %
Test family: Harvey's Legal Agent Benchmark
Legal · Family index: 46.7 / 100Nominal family weight in the overall panel: 1.79 % · Coverage of this test family: 100 %
Test family: LegalBench
Legal · Family index: 55.1 / 100Nominal family weight in the overall panel: 1.79 % · Coverage of this test family: 100 %
Test family: Legal Research Bench
Legal · Family index: 46.7 / 100Nominal family weight in the overall panel: 1.79 % · Coverage of this test family: 100 %
Test family: MedCode
Health · Family index: 94.0 / 100Nominal family weight in the overall panel: 2.38 % · Coverage of this test family: 100 %
Test family: MedScribe
Health · Family index: 29.4 / 100Nominal family weight in the overall panel: 2.38 % · Coverage of this test family: 100 %
Test family: MMLU-Pro
Knowledge · Family index: 89.6 / 100Nominal family weight in the overall panel: 2.38 % · Coverage of this test family: 100 %
Test family: MortgageTax
Finance · Family index: 71.2 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: ProgramBench
Coding · Family index: 24.5 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 100 %
Test family: ProofBench
Mathematics · Family index: 37.2 / 100Nominal family weight in the overall panel: 1.43 % · Coverage of this test family: 100 %
Test family: SAGE
Education · Family index: 75.9 / 100Nominal family weight in the overall panel: 3.57 % · Coverage of this test family: 100 %
Test family: SkillsBench
Agents · Family index: 55.2 / 100Nominal family weight in the overall panel: 0.6 % · Coverage of this test family: 100 %
Test family: Public Benefits Bench v1.1
Productivity · Family index: 34.2 / 100Nominal family weight in the overall panel: 3.57 % · Coverage of this test family: 100 %
Test family: Tax Agent Bench
Finance · Family index: 43.6 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: TaxEval v2
Finance · Family index: 67.1 / 100Nominal family weight in the overall panel: 1.02 % · Coverage of this test family: 100 %
Test family: Vibe Code Bench v1.1
Coding · Family index: 47.2 / 100Nominal family weight in the overall panel: 0.48 % · Coverage of this test family: 100 %
Benchmark sources at a glance
Each evaluator measures different capabilities under its own conditions. Test series, composite indices and their components stay separate. The detailed results below retain their scales, configurations and source dates.
| Evaluator | Test series | Composite indices | Recorded active measurements |
|---|---|---|---|
| Artificial Analysis | 13 | 1 | 24 |
| Vals AI | 29 | 2 | 217 |
| LMArena | 3 | 0 | 84 |
| Epoch AI | 6 | 0 | 9 |
| Princeton HAL | 2 | 0 | 12 |
| Scale AI Evals | 1 | 0 | 1 |
Coverage is not a quality ranking. Overlapping tests and multiple settings do not count as independent experiments. Vendor claims, historical and disputed results are excluded from this overview and remain labelled in the detailed evidence.
Page 1 of 31
Published benchmarks
Agent tasks
GAIA Reliability
Reliability evaluation on 165 GAIA tasks with repetitions and perturbations. Accuracy, reliability, consistency, predictability, robustness, and safety are separate metrics.
| Sub-task | Gemini 3.5 Flash (This model) | Claude Opus 4.5 | Claude Opus 4.7 | Gemini 3.1 Pro Preview | GPT-4o mini | Gemini 2.5 Pro | Gemini 2.5 Flash | o1 | GPT-5.5 |
|---|---|---|---|---|---|---|---|---|---|
| Consistency | 0.75 | 0.81 (best value in this row) | 0.81 (best value in this row) | 0.76 | 0.73 | 0.72 | 0.7 | 0.68 | 0.61 |
| Predictability | 0.8 | 0.84 (best value in this row) | 0.73 | 0.78 | 0.58 | 0.71 | 0.64 | 0.68 | 0.8 |
| Reliability | 0.84 | 0.85 (best value in this row) | 0.84 | 0.82 | 0.76 | 0.78 | 0.74 | 0.79 | 0.79 |
| Robustness | 0.96 | 0.91 | 0.99 | 0.94 | 0.97 | 0.91 | 0.87 | 1 (best value in this row) | 0.95 |
| Safety | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) | 1 (best value in this row) |
Unit: ratinghigher is betterAs of August 6, 2026Source: Princeton HALbars scaled within each row, not across rows
GAIA Reliability
Accuracy
Reliability evaluation on 165 GAIA tasks with repetitions and perturbations. Accuracy, reliability, consistency, predictability, robustness, and safety are separate metrics.
higher is betterAs of August 6, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- hal-gaia-reliability-hal-reliability-overall
- Metric
- exact-match accuracy
- Scale
- 0-100
- Participants
- 9
- Methodology and context
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-04b16718a8e7. Source payload hash: 04b16718a8e7a8a04fb0ad6ecc2bb7b0afcf74d448c4c1076907ec22e5cb331c. Parser version: curated-v1. Imported at: 2026-10-15
τ-bench Airline Clean
26 cleaned customer-service tasks involving dialogue, policy compliance, and mutating backend tools. The cleaned subset removes 24 broken or ambiguous tasks. Accuracy and reliability dimensions measure different properties.
| Sub-task | Gemini 3.5 Flash (This model) | Claude Opus 4.7 | Claude Opus 4.5 | GPT-5.5 | Gemini 3.1 Pro Preview | Gemini 2.5 Flash | Gemini 2.5 Pro | GPT-4o mini | o1 |
|---|---|---|---|---|---|---|---|---|---|
| Consistency | 0.78 | 0.88 (best value in this row) | 0.84 | 0.84 | 0.81 | 0.75 | 0.74 | 0.74 | 0.71 |
| Predictability | 0.82 | 0.87 (best value in this row) | 0.85 | 0.84 | 0.83 | 0.6 | 0.78 | 0.35 | 0.74 |
| Reliability | 0.86 | 0.89 (best value in this row) | 0.88 | 0.89 (best value in this row) | 0.86 | 0.72 | 0.81 | 0.67 | 0.81 |
| Robustness | 0.97 | 0.91 | 0.97 | 0.97 | 0.94 | 0.82 | 0.93 | 0.93 | 0.98 (best value in this row) |
| Safety | 0.99 (best value in this row) | 0.94 | 0.98 | 0.96 | 0.97 | 0.9 | 0.92 | 0.85 | 0.91 |
Unit: ratinghigher is betterAs of August 5, 2026Source: Princeton HALbars scaled within each row, not across rows
τ-bench Airline Clean
Accuracy
26 cleaned customer-service tasks involving dialogue, policy compliance, and mutating backend tools. The cleaned subset removes 24 broken or ambiguous tasks. Accuracy and reliability dimensions measure different properties.
higher is betterAs of August 5, 2026Scale: 0% - 100%
Methodology and context
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-overall
- Metric
- task success rate
- Scale
- 0-100
- Participants
- 9
- Methodology and context
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-8023c329706f. Source payload hash: 8023c329706fe5ccb7162223319f0987121f8e12c42400e3949a4c00c435852e. Parser version: curated-v1. Imported at: 2026-10-15
Recorded measurement details
GAIA Reliability: Consistency (Show measurement, test conditions, and source)
- Source value
- 0.75
- Score
- 0.75
- Metric
- aggregate consistency
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- consistency
- Task label
- Consistency
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-consistency
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-e688b934dbf7. Source payload hash: e688b934dbf7159457b273d9c4c32f9418a6bfbb6419b750eec843e6095f0599. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Predictability (Show measurement, test conditions, and source)
- Source value
- 0.8
- Score
- 0.8
- Metric
- aggregate predictability
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- predictability
- Task label
- Predictability
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-predictability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-13cadabbccdb. Source payload hash: 13cadabbccdb99ef7953b9be9486cc0fe4ad71fb44cde50a191447afb3158dde. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Reliability (Show measurement, test conditions, and source)
- Source value
- 0.84
- Score
- 0.84
- Metric
- aggregate reliability
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- reliability
- Task label
- Reliability
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-reliability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-b28b129b5b48. Source payload hash: b28b129b5b480c227369c491659d296e03895268b245f6fff4978c8ca4b9cc20. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Robustness (Show measurement, test conditions, and source)
- Source value
- 0.96
- Score
- 0.96
- Metric
- aggregate robustness
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- robustness
- Task label
- Robustness
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-robustness
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-394a0e4ccb07. Source payload hash: 394a0e4ccb076b62c4939a6bc2269eca5039c1e6a43c0092ab25b58ca9532d25. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Safety (Show measurement, test conditions, and source)
- Source value
- 1
- Score
- 1
- Metric
- aggregate safety
- Unit
- rating
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- safety
- Task label
- Safety
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-safety
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-44e5d794585c. Source payload hash: 44e5d794585c8bdefa5473d4455ef7037858293c5fb4a2f5c3ccf7f20b14fdab. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
GAIA Reliability: Accuracy (Show measurement, test conditions, and source)
- Source value
- 79.2
- Score
- 79.2
- Metric
- exact-match accuracy
- Unit
- %
- Benchmark version
- final leak-cleaned public validation set
- Category
- agentic
- Task
- overall
- Task label
- Accuracy
- Direction
- Higher is better
- Comparison cohort
- hal-gaia-reliability-hal-reliability-overall
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-06
- Methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-06-04b16718a8e7. Source payload hash: 04b16718a8e7a8a04fb0ad6ecc2bb7b0afcf74d448c4c1076907ec22e5cb331c. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 165 GAIA tasks with repeated runs and perturbations. HAL does not publish cost for this evaluation.
τ-bench Airline Clean: Consistency (Show measurement, test conditions, and source)
- Source value
- 0.78
- Score
- 0.78
- Metric
- aggregate consistency
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- consistency
- Task label
- Consistency
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-consistency
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-1e9fbd126e3d. Source payload hash: 1e9fbd126e3d45828898b17c1e33cefe3322294e52407c0af30d6606ce4dc5e8. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Predictability (Show measurement, test conditions, and source)
- Source value
- 0.82
- Score
- 0.82
- Metric
- aggregate predictability
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- predictability
- Task label
- Predictability
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-predictability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-30b92ab084c6. Source payload hash: 30b92ab084c6aa7b2624e97268269f1db56fc3616930e3953ac6d042884de78b. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Reliability (Show measurement, test conditions, and source)
- Source value
- 0.86
- Score
- 0.86
- Metric
- aggregate reliability
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- reliability
- Task label
- Reliability
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-reliability
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-1fcd61644789. Source payload hash: 1fcd616447897befa1c501195920713f506aff42a09dd11883c313e0d47c26ae. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Robustness (Show measurement, test conditions, and source)
- Source value
- 0.97
- Score
- 0.97
- Metric
- aggregate robustness
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- robustness
- Task label
- Robustness
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-robustness
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-9214146c7ee0. Source payload hash: 9214146c7ee0b1c06426d1aa49cab89ba1338cbb4a6e19308837436f730a7b18. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Safety (Show measurement, test conditions, and source)
- Source value
- 0.99
- Score
- 0.99
- Metric
- aggregate safety
- Unit
- rating
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- safety
- Task label
- Safety
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-safety
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-f62581e02340. Source payload hash: f62581e0234042503a70b140cf27a898d1c3ebcce6ccacd3ba2040a996f12499. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
τ-bench Airline Clean: Accuracy (Show measurement, test conditions, and source)
- Source value
- 80.8
- Score
- 80.8
- Metric
- task success rate
- Unit
- %
- Benchmark version
- 26-task cleaned subset
- Category
- agentic
- Task
- overall
- Task label
- Accuracy
- Direction
- Higher is better
- Comparison cohort
- hal-tau-airline-clean-hal-reliability-overall
- Source type
- independent-evaluator
- Evaluator
- Princeton HAL
- Status
- active
- Retrieved at
- 2026-08-05
- Methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.. Source model ID: curated/gemini-3.5-flash. Source model name: Gemini 3.5 Flash. Provider reported by source: Google. Evaluator: Princeton HAL. Dataset type: public. Harness: HAL Reliability. Agent: HAL Reliability. Source ID: princeton-hal. Evidence capture ID: curated-princeton-hal-2026-08-05-8023c329706f. Source payload hash: 8023c329706fe5ccb7162223319f0987121f8e12c42400e3949a4c00c435852e. Parser version: curated-v1. Imported at: 2026-10-15
- Comparison methodology
- 26 cleaned airline tasks. Princeton HAL recomputed every metric from scratch on this subset.
Page 1 of 31
Our tests
SVG short test
One SVG prompt and fixed output settings. Captures are shown only with 0 reported reasoning tokens or documented lack of reasoning controls. Identical test conditions do not make the models' internal compute identical.
Earlier tests
Pelican on a bicycle
A community classic for free-form SVG drawing.
Gemini 3.5 Flash
The response reached the token limit. No complete recording is available.
Reasoning (requested): high
Current attempt
- Run date
- Oct 6, 2026
- API provider
- openrouter
- Requested API model
- google/gemini-3.5-flash
- API model reported in response
- google/gemini-3.5-flash
- Reasoning setting
- high
- Pinned endpoint
- Google AI Studio | google/gemini-3.5-flash-20260519
- Temperature
- Provider default
- Token limit including reasoning
- 8,192 tokens
- Test protocol
- community-visual-tests-v1
- Technical checks
- Failed
- Response time
- 31.7 s
Not yet visually reviewed. A successful recording does not confirm correct physics or full compliance with the prompt.
Earlier attempts (3)
- Oct 2, 2026API request failedReasoning setting: high
- Oct 1, 2026API request failedReasoning setting: high
- Sep 7, 2026API request failedReasoning setting: high
Prompt and test conditions
Generate an SVG of a pelican riding a bicycle
Token limit including reasoning: 8,192 tokens
Task origin (Simon Willison)Popularity
37.9 / 100
Specifications and access
| Specification | Value and source |
|---|---|
| Weights | ProprietarySource |
| Access | APISource |
| Context window | 1,048,576 tokensSource |
| Notes | Released May 19, 2026. Google does not publish its parameter count, architecture, or knowledge cutoff.Source |
| API model ID | gemini-3.5-flashSource |
| Source status | stableSource |
| Maximum output | 65,536 tokensSource |
| Input | Text, Image, Video, Audio, pdfSource |
| Output | TextSource |
| Reasoning mode | low, medium, highSource |
| Tool calling | YesSource |
| Structured output | YesSource |
| Release | 2026-05-19Source |
| Capabilities | function calling, structured outputs, search grounding, Google Maps grounding, URL context, code execution, file search, computer useSource |
| General availability | 2026-05-19 (General availability (GA).)Source |