Qwen 3.5 397B A17B
Language modelActive
Qwen 3.5 397B A17B
Alibaba
- Released
- February 15, 2026
- Data date
- September 8, 2026
- Successor
- Qwen 3.8 2.4T A95B
- Benchmark-Meta-Score
- Results from one source only
- Speed
- 86.9 tokens/sOutput throughput
- Price
- Input · Output / 1M tokens
- Input
- Text, Image, Video
- Output
- Text
Qwen 3.5 397B A17B is the largest open Qwen3.5 checkpoint, combining early image-text fusion with a sparse Mixture of Experts. Seventeen billion of its 397 billion parameters are active.
The repository supports 262,144 native context tokens and extension to 1,010,000. Qwen names Qwen 3.8 2.4T A95B as a later open-weight expansion built on this foundation. For managed inference, Qwen3.5-Plus is a separate configuration with 1M context by default and built-in tools.
Specifications and access
| Specification | Value and source |
|---|---|
| Weights | Open weightsSource |
| Access | API, LocalSource |
| Parameters | 397BSource |
| Active parameters | 17BSource |
| Context window | 262,144 tokensSource |
| Weights repository | Qwen/Qwen3.5-397B-A17BSource |
| License | Apache-2.0Source |
| Architecture | hybrid Gated DeltaNet + Gated Attention; dense or sparse MoE with vision encoderSource |
| Input | Text, Image, VideoSource |
| Output | TextSource |
| Reasoning mode | thinking by default; switchable non-thinking/instruct modeSource |
| Tool calling | YesSource |
| Structured output | YesSource |
| Native context window | 262,144 tokensSource |
| Maximum context window | 1,010,000 tokensSource |
| Parameters | 397 BSource |
| Active parameters | 17 BSource |
| Languages | 201 languages and dialectsSource |
| Repository created | 2026-02-16 (Repository creation date is a precise publication proxy, but may differ from the public announcement date or an individual checkpoint's first availability.)Source |
Benchmark meta-analysis
Overall and area scores
These Gradually indexes place selected published measurements on a 0 to 100 scale. They are not accuracy percentages and do not guarantee results in your application.
Retrieval dates of the sources used 2026-10-19 - 2026-10-19
Overall index
Not enough independent results for a score yet
An overall score needs results from at least two sources.
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Test series
- 10
- Test families
- 10 / 65
- Sources
- 1
- Share of the fixed test panel (informative)
- 12 %
Coding
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.1
- 95% interval
- 0.0 to 0.7
- Test series
- 1
- Test families
- 1 / 15
- Sources
- 1
- Share of the fixed test panel (informative)
- 3 %
Reasoning
Index 6.8 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 3.4
- 95% interval
- 2.3 to 16.3
- Test series
- 3
- Test families
- 3 / 7
- Sources
- 1
- Share of the fixed test panel (informative)
- 36 %
Knowledge
Index below 1 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 0.2
- 95% interval
- 0.0 to 1.1
- Test series
- 1
- Test families
- 1 / 3
- Sources
- 1
- Share of the fixed test panel (informative)
- 33 %
Agents
Index 1.4 / 100
provisional
Published configurations differ within this comparison cohort. Treat the index as provisional.
- Standard error
- ± 1.3
- 95% interval
- 0.2 to 6.9
- Test series
- 5
- Test families
- 5 / 12
- Sources
- 1
- Share of the fixed test panel (informative)
- 42 %
Documented method · Included test families and measurements
Method version: benchmark-meta-score-v3
The index includes active measurements from reviewed primary sources when at least five models are compared in the same documented comparison slice across dataset, harness, agent, tools, and trial count. Each benchmark series and evaluator contributes one shared setup slice from the newest capture. When one source and series include several explicitly documented reasoning tiers, the highest documented tier (none through max) is selected, never the best result value. The explicit source label “Non-reasoning” is treated as none. Unknown or ambiguous tiers exclude the model from that series. Within each series, percentages and rates are logit-transformed, other units stay unchanged, and all values are then standardized. A one-factor model then estimates one shared strength per model and, per series, how sharply it separates the models. Area scores move away from that strength only as far as the model's own results in the area support. The overall score weights the core areas at 75% and finance, law and health at 25%. It requires at least eight test series from two sources and three areas. The 0 to 100 scale refers to a fixed reference group of 29 language models whose average sits at 50.
Published model-specific parameters can differ. This is not a controlled experiment; comparisons with such differences are marked provisional.
Unmeasured series are not entered as zero. Where a model has no own results in a core area, its overall strength stands in, and the note “estimated without own results in …” names the area. Fewer test series lead to a larger standard error.
The 95% interval follows from the standard error of 200 recalculations with randomly redrawn test families. It describes how precisely the recorded test series determine the value. It is no prediction for future tasks or for later recalculations with other participants.
The source count is not a count of independent experiments.
OECD handbook on composite indicatorsAll 10 test families and 10 of 10 raw measurements are listed here. The complete calculation for every model is available in the JSON export.
Test family: AA-Briefcase
Agents · Mean rank percentile of the family: 31.7 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-aa-briefcase
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 562.4 Elo
- Metric:
- AA-Briefcase Elo
- Version:
- v1.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 31.7 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-aa-briefcase:v1.1:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-aa-briefcase","v1.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AA-LCR
Reasoning · Mean rank percentile of the family: 48.8 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-aa-lcr
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.53 %
- Raw value:
- 77.3333333333333 %
- Metric:
- Pass rate
- Version:
- v1.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 48.8 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-aa-lcr:v1.1:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-aa-lcr","v1.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 77.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: APEX-Agents-AA
Agents · Mean rank percentile of the family: 34.6 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-apex-agents-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 15.339233038348098 %
- Metric:
- Pass@1 success rate
- Version:
- AA public cohort: 452 tasks
- Participating models:
- 27
- Rank percentile within the cohort:
- 34.6 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-apex-agents-aa:AA public cohort: 452 tasks:standalone:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-apex-agents-aa","AA public cohort: 452 tasks","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Harness:
- Artificial Analysis Stirrup
- Agent:
- Stirrup agent
- Tool policy:
- 200-turn cap; MCP-backed workplace tools in Archipelago
- Attempts:
- pass@1, averaged over 3 repeats per task
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 452 tasks from the public 480-task release, with three repeats per task. A run passes only if every rubric criterion is satisfied; the leaderboard averages pass@1 across repeats.. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: 200-turn cap; MCP-backed workplace tools in Archipelago. Attempts: pass@1, averaged over 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 452 tasks from the public 480-task release, with three repeats per task. A run passes only if every rubric criterion is satisfied; the leaderboard averages pass@1 across repeats.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: 200-turn cap; MCP-backed workplace tools in Archipelago. Attempts: pass@1, averaged over 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 15.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AutomationBench-AA
Agents · Mean rank percentile of the family: 41.5 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-automationbench-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 6.886670023070867 %
- Metric:
- Objectives completed without guardrail violations
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 41.5 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-automationbench-aa:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-automationbench-aa","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 6.9%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: GDP.pdf
Reasoning · Mean rank percentile of the family: 55.5 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-gdp-pdf
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 1.53 %
- Raw value:
- 12.2 %
- Metric:
- All-pass rate
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 55.5 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-gdp-pdf:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-gdp-pdf","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 12.2%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: GDPval-AA
Agents · Mean rank percentile of the family: 32.9 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-gdpval-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 792.97 Elo
- Metric:
- GDPval-AA Elo
- Version:
- v2.1
- Participating models:
- 83
- Rank percentile within the cohort:
- 32.9 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-gdpval-aa:v2.1:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-gdpval-aa","v2.1","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: ITBench-AA (SRE)
Agents · Mean rank percentile of the family: 28.6 / 100Family weight in the fixed test panel (informative): 0.89 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-itbench-aa
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 0.89 %
- Raw value:
- 34.0866290018832 %
- Metric:
- Average precision at full recall
- Version:
- AA cohort: 59 tasks
- Participating models:
- 36
- Rank percentile within the cohort:
- 28.6 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-itbench-aa:AA cohort: 59 tasks:standalone:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-itbench-aa","AA cohort: 59 tasks","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Harness:
- Artificial Analysis Stirrup
- Agent:
- Stirrup agent
- Tool policy:
- Shell access to an offline Kubernetes incident snapshot
- Attempts:
- Average precision at full recall, 3 repeats per task
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 34.1%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: AA-Omniscience
Knowledge · Mean rank percentile of the family: 25.6 / 100Family weight in the fixed test panel (informative): 3.57 % · Coverage of this test family: 100 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-omniscience
- Series weight within the test family:
- 100 %
- Nominal contribution to the overall panel:
- 3.57 %
- Raw value:
- -30.75 points
- Metric:
- AA-Omniscience Index
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 25.6 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-omniscience:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-omniscience","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: Humanity’s Last Exam
Reasoning · Mean rank percentile of the family: 40.2 / 100Family weight in the fixed test panel (informative): 1.53 % · Coverage of this test family: 50 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-humanitys-last-exam
- Series weight within the test family:
- 50 %
- Nominal contribution to the overall panel:
- 0.77 %
- Raw value:
- 28.962001853568097 %
- Metric:
- Text-only pass@1 accuracy
- Version:
- unspecified
- Participating models:
- 83
- Rank percentile within the cohort:
- 40.2 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-humanitys-last-exam:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-humanitys-last-exam","unspecified","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 29.0%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Test family: Terminal-Bench
Coding · Mean rank percentile of the family: 17.7 / 100Family weight in the fixed test panel (informative): 0.71 % · Coverage of this test family: 50 %
- Artificial Analysis
Published configurations differ within this cohort.
- Measurement:
- artificial-analysis-terminal-bench-4
- Series weight within the test family:
- 50 %
- Nominal contribution to the overall panel:
- 0.36 %
- Raw value:
- 0 %
- Metric:
- Task success
- Version:
- 4.0
- Participating models:
- 83
- Rank percentile within the cohort:
- 17.7 %
- Retrieved:
- 2026-10-19
- Comparison cohort:
- artificial-analysis-terminal-bench-4:4.0:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Reasoning tier:
- Reasoning
- Documented comparison setup
- Publication cohort:
- ["artificial-analysis-terminal-bench-4","4.0","aa-model-page-2026-10-19-8a022369738c"]
- Dataset type:
- artificial-analysis-evaluation
- Setup and model parameters
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 0.0%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Artificial Analysis
Benchmark sources at a glance
Each evaluator measures different capabilities under its own conditions. Test series, composite indices and their components stay separate. The detailed results below retain their scales, configurations and source dates.
| Evaluator | Test series | Composite indices | Recorded active measurements |
|---|---|---|---|
| Artificial Analysis | 12 | 1 | 26 |
Coverage is not a quality ranking. Overlapping tests and multiple settings do not count as independent experiments. Vendor claims, historical and disputed results are excluded from this overview and remain labelled in the detailed evidence.
7 measurements from these sources still lack a verified test-family identity and do not increase the test-series count.
Page 1 of 10
Measurements without matching peer values
These measurements have no matching peer values under the same test conditions. Their original values and sources remain available here.
AA-LCR: Overall (Show measurement, test conditions, and source)
- Source value
- 77.3333333333333
- Score
- 77.3333333333333
- Metric
- Pass rate
- Unit
- %
- Benchmark version
- v1.1
- Category
- reasoning
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-aa-lcr:v1.1:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 77.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
APEX-Agents-AA: Overall (Show measurement, test conditions, and source)
- Source value
- 15.339233038348098
- Score
- 15.339233038348098
- Metric
- Pass@1 success rate
- Unit
- %
- Benchmark version
- AA public cohort: 452 tasks
- Category
- agentic
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-apex-agents-aa:AA public cohort: 452 tasks:standalone:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 452 tasks from the public 480-task release, with three repeats per task. A run passes only if every rubric criterion is satisfied; the leaderboard averages pass@1 across repeats.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: 200-turn cap; MCP-backed workplace tools in Archipelago. Attempts: pass@1, averaged over 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 15.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 452 tasks from the public 480-task release, with three repeats per task. A run passes only if every rubric criterion is satisfied; the leaderboard averages pass@1 across repeats.. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: 200-turn cap; MCP-backed workplace tools in Archipelago. Attempts: pass@1, averaged over 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning
- Context
- Two Investment Banking worlds are excluded because they depend on external runtime APIs.
AutomationBench-AA: Overall (Show measurement, test conditions, and source)
- Source value
- 6.886670023070867
- Score
- 6.886670023070867
- Metric
- Objectives completed without guardrail violations
- Unit
- %
- Benchmark version
- unspecified
- Category
- agentic
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-automationbench-aa:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 6.9%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
CritPt: Overall (Show measurement, test conditions, and source)
- Source value
- 1.70612244897959
- Score
- 1.70612244897959
- Metric
- Score
- Unit
- %
- Benchmark version
- unspecified
- Category
- reasoning
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-critpt:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- disputed
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 1.7%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
- Context
- Artificial Analysis marks CritPt as under review; dataset revisions may change results.. The source dataset marks this result as disputed. It remains visible as documented evidence, but is not reliable comparative proof.
GDP.pdf: Overall (Show measurement, test conditions, and source)
- Source value
- 12.2
- Score
- 12.2
- Metric
- All-pass rate
- Unit
- %
- Benchmark version
- unspecified
- Category
- reasoning
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-gdp-pdf:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 12.2%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
Humanity's Last Exam: Overall (Show measurement, test conditions, and source)
- Source value
- 28.962001853568097
- Score
- 28.962001853568097
- Metric
- Text-only pass@1 accuracy
- Unit
- %
- Benchmark version
- unspecified
- Category
- reasoning
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-humanitys-last-exam:unspecified:v4.3.2:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 29.0%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption.. Reasoning: reasoning enabled. Reasoning effort: Reasoning
- Context
- Artificial Analysis describes this comparable run as 2,158 text-only questions, not the full 2,500-question dataset.
IFBench: Overall (Show measurement, test conditions, and source)
- Source value
- 78.7755102040816
- Score
- 78.7755102040816
- Metric
- IFBench prompt-level loose accuracy (pass@1)
- Unit
- %
- Benchmark version
- AA test set: 294 questions
- Category
- general
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-ifbench:AA test set: 294 questions:standalone:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 78.8%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled. Reasoning effort: Reasoning
- Context
- The often cited 58 refers to verifiable out-of-domain constraint types, not the number of questions in AA's test run.
IFBench: Overall (Show measurement, test conditions, and source)
- Source value
- 51.632653061224495
- Score
- 51.632653061224495
- Metric
- IFBench prompt-level loose accuracy (pass@1)
- Unit
- %
- Benchmark version
- AA test set: 294 questions
- Category
- general
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-ifbench:AA test set: 294 questions:standalone:non-reasoning:tokens=0:suffix=Non-reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Non-reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b-non-reasoning, UUID 30ef2a79-e800-4165-9f13-2a338f120db7, reasoning=false, source reasoning-token configuration budget=0, suffix=Non-reasoning). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Source model ID: artificial-analysis/30ef2a79-e800-4165-9f13-2a338f120db7/qwen3-5-397b-a17b/qwen3-5-397b-a17b-non-reasoning. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Non-reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: non-reasoning. Reasoning effort: Non-reasoning. Score display: 51.6%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Non-reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b-non-reasoning, UUID 30ef2a79-e800-4165-9f13-2a338f120db7, reasoning=false, source reasoning-token configuration budget=0, suffix=Non-reasoning). This configured value is not measured token consumption. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: non-reasoning. Reasoning effort: Non-reasoning
- Context
- The often cited 58 refers to verifiable out-of-domain constraint types, not the number of questions in AA's test run.
ITBench-AA (SRE): Overall (Show measurement, test conditions, and source)
- Source value
- 34.0866290018832
- Score
- 34.0866290018832
- Metric
- Average precision at full recall
- Unit
- %
- Benchmark version
- AA cohort: 59 tasks
- Category
- agentic
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-itbench-aa:AA cohort: 59 tasks:standalone:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 34.1%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. 59 Kubernetes incident tasks, 40 public and 19 private, with three repeats per task. A run scores zero if it misses any true root cause; otherwise it receives its precision at full recall. This is the SRE slice of ITBench.. Harness: Artificial Analysis Stirrup. Agent: Stirrup agent. Tool policy: Shell access to an offline Kubernetes incident snapshot. Attempts: Average precision at full recall, 3 repeats per task. Reasoning: reasoning enabled. Reasoning effort: Reasoning
- Context
- This AA implementation covers Site Reliability Engineering only, not ITBench's other CISO and FinOps domains.
MMMU-Pro: Overall (Show measurement, test conditions, and source)
- Source value
- 77.28323699421969
- Score
- 77.28323699421969
- Metric
- pass@1 accuracy
- Unit
- %
- Benchmark version
- AA cohort: 1,730 questions
- Category
- multimodal
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-mmmu-pro:AA cohort: 1,730 questions:standalone:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. The AA run covers 1,730 questions with ten answer options. Answers are extracted by regex and scored pass@1.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Ten-option multiple choice; regex answer extraction. Attempts: pass@1, one run per question. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 77.3%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. The AA run covers 1,730 questions with ten answer options. Answers are extracted by regex and scored pass@1.. Tool policy: Ten-option multiple choice; regex answer extraction. Attempts: pass@1, one run per question. Reasoning: reasoning enabled. Reasoning effort: Reasoning
- Context
- The benchmark page describes 3,460 questions in the full MMMU-Pro dataset; AA's methodology specifies a 1,730-question run.
MMMU-Pro: Overall (Show measurement, test conditions, and source)
- Source value
- 52.6589595375723
- Score
- 52.6589595375723
- Metric
- pass@1 accuracy
- Unit
- %
- Benchmark version
- AA cohort: 1,730 questions
- Category
- multimodal
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-mmmu-pro:AA cohort: 1,730 questions:standalone:non-reasoning:tokens=0:suffix=Non-reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Non-reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b-non-reasoning, UUID 30ef2a79-e800-4165-9f13-2a338f120db7, reasoning=false, source reasoning-token configuration budget=0, suffix=Non-reasoning). This configured value is not measured token consumption. The AA run covers 1,730 questions with ten answer options. Answers are extracted by regex and scored pass@1.. Source model ID: artificial-analysis/30ef2a79-e800-4165-9f13-2a338f120db7/qwen3-5-397b-a17b/qwen3-5-397b-a17b-non-reasoning. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Non-reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Ten-option multiple choice; regex answer extraction. Attempts: pass@1, one run per question. Reasoning: non-reasoning. Reasoning effort: Non-reasoning. Score display: 52.7%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Non-reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b-non-reasoning, UUID 30ef2a79-e800-4165-9f13-2a338f120db7, reasoning=false, source reasoning-token configuration budget=0, suffix=Non-reasoning). This configured value is not measured token consumption. The AA run covers 1,730 questions with ten answer options. Answers are extracted by regex and scored pass@1.. Tool policy: Ten-option multiple choice; regex answer extraction. Attempts: pass@1, one run per question. Reasoning: non-reasoning. Reasoning effort: Non-reasoning
- Context
- The benchmark page describes 3,460 questions in the full MMMU-Pro dataset; AA's methodology specifies a 1,730-question run.
AA-Omniscience Accuracy: Overall (Show measurement, test conditions, and source)
- Source value
- 30.8
- Score
- 30.8
- Metric
- AA-Omniscience Accuracy
- Unit
- %
- Benchmark version
- AA Omniscience diagnostic
- Category
- knowledge
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- artificial-analysis-omniscience-accuracy:AA Omniscience diagnostic:standalone:reasoning:tokens=3186:suffix=Reasoning
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-10-19
- Methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. Proportion of all questions answered correctly, regardless of whether the model answers or abstains on other questions.. Source model ID: artificial-analysis/0e66bae9-41f1-42fc-9276-ce8cb6f72919/qwen3-5-397b-a17b/qwen3-5-397b-a17b. Source model name: Qwen 3.5 397B A17B. Provider reported by source: Alibaba. Model snapshot: Qwen3.5 397B A17B (Reasoning). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Attempts: AA-reported aggregate; diagnostic-specific run count is not stated on the leaderboard. Reasoning: reasoning enabled. Reasoning effort: Reasoning. Score display: 30.8%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-10-19-8a022369738c. Source payload hash: 8a022369738cbe2a0e2f9bf486a2882865dbb0c6f1396dfec84059a802560e3c. Parser version: artificial-analysis-rsc-models-v2. Imported at: 2026-10-19
- Comparison methodology
- Exact AA configuration: Qwen3.5 397B A17B (Reasoning) (release qwen3-5-397b-a17b, row qwen3-5-397b-a17b, UUID 0e66bae9-41f1-42fc-9276-ce8cb6f72919, reasoning=true, source reasoning-token configuration budget=3186, suffix=Reasoning). This configured value is not measured token consumption. Proportion of all questions answered correctly, regardless of whether the model answers or abstains on other questions.. Attempts: AA-reported aggregate; diagnostic-specific run count is not stated on the leaderboard. Reasoning: reasoning enabled. Reasoning effort: Reasoning
- Context
- This is the separately published accuracy diagnostic, not the AA-Omniscience Index. AA identifies higher values as better.
Page 1 of 10
Our tests
SVG short test
One SVG prompt and fixed output settings. Captures are shown only with 0 reported reasoning tokens or documented lack of reasoning controls. Identical test conditions do not make the models' internal compute identical.
Earlier tests
Pelican on a bicycle
A community classic for free-form SVG drawing.
Qwen 3.5 397B A17B
Reasoning (requested): high
Current attempt
- Run date
- Oct 1, 2026
- API provider
- openrouter
- Requested API model
- qwen/qwen3.5-397b-a17b
- API model reported in response
- qwen/qwen3.5-397b-a17b
- Reasoning setting
- high
- Pinned endpoint
- Alibaba | qwen/qwen3.5-397b-a17b-20260216
- Temperature
- Provider default
- Token limit including reasoning
- 8,192 tokens
- Test protocol
- community-visual-tests-v1
- Technical checks
- Passed, not a quality rating
- Response time
- 35.5 s
Visually reviewed. This individual test does not establish overall model quality.
Prompt and test conditions
Generate an SVG of a pelican riding a bicycle
Token limit including reasoning: 8,192 tokens
Task origin (Simon Willison)Popularity
46.9 / 100