Mistral AILanguage model
Devstral Small
Devstral Small has no overall index value yet because there are too few independent benchmark results.
PopularityRank 251 of 785
42out of 100same as last month
based on two data groups, mostly usage
Summary
Devstral Small is the 24B coding-agent checkpoint released in July 2025. It understands codebases, calls functions, and processes 128,000 tokens of context. Alongside Mistral’s function format, it supports XML-based tool descriptions.
Devstral Small 2 continues the line with a larger context window and multimodal input.
All benchmark results
Every published result with its source, test setup and comparison models.
Show all 23 published benchmark results
Page 1 of 2
Measurements without matching peer values
These measurements have no matching peer values under the same test conditions. Their original values and sources remain available here.
CritPt v2: Overall (Show measurement, test conditions, and source)
- Source value
- 0
- Score
- 0
- Metric
- pass@1 accuracy
- Unit
- %
- Benchmark version
- v2
- Category
- reasoning
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- disputed
- Retrieved at
- 2026-11-02
- Methodology
- Exact AA configuration: Devstral Small (Jul '25) (release devstral-small, row devstral-small, UUID 9eae4ec4-61b8-48bc-9843-3edd506ae933, reasoning=false, effort=not specified, suffix=not specified). AA marks this model variant as non-reasoning and reports 0 reasoning tokens. The full CritPt v2 run covers 70 questions with five repeats. The official grading server reports pass@1.. Source model ID: artificial-analysis/9eae4ec4-61b8-48bc-9843-3edd506ae933/devstral-small/devstral-small. Source model name: Devstral Small. Provider reported by source: Mistral. Model snapshot: Devstral Small (Jul '25). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Python functions, symbolic expressions, and numerical answers; official grading server. Attempts: 70 questions, five repeats per question, pass@1. Reasoning: non-reasoning. Reasoning effort: none. Score display: 0.0%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-11-02-67d46be20a5b. Source payload hash: 67d46be20a5b412db6cfb60ad190441fcaa5513500b8fd47d033168d0e97b6b2. Parser version: artificial-analysis-rsc-models-v3. Imported at: 2026-11-02. Source cohort: artificial-analysis-critpt-v2:v2:standalone:non-reasoning:effort=unspecified:suffix=unspecified
- Comparison methodology
- Exact AA configuration: Devstral Small (Jul '25) (release devstral-small, row devstral-small, UUID 9eae4ec4-61b8-48bc-9843-3edd506ae933, reasoning=false, effort=not specified, suffix=not specified). AA marks this model variant as non-reasoning and reports 0 reasoning tokens. The full CritPt v2 run covers 70 questions with five repeats. The official grading server reports pass@1.. Tool policy: Python functions, symbolic expressions, and numerical answers; official grading server. Attempts: 70 questions, five repeats per question, pass@1. Reasoning: non-reasoning. Reasoning effort: none
- Context
- This AA source field contains the original CritPt score. The earlier CritPt v2 attribution was incorrect; the value and former label remain as disputed historical evidence.. The source dataset marks this result as disputed. It remains visible as documented evidence, but is not reliable comparative proof.
IFBench: Overall (Show measurement, test conditions, and source)
- Source value
- 34.5578231292517
- Score
- 34.5578231292517
- Metric
- IFBench prompt-level loose accuracy (pass@1)
- Unit
- %
- Benchmark version
- AA test set: 294 questions
- Category
- general
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- superseded
- Retrieved at
- 2026-11-02
- Methodology
- Exact AA configuration: Devstral Small (Jul '25) (release devstral-small, row devstral-small, UUID 9eae4ec4-61b8-48bc-9843-3edd506ae933, reasoning=false, effort=not specified, suffix=not specified). AA marks this model variant as non-reasoning and reports 0 reasoning tokens. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Source model ID: artificial-analysis/9eae4ec4-61b8-48bc-9843-3edd506ae933/devstral-small/devstral-small. Source model name: Devstral Small. Provider reported by source: Mistral. Model snapshot: Devstral Small (Jul '25). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: non-reasoning. Reasoning effort: none. Score display: 34.6%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-11-02-67d46be20a5b. Source payload hash: 67d46be20a5b412db6cfb60ad190441fcaa5513500b8fd47d033168d0e97b6b2. Parser version: artificial-analysis-rsc-models-v3. Imported at: 2026-11-02. Source cohort: artificial-analysis-ifbench:AA test set: 294 questions:standalone:non-reasoning:effort=unspecified:suffix=unspecified
- Comparison methodology
- Exact AA configuration: Devstral Small (Jul '25) (release devstral-small, row devstral-small, UUID 9eae4ec4-61b8-48bc-9843-3edd506ae933, reasoning=false, effort=not specified, suffix=not specified). AA marks this model variant as non-reasoning and reports 0 reasoning tokens. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: non-reasoning. Reasoning effort: none
- Context
- The often cited 58 refers to verifiable out-of-domain constraint types, not the number of questions in AA's test run.. This result is retained as historical evidence and is not interchangeable with current results.
IFBench: Overall (Show measurement, test conditions, and source)
- Source value
- 34.5578231292517
- Score
- 34.5578231292517
- Metric
- IFBench prompt-level loose accuracy (pass@1)
- Unit
- %
- Benchmark version
- AA test set: 294 questions
- Category
- general
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- active
- Retrieved at
- 2026-11-07
- Methodology
- Exact AA configuration: Devstral Small (Jul '25) (release devstral-small, row devstral-small, UUID 9eae4ec4-61b8-48bc-9843-3edd506ae933, reasoning=false, effort=not specified, suffix=not specified). This source capture does not provide an average reasoning-token count. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Source model ID: artificial-analysis/9eae4ec4-61b8-48bc-9843-3edd506ae933/devstral-small/devstral-small. Source model name: Devstral Small. Provider reported by source: Mistral. Model snapshot: Devstral Small (Jul '25). Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: non-reasoning. Reasoning effort: none. Score display: 34.6%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-11-07-b96e3ffc8c5b. Source payload hash: b96e3ffc8c5b052aa56f96d05e89d02b8cc234a75085820fca0ca2a39e498274. Parser version: artificial-analysis-rsc-models-v4. Imported at: 2026-11-07. Source cohort: artificial-analysis-ifbench:AA test set: 294 questions:standalone:non-reasoning:effort=unspecified:suffix=unspecified
- Comparison methodology
- Exact AA configuration: Devstral Small (Jul '25) (release devstral-small, row devstral-small, UUID 9eae4ec4-61b8-48bc-9843-3edd506ae933, reasoning=false, effort=not specified, suffix=not specified). This source capture does not provide an average reasoning-token count. Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: non-reasoning. Reasoning effort: none
- Context
- The often cited 58 refers to verifiable out-of-domain constraint types, not the number of questions in AA's test run.
SWE-Bench Verified (Show measurement, test conditions, and source)
- Source value
- 53.6
- Score
- 53.6
- Metric
- score as published
- Unit
- %
- Category
- coding
- Direction
- Higher is better
- Source type
- vendor-reported
- Evaluator
- Mistral AI Devstral Small 1.1 model card
- Status
- active
- Retrieved at
- 2026-10-15
- Methodology
- Vendor-reported result. Exact Devstral Small 1.1 row; table names OpenHands Scaffold. Task snapshot and inference settings are not stated.
SWE-bench Verified, Devstral-Small-2507 base-agent resolution rate (Show measurement, test conditions, and source)
- Source value
- 30
- Score
- 30
- Metric
- Issue resolution rate, base agent
- Unit
- %
- Benchmark version
- arXiv:2509.02360v1 Table 2, SWE-bench Verified
- Category
- software engineering
- Direction
- Higher is better
- Sample
- 500
- Source type
- independent-evaluator
- Evaluator
- When Agents go Astray authors, Carnegie Mellon University and IBM Research
- Status
- active
- Retrieved at
- 2026-10-17
- Methodology
- Independent evaluation. The exact Table 2 base-policy row names Devstral-Small-2507. It uses the SWE-agent framework without a SWE-PRM; the paper uses SWE-bench Verified's 500 instances, temperature 0.0, top_p 1.0, a 75-step maximum and two NVIDIA A100 GPUs.
- Context
- This is only the base-agent row. PRM-guided Devstral rows are separate configurations and are not imported here. The paper does not publish an exact model weight revision, quantization, hosted API route or CI.
SoMe social media agent benchmark, average task completion rate (Table 4) (Show measurement, test conditions, and source)
- Source value
- 82.57
- Score
- 82.57
- Metric
- Average task completion rate (TCR) over 8 tasks
- Unit
- %
- Category
- agentic
- Direction
- Higher is better
- Source type
- independent-evaluator
- Evaluator
- SoMe: A Realistic Benchmark for LLM-based Social Media Agents (AAAI-26)
- Status
- active
- Retrieved at
- 2026-10-20
- Methodology
- Independent evaluation. Open-source Devstral-Small-2507 (24B), served with vLLM on 8 NVIDIA A800-SMX4-80GB GPUs; the paper says all LLMs use MCP agent tools. A per-model denominator, run date, exact weights commit, and decoding settings are not reported.
- Context
- No model-specific sample count or confidence interval is stated. The 17,869 annotated task queries describe the dataset and are not used as this model's N.
Tool-output JSON total exact-match accuracy (Show measurement, test conditions, and source)
- Source value
- 0.672
- Score
- 0.672
- Metric
- exact-match accuracy
- Unit
- ratio
- Benchmark version
- EACL 2026 Long Papers Table 7, exact Devstral-Small-2507 checkpoint
- Category
- tool-use
- Direction
- Higher is better
- Sample
- 1298
- Source type
- independent-evaluator
- Evaluator
- How Good Are LLMs at Processing Tool Outputs? (EACL 2026 Long Papers)
- Status
- active
- Retrieved at
- 2026-10-16
- Methodology
- Independent evaluation. Exact Devstral-Small-2507 row; model output type Code, reasoning None, schema None, response Full; zero-shot and greedy decoding at temperature 0, maximum output 1,000 tokens; total question population.
- Context
- Task-specific exact-match result for processing JSON tool outputs, not a software-engineering or general model-quality score. The paper names the checkpoint but does not pin a repository revision.
Tool-output JSON extractive exact-match accuracy (Show measurement, test conditions, and source)
- Source value
- 0.719
- Score
- 0.719
- Metric
- exact-match accuracy
- Unit
- ratio
- Benchmark version
- EACL 2026 Long Papers Table 7, exact Devstral-Small-2507 checkpoint
- Category
- tool-use
- Direction
- Higher is better
- Sample
- 580
- Source type
- independent-evaluator
- Evaluator
- How Good Are LLMs at Processing Tool Outputs? (EACL 2026 Long Papers)
- Status
- active
- Retrieved at
- 2026-10-16
- Methodology
- Independent evaluation. Exact Devstral-Small-2507 row; model output type Code, reasoning None, schema None, response Full; zero-shot and greedy decoding at temperature 0, maximum output 1,000 tokens; extractive question population.
- Context
- Task-specific exact-match result for processing JSON tool outputs, not a software-engineering or general model-quality score. The paper names the checkpoint but does not pin a repository revision.
Tool-output JSON filtering exact-match accuracy (Show measurement, test conditions, and source)
- Source value
- 0.558
- Score
- 0.558
- Metric
- exact-match accuracy
- Unit
- ratio
- Benchmark version
- EACL 2026 Long Papers Table 7, exact Devstral-Small-2507 checkpoint
- Category
- tool-use
- Direction
- Higher is better
- Sample
- 394
- Source type
- independent-evaluator
- Evaluator
- How Good Are LLMs at Processing Tool Outputs? (EACL 2026 Long Papers)
- Status
- active
- Retrieved at
- 2026-10-16
- Methodology
- Independent evaluation. Exact Devstral-Small-2507 row; model output type Code, reasoning None, schema None, response Full; zero-shot and greedy decoding at temperature 0, maximum output 1,000 tokens; filtering question population.
- Context
- Task-specific exact-match result for processing JSON tool outputs, not a software-engineering or general model-quality score. The paper names the checkpoint but does not pin a repository revision.
Tool-output JSON aggregation exact-match accuracy (Show measurement, test conditions, and source)
- Source value
- 0.725
- Score
- 0.725
- Metric
- exact-match accuracy
- Unit
- ratio
- Benchmark version
- EACL 2026 Long Papers Table 7, exact Devstral-Small-2507 checkpoint
- Category
- tool-use
- Direction
- Higher is better
- Sample
- 324
- Source type
- independent-evaluator
- Evaluator
- How Good Are LLMs at Processing Tool Outputs? (EACL 2026 Long Papers)
- Status
- active
- Retrieved at
- 2026-10-16
- Methodology
- Independent evaluation. Exact Devstral-Small-2507 row; model output type Code, reasoning None, schema None, response Full; zero-shot and greedy decoding at temperature 0, maximum output 1,000 tokens; aggregation question population.
- Context
- Task-specific exact-match result for processing JSON tool outputs, not a software-engineering or general model-quality score. The paper names the checkpoint but does not pin a repository revision.
Tool-output JSON total exact-match accuracy (Show measurement, test conditions, and source)
- Source value
- 0.671
- Score
- 0.671
- Metric
- exact-match accuracy
- Unit
- ratio
- Benchmark version
- EACL 2026 Long Papers Table 7, exact Devstral-Small-2507 checkpoint
- Category
- tool-use
- Direction
- Higher is better
- Sample
- 1298
- Source type
- independent-evaluator
- Evaluator
- How Good Are LLMs at Processing Tool Outputs? (EACL 2026 Long Papers)
- Status
- active
- Retrieved at
- 2026-10-16
- Methodology
- Independent evaluation. Exact Devstral-Small-2507 row; model output type Code, reasoning None, schema Full, response Full; zero-shot and greedy decoding at temperature 0, maximum output 1,000 tokens; total question population.
- Context
- Task-specific exact-match result for processing JSON tool outputs, not a software-engineering or general model-quality score. The paper names the checkpoint but does not pin a repository revision.
Tool-output JSON extractive exact-match accuracy (Show measurement, test conditions, and source)
- Source value
- 0.778
- Score
- 0.778
- Metric
- exact-match accuracy
- Unit
- ratio
- Benchmark version
- EACL 2026 Long Papers Table 7, exact Devstral-Small-2507 checkpoint
- Category
- tool-use
- Direction
- Higher is better
- Sample
- 580
- Source type
- independent-evaluator
- Evaluator
- How Good Are LLMs at Processing Tool Outputs? (EACL 2026 Long Papers)
- Status
- active
- Retrieved at
- 2026-10-16
- Methodology
- Independent evaluation. Exact Devstral-Small-2507 row; model output type Code, reasoning None, schema Full, response Full; zero-shot and greedy decoding at temperature 0, maximum output 1,000 tokens; extractive question population.
- Context
- Task-specific exact-match result for processing JSON tool outputs, not a software-engineering or general model-quality score. The paper names the checkpoint but does not pin a repository revision.
Page 1 of 2
Popularity compared
How often the model is searched for, used and mentioned, compared with every model of its category.
Popularity index
The 100 most popular models of this category, 0 to 100, higher is more popular
20 of 101 models
As of November 2026 · Source: gradually.ai popularity index
Over time
Weekly values from October 18, 2026 to November 7, 2026
Devstral SmallLing 3.1 Flash 42Qwen 3 235B A22B 42Xing4.0-29B-A4B 42Hy-Memory-30B-A3B 42
Method changes in the timeline, a jump at these points is no market event:
- 1November 1, 2026: Value only with search or usage data, 30-day attention
What is gaining popularity right now, across models, vendors, harnesses and AI subscriptions, is collected in All AI trends.
Similar models
More models of the same kind as Devstral Small, chosen by model line, developer, size, price and release date.
- Magistral SmallMistral AI · same developer
- Voxtral Small 24BMistral AI · same developer
- Magistral Small 1.2Mistral AI · same developer
- Voxtral Small (latest)Mistral AI · same developer
- Devstral Small 2Mistral AI · same developer
- Ministral 3 14BMistral AI · same developer
- Codestral-22B-v0.1Mistral AI · same developer
- Codestral (latest)Mistral AI · same developer
All specifications
Architecture
Input and output
Watermarking and provenance
AI watermarking refers to invisible signals embedded in the output. Visible marks and provenance metadata are listed separately. Unknown means there is insufficient evidence for that particular field.
- AI watermarking
- Unknown
- no source
- Watermarking technique
- Not known
- no source
Licence and access
Frequently asked questions about Devstral Small
What is the knowledge cutoff of Devstral Small?
May 2025, according to Mistral AI. That is 2 months before release.
Stated by Mistral AI
MayJunJulKnowledge cutoffMay 2025ReleasedJul 10, 2025
How large is the context window of Devstral Small?
128,000 tokens. That is how much text Devstral Small reads at once, about 190 A4 pages.
Page count as a rule of thumb from Artificial Analysis
GLM-5.1200K
DeepSeek-V3 0324164K
GLM-4.5131K
GPT OSS 120B131K
Devstral SmallRank 98 of 103128K
DeepSeek-V3.2128K
DeepSeek-R1128K
Can I run Devstral Small locally?
Yes, the weights are open.
Stated by Mistral AI
- Open weights
- yes
- License
- Apache 2.0
- Parameters
- 24B
- active per token
- n/a
Sources and data date
Show all 15 sources
Benchmark sources at a glance
Which independent evaluators tested this model. This is coverage, not a quality ranking.
| Evaluator | Test series | Composite indices | Recorded active measurements |
|---|---|---|---|
| Artificial Analysis | 1 | 0 | 1 |
Without vendor claims and historical or disputed results; those stay labelled in the single values.