- Source value
- 69.7959183673
- Metric
- IFBench prompt-level loose accuracy (pass@1)
- Unit
- %
- Benchmark version
- AA test set: 294 questions
- Category
- general
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Source type
- independent-evaluator
- Evaluator
- Artificial Analysis
- Status
- superseded
- Retrieved at
- 2026-11-02
- Methodology
- Exact AA configuration: Mercury 2 (release mercury-2, row mercury-2, UUID 016d330a-2141-4afa-b2fc-62b314423dc1, reasoning=true, effort=high, suffix=not specified). AA source value for average reasoning tokens before the answer: 0. Its methodology averages 60 varied prompts and assumes 2,000 when no average is available. This is not a token cap.
Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Source model ID: artificial-analysis/016d330a-2141-4afa-b2fc-62b314423dc1/mercury-2/mercury-2. Source model name: Mercury 2. Provider reported by source: Inception. Model snapshot: Mercury 2. Evaluator: Artificial Analysis. Dataset type: artificial-analysis-evaluation. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled. Reasoning effort: high. Score display: 69.8%. Source ID: artificial-analysis. Evidence capture ID: aa-model-page-2026-11-02-67d46be20a5b. Source payload hash: 67d46be20a5b412db6cfb60ad190441fcaa5513500b8fd47d033168d0e97b6b2. Parser version: artificial-analysis-rsc-models-v3. Imported at: 2026-11-02. Source cohort: artificial-analysis-ifbench:AA test set: 294 questions:standalone:reasoning:effort=high:suffix=unspecified
- Comparison methodology
- Exact AA configuration: Mercury 2 (release mercury-2, row mercury-2, UUID 016d330a-2141-4afa-b2fc-62b314423dc1, reasoning=true, effort=high, suffix=not specified). AA source value for average reasoning tokens before the answer: 0. Its methodology averages 60 varied prompts and assumes 2,000 when no average is available. This is not a token cap.
Artificial Analysis runs 294 questions from the IFBench test set with five repeats per question. The official evaluator scores prompt-level accuracy in loose mode as pass@1.. Tool policy: Single-turn answer evaluated with the official IFBench source code. Attempts: pass@1, 5 repeats per question. Reasoning: reasoning enabled. Reasoning effort: high
- Context
- The often cited 58 refers to verifiable out-of-domain constraint types, not the number of questions in AA's test run.. This result is retained as historical evidence and is not interchangeable with current results.