- Source value
- 40.1
- Score
- 40.1
- Metric
- both-orders-correct, engineering/research macro average
- Unit
- %
- Benchmark version
- arxiv:2609.25804v2
- Category
- agentic
- Task
- overall
- Task label
- Overall
- Direction
- Higher is better
- Comparison cohort
- taste-bench-paper-v2-502
- Source type
- independent-evaluator
- Evaluator
- Taste-Bench paper
- Status
- active
- Retrieved at
- 2026-09-26
- Methodology
- Same prompt for all models. Each of 502 questions is shown twice with the options reversed. Average of engineering and research success rates, requiring both orders to be correct. 65,536-token output budget; unparseable answers count as wrong.. Source model ID: curated/gpt-5.4-mini. Source model name: GPT-5.4 mini. Provider reported by source: OpenAI. Evaluator: Taste-Bench paper. Dataset type: public. Leaderboard date: 2026-09-23. Attempts: 502 questions, two option orders each. Reasoning effort: high. Maximum output tokens: 65536. Source ID: taste-bench-paper. Evidence capture ID: curated-taste-bench-paper-2026-09-26-8d521554a8e2. Source payload hash: 8d521554a8e2145aac33904b3e38d2b8fa2ad428110361d37e0b277ded8478b8. Parser version: curated-v1. Imported at: 2026-09-29
- Comparison methodology
- Same prompt for all models. Each of 502 questions is shown twice with the options reversed. Average of engineering and research success rates, requiring both orders to be correct. 65,536-token output budget; unparseable answers count as wrong.
- Context
- 112 of 1,004 responses were unparseable and counted as wrong. Not a Gradually test.