- Source value
- 6.1
- Score
- 6.1
- Metric
- format-invariant word error rate (WER), lower is better
- Unit
- %
- Benchmark version
- VoiceCodeBench v2, arXiv:2608.28916v2 Table 5; per-model artifact date is not linked to the aggregate CSV revision
- Category
- audio
- Direction
- Lower is better
- Source type
- community-reported
- Evaluator
- BeSimple AI Voice Code Bench
- Status
- active
- Retrieved at
- 2026-10-17
- Methodology
- Community-reported result. The arXiv v2 Table 5 row for `OpenAI gpt-transcribe` is raw-audio-only batch ASR on VoiceCodeBench, with no benchmark prompt, target/entity list, domain labels, custom vocabulary, grammar constraints or post-ASR correction; it publishes WER 6.1%, and this cell uses the paper's published precision. A separately pinned per-model artifact names API model ID `gpt-transcribe`, endpoint `https://api.openai.com/v1/audio/transcriptions`, batch mode and evaluation date 2026-09-01. The separately pinned aggregate CSV contains the matching row name `openai_gpt_transcribe` and WER 0.061431750390606506 in its native ratio field. The CSV is pinned at repository revision `8edfa398...`, while the per-model artifact is pinned at `bef2824...`. No published commit-level link assigns both files to the same run, so this cell does not convert a more precise percent value or assign the JSON artifact date to the CSV row. The word denominator, confidence interval and immutable API backend snapshot are not published.
- Context
- BeSimple AI is the commercial external benchmark publisher. The available sources do not establish independent execution or independent reproduction, so this is classified as a neutral community-reported result, not an OpenAI/vendor claim. The API model is an unversioned alias and no immutable serving snapshot is published. Current OpenAI documentation confirms the current file-transcription endpoint, but does not establish which backend snapshot served the 2026-09-01 run. The existing Vals AI VoiceCodeBench page states that its own scores are not directly comparable to paper baselines because model versions, providers and streaming settings may differ. No Vals GPT-Transcribe batch observation was present in the frozen lane input; no inference is made about why scores differ across configurations.