How we measure AI models

Right now 3,642 measurements from 84 test series go in. 115 models have an overall value, 95 of them a place. A test series is one benchmark that several models have taken.

• Only independent evaluators and official leaderboards count. A number from the maker's own model card does not go in.
• Only each source's current listing counts. Older retrievals do not replace a newer series.
• For each source and benchmark one test setup counts, the one in which the most models are clearly matched. There is no picking of the best setup per model.
• If a model has several configurations in the same benchmark, the one with the highest documented reasoning effort counts. The result never decides the selection.
• A series needs at least 5 models.
• Composite indices from other providers do not count on top of their individual tests, so nothing goes in twice.

• Each series is brought to a common scale: a value counts by how far it lies above or below the average of its series. Percentages are stretched so that the gap between 95 and 98 percent counts for more than the gap between 50 and 53.
• From all series the index estimates one common strength for each model. Series that separate strong and weak models well count for more.
• If a model has only a few test series so far, the index pulls its strength slightly towards the middle. That way a single outlier does not lift a new model straight to the top. Since version 5 this pull is weaker: a model with 8 test series lands on average where it would stand with all series, instead of about 1.5 to 2 points below as up to version 4.
• On top of that comes one deviation per area from the common strength. If a model has only a few measurements in an area, the deviation is pulled towards zero. The areas are coding, reasoning, math, knowledge, agents, general tasks and multimodal tasks (core areas) and finance, law and health.
• The overall value weights the 7 core areas with 75% and the 3 other areas together with 25%. An area without a measurement of its own counts with the model's common strength, not as zero.

Value

100 is the average of a fixed reference group of models. A gap of 15 points means the higher model comes out ahead in roughly three of four head-to-head comparisons in a typical benchmark. The scale is open at both ends and is not a percentage.

Place

The position among all models with a place, sorted by overall value. A model gets a place only when its value is certain enough: the uncertainty may be at most 7.35 points. Otherwise the value is shown as provisional. A model that already has a place keeps it up to an uncertainty of 7.65 points, so small fluctuations of the calculation do not shift places.

Performance groups

The top group (3 models) contains the winner and the models that are level with it. Then come the upper group from 115 points (7 models), the middle group from 85 points (41) and below it the lower group (44).

Level

A model is level with the winner if it lands ahead of it in at least 5% of the recalculations. If the winner is the same as in the newest index state, models already in that top group remain level down to 3.5%, while a new model joins only from 6.5%. For this the index recalculates 1,000 times, each time letting test families count several times or not at all at random.

Win chance

The share of those recalculations in which a model is in first place. The win chances of one area add up to at most 100%.

An overall value needs at least 8 test series, 2 sources and 3 areas with measurements of their own. If any of these is missing, there is no value. That is a gap, not a zero.

Models that lack only the second source show a provisional value, always labelled “one source” (18 models). They get no place, no group and no win chance, and they do not move the scale.

Models without an independent overall value can get an estimate from vendor figures. It is shown separately and never goes into the index. It appears only when enough models are available to calibrate it and the estimate reproduces their order well enough.

Vals AI supplies 58% of all observations in the index. If the index is recalculated without this source, Claude Opus 5.5 leads instead of Claude Fable 5.2.

The top therefore depends on a single source. Read it as a hint, not a verdict.

• Models with few measurements carry more uncertainty. New models therefore often get a place only after a few more tests, and their value can still move by several points until then.
• On average the vendor estimate lies about 10 points away from the independent value, so it is only a rough guide. For the models it rates highest the vendor estimate comes out slightly too low. That difference is not large enough to count as established.
• Each series uses the highest documented reasoning effort. Some top models are measured only up to a lower effort, so the index partly compares compute too. The charts mark this.
• Benchmarks measure tasks that someone picked. Your task may look different.

1. Version 1October 19, 2026

   Rank percentiles, every area weighted equally.

2. Version 2October 20, 2026

   Fixed editorial weights per area and test family.

3. Version 3October 20, 2026

   Statistical model with one common strength per model, recalculations instead of fixed tie limits.

4. Version 4November 4, 2026

   Scale with 100 as the mean and 15 points as the spread, unmeasured areas treated as unknown, evaluation with held-out test families.

5. Additions in version 4November 10, 2026

   More recalculations, performance groups, a note on the share of the largest source and provisional values for models with a single source.

6. Version 5November 11, 2026

   A weaker pull towards the middle for models with few test series. The limit for a place stays the same in points. Values from versions 4 and 5 are not directly comparable.

7. Additions in version 5November 11, 2026

   A place and a seat in the top group are no longer lost through small fluctuations of the calculation. The vendor estimate pulls towards the middle as strongly as it did up to version 4 again.

The newest measurement in the index is from November 12, 2026.

• Audit file (JSON)

  all included and excluded measurements with reasons

• Data status

  how recent each dataset is

• LLM comparison

  the index in the leaderboard

• Best LLM by task

  the area values as their own rankings

• Compare multiple LLMs

  up to five models side by side

The popularity score shows how many people search for, use and discuss a model. It runs from 0 to 100 and each area has its own scale. Scores compare within one area, not across areas.

• Search counts for 54%, usage for 36% (for example API tokens, downloads or installs) and mentions in news and newsletters for 10%.
• Only channels with a value count. When a channel is missing, its weight is spread over the others. A missing value is unknown and never 0.
• Mentions enter as the mean of the last 30 days, search volume as a monthly value. A model with mentions only does not get a value yet.
• A trend counts as “Rising” from +4 points against about four weeks earlier and “Rising fast” from +10. “New” marks rising models that are at most 14 days old.
• Harnesses and AI plans use the same weights. For the best-known people in the industry, Wikipedia views count for 70% and mentions in newsletters and feeds for 30%.

In the LLM comparison you can switch the value between World, EU and Germany. Regionally only search and attention count, because there is no regional source for usage.

Popular does not mean good. When the method changes, for example through new data sources or a new scale, a value can jump. The timelines of the model profiles mark such changes, and they are no market event.

• AI trends

  the most popular models, vendors, harnesses and AI plans

For vendors we collect our own signals. Simply adding up the model scores would measure the popularity of single models and favor vendors with many models.

• Search (54%): the worldwide monthly search volume for the vendor, including its AI brand, such as ChatGPT for OpenAI. Per month the strongest spelling counts, spellings are never added.
• Usage (36%): the vendor's tokens on OpenRouter and the downloads of its models on Hugging Face, weighted equally.
• Attention (10%): the share of stories in a fixed public sample of RSS feeds and Hacker News that name the vendor, as the mean of the last 30 days.
• Each channel is mapped to 0 to 100. When a channel is missing, only the others count. A missing channel is unknown and never 0.

Downloads are not a user count, and tokens on OpenRouter show only part of the market. Here too, popular does not mean good.

• AI trends

  the most popular vendors

Speed is not a measurement of our own. The values come from Artificial Analysis and state output tokens per second. Where no current measurement exists, the value stays empty.

• The fastest LLMs

  the speed chart in the LLM comparison

The chart “Performance vs. cost per task” sets the overall index against the cost Vals AI paid per task of the Vals Index, at the reasoning tier stated for each model. Higher tiers cost more because the model thinks for longer. The cost axis is logarithmic.

• Performance vs. cost per task

  the chart in the LLM comparison

GraduallyBench Image v2 is our own image test. Every model gets the same four tasks. Two AI judges compare the images in pairs and blind. That yields points from 0 to 100 per task, and the overall value is the mean of the four fixed tasks.

• One image per model counts for each task. The ranking therefore rests on one attempt and stays separate from research benchmarks and the blind votes of the arenas.
• Models that still lack tasks or the shared rating get no place.
• Next to our test, the image model comparison shows the ratings from the anonymous LMArena blind comparisons.

For video models we show sample clips and the ratings from arena blind votes. We do not give videos a score of our own.

• Image model comparison

  all images from GraduallyBench Image v2

• Video model comparison

  sample clips, arena rating and prices

The category pages for decision and audio models show three automatically evaluated tests. They are pilot protocols, give no overall ranking and apply only to the exact tested model identifier and route. Every tested model carries its status, for example “Benchmark run”, “Partially tested”, “Technical test” or “Short functional test”.

GraduallyBench Decision v1

60 test cases from 30 scenarios in German and English. Five equally weighted task families (triage, routing, selection with constraints, numbers and tables, sufficient evidence) form a score from 0 to 100. Failed requests and invalid answers count as unsuccessful. Robustness variants and development cases are reported separately.

GraduallyBench STT v1

Speech recognition on 32 read-speech recordings in German and English with four speakers each, plus four digital-silence clips. We measure word error rate (WER) and character error rate (CER), and for silence the invented words. Spontaneous speech and background noise are outside the test.

GraduallyBench TTS v1

The model speaks a text, a recognition model transcribes the recording back, and we compare the return text with the input text. That measures how accurately the content arrives, not how natural a voice sounds.

The reference answers follow stored rules and source transcripts. Task wording, translations and transcripts have not been checked by humans.

• Decision models

  results of GraduallyBench Decision v1

• Audio models

  speech recognition and speech output