TypeSafe AIDecision modelEarly access

Jev

PopularityRank 221 of 875

43out of 100same as last month

Summary

Jev is TypeSafe AI’s first publicly available System One checkpoint, built for decisions inside software. You send an unstructured state and typed questions. Choice and Score return probabilities and confidence; Noul returns the probability that a statement is true. That lets an application route, rate, or branch a workflow without parsing generated prose.

Jev runs through the jev-latest API model ID and remains in early access. Because the checkpoint emits no answer tokens, text composition is outside its intended use.

GraduallyBench Decision

Automatically evaluated against predefined rules.

The pilot covers 60 test cases from 30 scenarios in German and English. Five equally weighted task families form the score from 0 to 100. Failed requests and invalid answers count as unsuccessful decisions. The 30 robustness variants and 40 development cases are reported separately.

The reference answers follow stored rules and calculations. The automatic reference check took place after the model runs. The task wording and translations have not been checked by human reviewers. Results apply to the exact tested model and route.

Protocol decision-choice-automated-v1. No verified provider invoice.

Jev 1.13.0

TypeSafe · jev-1.13.0 · https://api.typesafe.ai/v1/systemone · Tested Nov 5, 2026

Required decisions across the test plan: 93.3 % (56 / 60)
Valid test responses: 60 / 60
Correct robustness variants: 30 / 30
Correct development cases, outside the test score: 37 / 40
Unnecessary escalations: 0 / 52
Missed required escalations: 2 / 8
Hosted request, median / p95: 334 / 1,030 ms (n = 130)
API cost: Not available

Includes network time and client processing. Jev may retry once internally; its exact HTTP attempt count is unavailable.

Results by task family
Task	Correct / planned	Timeouts	Transport errors	Invalid answers
Triage	11 / 12	0	0	0
Routing	12 / 12	0	0	0
Selection with constraints	12 / 12	0	0	0
Numbers and tables	10 / 12	0	0	0
Sufficient evidence	11 / 12	0	0	0

All benchmark results

Every published result with its source, test setup and comparison models.

Show all 189 published benchmark results

Measurements without matching peer values

These measurements have no matching peer values under the same test conditions. Their original values and sources remain available here.

MMLU-Pro (Show measurement, test conditions, and source)

Source value: 82.7
Score: 82.7
Metric: all-requested accuracy
Unit: %
Category: decision
Direction: Higher is better
Source type: independent-evaluator
Evaluator: JevResearch independent evaluation
Status: active
Retrieved at: 2026-10-15
Methodology: Independent evaluation. Service-reported jev-1.13.0; frozen independent suite; all-requested scoring; report rev. 3, 2026-10-03.

Source JevResearch independent evaluation

GPQA Diamond (Show measurement, test conditions, and source)

Source value: 76.5
Score: 76.5
Metric: all-requested accuracy
Unit: %
Category: decision
Direction: Higher is better
Source type: independent-evaluator
Evaluator: JevResearch independent evaluation
Status: active
Retrieved at: 2026-10-15
Methodology: Independent evaluation. Service-reported jev-1.13.0; frozen independent suite; all-requested scoring; report rev. 3, 2026-10-03.

Source JevResearch independent evaluation

Belebele across 122 languages (Show measurement, test conditions, and source)

Source value: 86.7
Score: 86.7
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; full evaluation split; zero-shot
Direction: Higher is better
Sample: 109800
Lower confidence bound: 86.5
Upper confidence bound: 86.9
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Paper evaluates Jev server version jev-1.13.0; full Belebele evaluation split, zero-shot, one frozen template, 95% percentile-bootstrap CI over 500 resamples. The current API-serving build is not certified by this paper.
Context: Historical result is pinned to jev-1.13.0, not a measurement of the current API alias or deployment.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

SIB-200 (Show measurement, test conditions, and source)

Source value: 81.5
Score: 81.5
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; full evaluation split; zero-shot
Direction: Higher is better
Sample: 41820
Lower confidence bound: 81.2
Upper confidence bound: 81.9
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Paper evaluates Jev server version jev-1.13.0; full SIB-200 evaluation split, zero-shot, one frozen template, 95% percentile-bootstrap CI over 500 resamples. The current API-serving build is not certified by this paper.
Context: Historical result is pinned to jev-1.13.0, not a measurement of the current API alias or deployment.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

AfriXNLI (Show measurement, test conditions, and source)

Source value: 64
Score: 64
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; full evaluation split; zero-shot
Direction: Higher is better
Sample: 10800
Lower confidence bound: 63.1
Upper confidence bound: 65
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Paper evaluates Jev server version jev-1.13.0; full AfriXNLI evaluation split, zero-shot, one frozen template, 95% percentile-bootstrap CI over 500 resamples. The current API-serving build is not certified by this paper.
Context: Historical result is pinned to jev-1.13.0, not a measurement of the current API alias or deployment.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

Belebele low-resource language minimum: Nigerian Fulfulde (Show measurement, test conditions, and source)

Source value: 37.4
Score: 37.4
Metric: accuracy; minimum among the reported language varieties
Unit: %
Benchmark version: arXiv:2609.37647v1; full evaluation split; zero-shot
Direction: Higher is better
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Jev server version jev-1.13.0; per-language result reported in paper prose. Per-language sample count and CI are not published.
Context: Do not infer per-language uncertainty or sample size from the overall benchmark table.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

SIB-200 low-resource language minimum: Santali (Ol Chiki) (Show measurement, test conditions, and source)

Source value: 19.6
Score: 19.6
Metric: accuracy; minimum among the reported language varieties
Unit: %
Benchmark version: arXiv:2609.37647v1; full evaluation split; zero-shot
Direction: Higher is better
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Jev server version jev-1.13.0; Santali in Ol Chiki script, seven-way classification. Per-language sample count and CI are not published.
Context: Do not infer per-language uncertainty or sample size from the overall benchmark table.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

AG News accuracy (Show measurement, test conditions, and source)

Source value: 88.5
Score: 88.5
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; Jev server version jev-1.13.0; zero-shot; full evaluation split
Category: decision
Direction: Higher is better
Sample: 7600
Lower confidence bound: 87.8
Upper confidence bound: 89.1
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Jev server version jev-1.13.0; zero-shot with one frozen dataset-specific template; full published evaluation split, N=7600. Table reports a 95% percentile-bootstrap CI over 500 resamples.
Context: Historical results for Jev server version 1.13.0 and the cited frozen protocols. They do not certify the current Jev API alias.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

IMDB accuracy (Show measurement, test conditions, and source)

Source value: 96.5
Score: 96.5
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; Jev server version jev-1.13.0; zero-shot; full evaluation split
Category: decision
Direction: Higher is better
Sample: 25000
Lower confidence bound: 96.3
Upper confidence bound: 96.7
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Jev server version jev-1.13.0; zero-shot with one frozen dataset-specific template; full published evaluation split, N=25000. Table reports a 95% percentile-bootstrap CI over 500 resamples.
Context: Historical results for Jev server version 1.13.0 and the cited frozen protocols. They do not certify the current Jev API alias.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

Rotten Tomatoes accuracy (Show measurement, test conditions, and source)

Source value: 93.30000000000001
Score: 93.30000000000001
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; Jev server version jev-1.13.0; zero-shot; full evaluation split
Category: decision
Direction: Higher is better
Sample: 1066
Lower confidence bound: 91.7
Upper confidence bound: 94.69999999999999
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Jev server version jev-1.13.0; zero-shot with one frozen dataset-specific template; full published evaluation split, N=1066. Table reports a 95% percentile-bootstrap CI over 500 resamples.
Context: Historical results for Jev server version 1.13.0 and the cited frozen protocols. They do not certify the current Jev API alias.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

SST-2 accuracy (Show measurement, test conditions, and source)

Source value: 96.39999999999999
Score: 96.39999999999999
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; Jev server version jev-1.13.0; zero-shot; full evaluation split
Category: decision
Direction: Higher is better
Sample: 872
Lower confidence bound: 95.1
Upper confidence bound: 97.5
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Jev server version jev-1.13.0; zero-shot with one frozen dataset-specific template; full published evaluation split, N=872. Table reports a 95% percentile-bootstrap CI over 500 resamples.
Context: Historical results for Jev server version 1.13.0 and the cited frozen protocols. They do not certify the current Jev API alias.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

Emotion accuracy (Show measurement, test conditions, and source)

Source value: 58.5
Score: 58.5
Metric: accuracy
Unit: %
Benchmark version: arXiv:2609.37647v1; Jev server version jev-1.13.0; zero-shot; full evaluation split
Category: decision
Direction: Higher is better
Sample: 2000
Lower confidence bound: 56.3
Upper confidence bound: 60.5
Source type: independent-evaluator
Evaluator: University of Bonn and Fraunhofer IAIS, Jev evaluation paper
Status: active
Retrieved at: 2026-10-16
Methodology: Independent evaluation. Jev server version jev-1.13.0; zero-shot with one frozen dataset-specific template; full published evaluation split, N=2000. Table reports a 95% percentile-bootstrap CI over 500 resamples.
Context: Historical results for Jev server version 1.13.0 and the cited frozen protocols. They do not certify the current Jev API alias.

Source University of Bonn and Fraunhofer IAIS, Jev evaluation paper

Page 1 of 16

Popularity compared

How often the model is searched for, used and mentioned, compared with every model of its category.

Popularity index

0 to 100, higher is more popular

20 of 37 models

AutoTrust JEV-27B-VL70

Clef51

GLiNER2.5-multi-Decide45

Decider 4B43

Jev43

pplx-decider-v1.1-27b42

Intern-Decision-4B40

Laya Typed Decisions40

Kev 9B40

pplx-decider-v1-27b39

Decider 0.8B39

Julia 139

Intern-Decision-0.8B38

GLiNER2.5-multi-Decide

Decider 4B

Jev

pplx-decider-v1.1-27b

As of November 2026 · Source: gradually.ai popularity index

Over time

Weekly values from October 18, 2026 to November 5, 2026

JevDecider 4Bpplx-decider-v1.1-27bGLiNER2.5-multi-DecideIntern-Decision-4B

All specifications

Architecture

Model class: System One model; Source: Model class

Watermarking and provenance

AI watermarking refers to invisible signals embedded in the output. Visible marks and provenance metadata are listed separately. Unknown means there is insufficient evidence for that particular field.

AI watermarking: Unknown; no source
Watermarking technique: Not known; no source

Licence and access

Access: API; Source: Access

API

API model ID: jev-latest; Source: API model ID

Further details

Result: Choice and Score with probabilities and confidence, Noul as a value from 0 to 1; Source: Result
Notes: System One model for typed decisions. Jev does not generate free-form text.; Source: Notes

Prices

API input: $0.042 per 1M tokensSource
Output charge: No separate output chargeSource

Sources and data date

Show all 34 sources

Benchmark sources at a glance

Which independent evaluators tested this model. This is coverage, not a quality ranking.

No active result from the reviewed external evaluators can currently be attributed to this model. Other source evidence remains available below.

Without vendor claims and historical or disputed results; those stay labelled in the single values.

Type	Evidence and data date
Research status	Research date October 15, 2026. 26 source URLs checked. This documents the inspected sources, not an exhaustive inventory of every publication.
Unresolved	benchmark-population-details Note Independent report gives 16,379 total requests across three suites; exact per-suite sample counts were not carried into these records.
Unresolved	current-api-serving-revision Note The paper pins Jev 1.13.0, but does not verify which build the current generic Jev API alias serves.
Not published	low-resource-language-sample-sizes-and-confidence-intervals Note The paper gives the Nigerian Fulfulde and Santali minima in prose but publishes no per-language sample counts or confidence intervals.
Unresolved	valid Jev responses for ECE and Brier_raw Note The Table 1 caption says ECE and Brier_raw use valid responses but does not state Jev’s valid-response count.
Not found in the inspected sources	Exact model result in the public Decision Index 0.4.1 panel Note This index edition has no row that can be mapped to the profile repository/checkpoint exactly. This does not imply the model lacks results in other benchmarks.
Additional source	TypeSafe documentation (retrieved September 21, 2026) · TypeSafe documentation · Editorial description reviewed October 2, 2026
Additional source	TypeSafe API documentation (retrieved September 21, 2026) · TypeSafe API documentation · Editorial description reviewed October 2, 2026
Additional source	TypeSafe pricing (retrieved September 21, 2026) · Introducing System One Models and Jev · Editorial description reviewed October 2, 2026
Additional source	Benchmark Heaven JevBench (retrieved October 15, 2026)
Additional source	JevResearch independent evaluation (retrieved October 15, 2026)
Additional source	University of Bonn and Fraunhofer IAIS, Jev evaluation paper (retrieved October 16, 2026)
Additional source	Decision_Model_Benchmark 0.2.0, raw confirmation metrics (retrieved October 16, 2026)
Additional source	Decision_Model_Benchmark 0.2.0, raw CLINC150 confirmation metrics (retrieved October 16, 2026)
Additional source	System One Arena (retrieved October 17, 2026)
Additional source	Jev in Medicine: A Benchmark Evaluation, arXiv:2609.34024v1 (retrieved October 17, 2026)
Additional source	System1-Arena: Which Decisions Can Be Left to System 1?, arXiv:2610.21244v1 (retrieved October 17, 2026)
Additional source	PriorBench Jev evaluation report (retrieved October 17, 2026)
Additional source	Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, and Leo Yu Zhang, JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models, Table 2 (retrieved October 17, 2026)
Additional source	AltSlate Labs LLP, pre-registered Jev option-set evaluation (retrieved October 18, 2026)
Additional source	DecideBench (retrieved October 18, 2026)
Additional source	JEVal, General Decision Models: Benchmarking and Insights Beyond Jev (retrieved October 19, 2026)
Additional source	Von model card (retrieved October 22, 2026)
Additional source	JevResearch (Jev operator) (retrieved October 23, 2026)
Additional source	JEVal, General Decision Models: Benchmarking and Insights Beyond Jev (retrieved October 24, 2026)
Additional source	Model metadata (retrieved October 16, 2026)
Additional source	Model metadata (retrieved October 17, 2026)1 2 3 4 5 6 7
Additional source	Model metadata (retrieved October 18, 2026)
Additional source	OpenRouter: Popularity index (derived), CC BY 4.0
Additional source	Vercel AI Gateway: Popularity index (derived), CC BY 4.0
Additional source	Ollama: cumulative downloads for the popularity index
Additional source	Replicate: cumulative runs for the popularity index
Additional source	Google Trending Now: limited feed selection for Germany and the US
Additional source	Creative Commons Attribution 4.0