JEV Performance on MMLU Across Model Versions

Jev’s MMLU results reveal more about a mismatch between benchmark and architecture than about progress across model versions. Jev is a System One decision model that assigns probabilities to options without generating text. Its strong MMLU scores need to be read task by task, especially in mathematics, where other tests cast doubt on how well it can do.

Why MMLU was built for text generators, not decision models

MMLU was designed to evaluate models that read a question and reason toward an answer. Its scoring method, construction, and subject mix assume a text-generating process underneath the selected answer. The benchmark records whether the choice is correct, but the intended interpretation reaches beyond that choice to the knowledge and reasoning that produced it. This distinction matters when the same answer format comes from a different kind of model.

Jev takes a question and a set of candidate options and returns a probability distribution over those options directly. It generates no intermediate text and performs no chain-of-thought reasoning steps. The interface still fits a multiple-choice test, so the same rules for a correct answer apply when scoring Jev. Yet compatibility at the output does not establish equivalence in the process. A correct choice can demonstrate useful decision performance without demonstrating the generative reasoning that the benchmark was built to assess.

How independent researchers brought Jev to MMLU anyway

TypeSafe published no standard benchmark results, deliberately leaving an evaluation gap that independent researchers filled. Two research groups tested Jev 1.13.0 on MMLU in September and October 2026. Their studies made comparisons possible without treating Jev as a text generator. Both groups examined whether Jev’s decisions and confidence were useful across benchmark types.

Li et al. (2026) evaluated Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding. They compared it with LLMs across three tiers, namely frontier, representative, and small. All comparisons took place within a finite-answer setting. This gave the study a common decision surface across models with different architectures. It also let the researchers distinguish performance on knowledge questions from performance on tasks that demanded calculation.

Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa’s Evaluating and Benchmarking the System One Model Jev supplies the primary evaluation of the calculation-heavy anomaly. The study used the tasksource/mmlu test split, containing 14,042 four-choice items, and C-Eval, with a similarly large item set. It compared Jev against Qwen3.8 and Gemma-4-E4B using identical zero-shot templates. Holding the templates constant made differences between models and benchmarks more interpretable. The tests showed how the models chose among the answers they were given, while leaving generated explanations out of the comparison.

Where Jev's MMLU scores look stronger than its architecture predicts

Jev’s accuracy on calculation-heavy MMLU items is higher than its accuracy on all other items. That is the reverse of the expected pattern for a model with no intermediate reasoning steps. The result is not simply that Jev answers some mathematical questions correctly. It is that calculation appears to be a relative strength within its own MMLU performance. That reversal is the central empirical puzzle.

TypeSafe AI’s failure-mode documentation makes the puzzle sharper. It lists math and numbers, including numeric precision, among Jev’s weaknesses and recommends keeping arithmetic in code. Yet abstract algebra, college mathematics, high-school mathematics, and college physics all score well above Jev’s overall MMLU average. The conflict gives a concrete reason to examine what the benchmark is measuring rather than accepting the subject labels as a capability description.

Qwen3.8 and Gemma-4-E4B move in the opposite direction. Both text-generating models score lower on MMLU items that are calculation-heavy than on the other MMLU items. Jev’s reversal stands out both against what its architecture would lead researchers to expect and against other models tested with the same templates. Outside mathematics, Li et al. (2026) found a more straightforward strength in knowledge and commonsense, with Jev obtaining the best score among evaluated models on MMLU-Redux and ARC-Challenge. Those results make competitive knowledge retrieval plausible, but leave open the question of why a no-reasoning model would outperform text generators on math-heavy MMLU items.

Why C-Eval and option-rotation tests suggest the MMLU math result is not a genuine strength

The strongest challenge to a mathematical interpretation comes from C-Eval. On its calculation-heavy subjects, Jev’s accuracy is substantially lower and its spread is much wider than on MMLU. High-school mathematics came in at 59.6% in the 2026 evaluation, far below the equivalent MMLU scores for the same model. The divergence in apparent subject strength between tests using the same architecture points toward something specific to the benchmark.

The option-rotation control provides a second, different kind of evidence. Rotating answer choices noticeably reduced Jev’s accuracy on the affected items. The question content remained the basis of the task, but rearranging the available answers disrupted performance. That sensitivity is consistent with a model partly responding to the position or frequency of correct answers within the option set. It weakens the case that the original accuracy reflects mathematical understanding alone.

Independent evaluators’ leading interpretation is that memorization of MMLU’s answer-choice distribution inflates Jev’s math scores. This account connects the unusual subject pattern with the rotation sensitivity and the weaker C-Eval results. It remains a hypothesis about the exact training mechanism. The strongest alternative is a task-specific exposure effect from encountering exam-style four-choice questions at high density during training. Such exposure could produce benchmark-specific success without requiring the narrower explanation of option-level frequency exploitation.

The rotation test is suggestive but cannot settle that distinction by itself. The evidence supports caution about transfer more firmly than it identifies the source of the advantage. Under either explanation, strong performance on this particular collection of questions is not enough to establish reliable calculation. The remaining ambiguity is itself informative because it shows that MMLU’s construction and Jev’s training interact in ways that an aggregate accuracy score conceals.

Where Jev's MMLU-adjacent scores expose a genuine architectural ceiling

Harder and less saturated benchmarks reveal a more interpretable limit. Jev sits close to the frontier median on standard MMLU but notably below it on MMLU-Pro. The latter is harder and increases the number of answer choices from four to ten. Both differences limit the value of answer-distribution memorization. The widening gap therefore challenges any reading of standard MMLU as evidence that Jev broadly matches frontier reasoning capability.

Because both the difficulty and the answer format change between the benchmarks, this comparison cannot show the effect of either one on its own. It does, however, show that Jev’s position relative to frontier models depends on the test construction. A score near the frontier median on one benchmark cannot simply be carried over to the other. This is especially important when interpreting the stronger mathematics subjects within standard MMLU.

Li et al. (2026) found the clearest calculation weakness on MathQA. Jev scored substantially below the frontier median and lower than all 19 comparison LLMs. Unlike the unusual MMLU math pattern, this result agrees with the architectural expectation and TypeSafe AI’s stated failure modes. It identifies multi-step calculation as a genuine weakness.

The separate calibration audit described in the notes also found that confidence quality varied by task. Jev had a large expected calibration error on a hidden-policy priority task, while its MMLU binary probabilities ranked well but fell in the wrong places around a fixed 0.5 threshold. Thresholds tuned on training data improved performance on downstream classification tasks such as UNFAIR-ToS. These findings show that probabilities can rank well even when decisions made at a given threshold are unreliable.

What the JevBench leaderboard adds — and what it cannot replace

JevBench provides a comparison surface built around Jev’s intended task type. On JevBench v1.4.2.2, Jev 1.13.0 from TypeSafe AI ranks fourth with a score of 63.29. Imajev leads with 67.37, followed by Plumb with 65.84 and decider v2 by Mapika with 64.13. These standings place Jev among systems evaluated for typed decisions rather than interpreting it through a benchmark built around text generation.

The contrast across benchmarks is instructive. Jev obtains the best evaluated scores on MMLU-Redux and ARC-Challenge, ranks fourth on JevBench, and falls below all comparison LLMs on MathQA. Those positions describe different tasks and need not agree. Nor does its leaderboard resolve the disputed mechanism behind Jev’s MMLU mathematics performance.

What MMLU data on Jev is actually useful for

MMLU is most useful for Jev as evidence of competitiveness in knowledge retrieval. On subjects testing factual recall rather than multi-step calculation, the scores appear meaningful and the C-Eval divergence does not appear. This supports a narrower conclusion than broad reasoning parity with frontier LLMs. Jev can make competitive selections when the task calls for knowledge, even though its process differs from text generation.

MMLU mathematics scores should be treated as suspect by default. The C-Eval divergence, option-rotation sensitivity, and conflict with TypeSafe AI’s failure-mode documentation all argue against transferring those scores to real calculation tasks. MMLU-Pro offers a more reliable frame for assessing Jev relative to frontier LLMs because its ten-choice format limits the value of answer-distribution memorization. It reduces the basis for treating standard MMLU’s unusually strong math results as decisive.

The version evidence is narrower still. As of October 2026, jev-1.13.0 and jev-1.14.0 are publicly released, but the evaluations discussed here establish a baseline for Jev 1.13.0. They provide a single MMLU data point. Future results will need to address the methodological questions, particularly whether a new training run reduces the C-Eval divergence on calculation-heavy subjects.

MathQA is the more relevant signal when a task involves calculation, while MMLU-Redux and ARC-Challenge inform knowledge-retrieval decisions. JevBench supplies evidence about calibration and cost-adjusted accuracy in the typed-decision setting Jev was built for. These measures answer different operational questions, so no single rank can substitute for all of them.