How TypeSafe’s Jev compares with Claude as a judge

Connect with Centific to discover what's next in AI.

See where to meet us

Connect with Centific to discover what's next in AI.

Platforms

Data Marketplace

Data Canvas

AI Data Foundry

OneForma

Data Procurement

AI Localization

Expert Network

Join our Expert Network

Build & Train AI

RL Environments

Data Collection & Creation

RLHF & Preference Optimization

Supervised Fine Tuning

Model Safety & Evaluation

Internationalization

Vertical AI

Physical AI

Speech & audio

Healthcare

Explore our full suite of AI platforms, data marketplaces, and expert services designed to build, train, fine-tune, and deploy reliable, production-grade AI systems at scale.

Platforms

Data Marketplace

Data Canvas

AI Data Foundry

OneForma

Data Procurement

AI Localization

Expert Network

Join our Expert Network

Build & Train AI

RL Environments

Data Collection & Creation

RLHF & Preference Optimization

Supervised Fine Tuning

Model Safety & Evaluation

Internationalization

Vertical AI

Physical AI

Speech & audio

Healthcare

Explore our full suite of AI platforms, data marketplaces, and expert services designed to build, train, fine-tune, and deploy reliable, production-grade AI systems at scale.

Book a Demo

Research insight

How TypeSafe’s Jev compares with Claude as a judge

Discover how TypeSafe’s Jev compares with Claude for AI evaluation, delivering comparable accuracy at a fraction of the cost while exposing limitations in newer models.

Published on Nov 2, 2026

•

10 min read time

Topics

AI Evaluation

LLM Benchmarks

AI Safety

Model Safety & Evaluation

Author(s)

Naman Khandelwal

Shivali Dalmia

Abhishek Mukherji

Sasakorn Phanitsombat

Summarize

Companies evaluating AI often use large language models (LLMs) to judge whether AI-generated responses are accurate, appropriate, or harmful. These models, known as LLM judges, read conversations, assess the responses, and return a verdict.

But judging can be expensive. A typical LLM judge reads the entire conversation, generates an explanation of its assessment, and then provides an answer. Companies pay for that processing even when their automated evaluation systems only need the final verdict.

In an earlier study, Centific’s Applied AI Research team found that judging consumed more tokens than generating the test scenarios themselves. The finding prompted us to investigate whether a decision model that produces answers without generating written reasoning could deliver comparable judging accuracy at a fraction of the cost.

We tested TypeSafe’s Jev against three Claude models using the three judging tests in our Token Optimization Benchmark. Jev is a decision model made by TypeSafe and offered through OpenRouter. We tested version 1.13 (typesafe/jev-1.13). A normal LLM judge writes out its reasoning and then gives an answer. Jev doesn’t write any reasoning. It reads the conversation and returns an answer, such as “harmful” or “safe,” along with a probability. It writes about 24 tokens per answer, you only pay for the text it reads, and in our runs it took about half a second per answer.

The problem: judging is expensive, and the newest models made it less reliable

For companies running these checks, there are three issues:

Cost grows with traffic. If a judge checks every live conversation, you pay for every check. In our runs, Sonnet 4 cost $114 for every 1,000 political-bias checks, at about 27,000 tokens each. A million checks a month would cost about $114,000, for that one check alone.
Upgrading made it worse. Anthropic released Claude Opus 5.5 and Sonnet 5.5 in late September 2026. In our tests, both refused most of the enterprise attack conversations (Opus 5.5 refused all 104), and when they did judge single replies, they called more than half of the harmful ones safe. Sonnet 5.5 is cheaper per token than Sonnet 4, so the problem isn’t price. It’s that the newer models were less reliable judges.
Prompt tricks only go so far. The tricks in our Token Optimization Benchmark shorten what the judge reads or writes, but saved only 10% to 38%.

Companies evaluating AI judges therefore need to look beyond model size and token pricing. The more useful comparison is how much it costs to obtain a reliable judgment, including whether the model consistently produces a usable answer.

How we compared Jev with Claude models

We used the same test items, prompts and answer keys as our Token Optimization Benchmark, and only changed the judge. We picked an answer-only judge because of what we saw in April: much of a judge’s cost comes from the text it reads and the reasoning it writes. Jev doesn’t write reasoning at all, so we wanted to see if it could still get the answers right.

To keep the comparison fair, every judge got exactly the same inputs. We show every cost next to an F1 score, which is a number from 0 to 1 that goes up when a judge catches real problems and down when it raises false alarms. We also compared each result with a fake judge that always answers “yes”. If that fake judge gets a good score, the score doesn’t tell you much.

The judges were Jev; Claude Sonnet 4, the benchmark’s original judge ($3 per million tokens read and $15 per million written, with results taken from our Token Optimization leaderboard); and Claude Sonnet 5.5 ($2 / $10) and Opus 5.5 ($4 / $20), which we ran with the same prompts.

Each test has an answer key, which is the set of correct labels we check every judge against.

Test	What’s being judged	Items	Answer key	What the judge returns	How we score it
Redteaming	Single AI replies from the WildGuardMix test set	498 (82 harmful)	Human labeled each reply harmful or safe	“Harmful” or “safe”	F1 on catching harmful replies
LM Behavior (political bias)	Conversations made with Anthropic’s Bloom tool	66; we score the 27 clearest (15 biased)	A Claude Opus 4.1 judge’s score: above 7 is biased, below 2 is neutral	A bias score from 1 to 10; above 5 counts as biased	F1, plus ranking quality (AUROC)
Enterprise redteaming	Multi-turn attacks on company chatbots	104 (89 violations)	Human reviewers rated severity from 1 to 4; 2 to 4 counts as a violation	“Violation” or “no violation”	F1; a missing answer counts as wrong

The benchmark calls the first test “Redteaming,” but it only checks single replies. The enterprise test is the only one with real multi-turn attacks, where someone keeps pushing the chatbot over several messages.

On this political-bias test, our cut-off of 5 is lower than the 7 that Anthropic’s Bloom uses. The newer Claude models ranked the conversations sensibly (ranking score AUROC across all 66 conversations: 0.88 for Sonnet 5.5 and 0.81 for Opus 5.5, against 0.98 for Jev) but gave low scores, so their F1 here mostly reflects that fixed cut-off rather than an inability to spot bias. If you use a judge that gives scores, set its cut-off on your own labeled data.

We also re-ran two prompt-shortening tricks from the benchmark: Selective Context, which removes the most predictable sentences, and LLMLingua-2, which removes low-value words. We only re-ran tricks that shorten what the judge reads, so every judge could be tested the same way. TALE-EP shortens the judge’s written reasoning, which Jev doesn’t produce. The Sonnet 4 TALE-EP results shown come from the benchmark.

A refusal means the model’s own safety system stopped it from answering. On Redteaming, a missing answer counts as “safe,” which is the benchmark’s rule. On the enterprise test, a missing answer counts as wrong. To check the refusals weren’t caused by our setup, we sent every refused item to Sonnet 4 with the same prompt, and it answered all of them. The refusals came from the newer models themselves. We also show 95% confidence ranges, which tell you how much a score could move just by chance on a test this small.

Jev vs. Claude: evaluation results

Our results show that the choice of judge affects evaluation costs far more than shortening prompts. They also challenge the assumption that newer models make better judges, at least on safety tests. Four findings show this.

Jev reduced AI evaluation costs more than prompt optimization

The cheapest prompt trick cut Sonnet 4’s cost by 10% on Redteaming, 17% on LM Behavior and 38% on the enterprise test. Jev cost 93 to 1,721 times less than Sonnet 4. If judging costs too much, check which model you use before spending time on prompts. Jev was also faster: about half a second per answer, against 3 to 39 seconds for Sonnet 5.5 and Opus 5.5 when they answered.

On single replies, Jev did as well as Sonnet 4 and better than the newest models

Jev’s F1 was 0.758 and Sonnet 4’s was 0.717. The difference could be down to chance, since it falls inside Jev’s 95% range of 0.68 to 0.83, so we count it as a tie. In the chart, the pink bar shows that range, and Sonnet 4’s dot falls inside it. Jev did clearly better than Sonnet 5.5 (by 0.25) and Opus 5.5 (by 0.33). For checking single replies for harm, we think Jev is worth a proper trial.

Claude Sonnet 5.5 and Opus 5.5 frequently refused safety evaluations

Jev answered all 668 items across the three tests. Every Claude judge left some items unanswered; even Sonnet 4 replied but gave no usable answer on 32 of the 104 enterprise conversations. Refusals weren’t the only problem. Of the 82 harmful replies, Jev caught 58. Sonnet 5.5 refused 19, and of the 63 it did judge, it marked 33 as safe. Opus 5.5 refused 28, and of the 54 it did judge, it marked 29 as safe. So even when the newer models answered, they called more than half of the harmful replies safe. On the enterprise test it was worse: Sonnet 5.5 gave a usable verdict on 6 of the 104 conversations, calling all 6 “not violated,” and Opus 5.5 answered none. If you move your judge to a newer model, re-run your test set first and count both refusals and misses.

This matters for model providers too. The safety systems that stop a chat assistant from engaging with harmful content also stopped these models from grading it, which is exactly the job a safety judge needs to do.

Shortening the input didn’t help Jev, and Jev stays far cheaper even if its price rises

Selective Context raised Jev’s Redteaming F1 by 0.019, which could easily be chance. LLMLingua-2 made Jev worse on the bias test: its ranking score (AUROC, where 1 is perfect and 0.5 is a coin flip) dropped from 0.981 to 0.840 across all 66 conversations. These tricks make more sense for expensive judges. Even if Jev’s price went up 10 times, it would still cost 9 to 172 times less than Sonnet 4.

Limitations of the Jev and Claude comparison test

These findings are promising, but the strength of the evidence varies across the three tests. Several limitations affect what we can conclude about Jev’s accuracy relative to Claude.

The enterprise test shows refusals, not accuracy. Human reviewers rated 89 of its 104 conversations as severity 2 or higher (a violation), so a judge that always says “violated” would beat every real judge on F1. The refusal counts stand; the accuracy scores there don’t tell you much.
The political-bias test is small and graded by a model. Only 27 conversations count, and the correct answers come from a Claude Opus 4.1 judge, not people.
We can’t compare Jev and Sonnet 4 item by item. On the red-team and political-bias tests, we only have Sonnet 4’s published average scores, not its answer on each item. So we can’t check whether the small gaps between them are real, and we treat the two as roughly level.

The strongest evidence comes from the single-reply red-teaming test, where Jev performed comparably to Sonnet 4 at a much lower cost. Before replacing an existing judge, companies should test Jev against human-reviewed examples from their own applications to confirm that the cost savings don’t come at the expense of accuracy.

How to choose an AI evaluation model

Our results suggest that for checks like these, you don’t need a large reasoning model as the judge. A small answer-only judge like Jev can be just as accurate, in half a second and at a fraction of the cost.

If you want to try it:

Start in the background. Run it next to your current judge on live traffic and compare their answers before switching.
Tune it on your own data. Have people label a few hundred of your own items, and use them to decide how confident Jev must be before you count something as harmful.
Keep a light check. Have a person review a small sample of its answers, since Jev gives verdicts without explanations.
Re-test when things change, such as a new model version or a new policy.

The decision to switch judges should depend on how well a model performs on your own data, rather than its size or reputation. If Jev delivers comparable accuracy, the savings could make it practical to evaluate far more AI interactions without increasing costs.

Evaluate AI models with Centific

At Centific, our Model Safety & Evaluation team measures every automated judge against expert human reviewers before trusting it. This study showed that the biggest saving in judging may come not from a shorter prompt or token optimization, but from choosing the right judge.

modular

AI solutions delivered?

Centific offers a plugin-based architecture built to scale your AI with your business, supporting end-to-end reliability and security. Streamline and accelerate deployment—whether on the cloud or at the edge—with a leading frontier AI data foundry.

Start Building

Connect data, models, and people — in one enterprise-ready platform.

Latest Insights

Ideas, insights, and

research from our team

From original research to field-tested perspectives—how leading organizations build, evaluate, and scale AI with confidence.

Explore

Research insight

How TypeSafe’s Jev compares with Claude as a judge

Nov 2, 2026

Research insight

Verifying robot data at scale: NVIDIA Cosmos Curator + Cosmos 3 in the Centific physical AI stack

Oct 23, 2026

Research insight

Why AI models struggle to verify robot training data: research with NVIDIA Cosmos

Oct 22, 2026

Connect with Centific

Stay ahead of what’s next

Stay ahead

Updates from the frontier of AI data.

Receive updates on platform improvements, new workflows, evaluation capabilities, data quality enhancements, and best practices for enterprise AI teams.

Book a Demo

Get a live walkthrough

Talk to our team

Careers

See all our open positions

Turn data into AI that works

Book a demo

How TypeSafe’s Jev compares with Claude as a judge

The problem: judging is expensive, and the newest models made it less reliable

How we compared Jev with Claude models

Jev vs. Claude: evaluation results

Jev reduced AI evaluation costs more than prompt optimization

On single replies, Jev did as well as Sonnet 4 and better than the newest models

Claude Sonnet 5.5 and Opus 5.5 frequently refused safety evaluations

Shortening the input didn’t help Jev, and Jev stays far cheaper even if its price rises

Limitations of the Jev and Claude comparison test

How to choose an AI evaluation model

Evaluate AI models with Centific

Are your ready to get

modular

AI solutions delivered?

Ideas, insights, and

Ideas, insights, and

research from our team

research from our team

Stay ahead of what’s next

Stay ahead

Updates from the frontier of AI data.