How flawed RL environments can produce misleading AI agent scores

Connect with Centific to discover what's next in AI.

See where to meet us

Connect with Centific.

Find an event

Platforms

Data Marketplace

Data Canvas

AI Data Foundry

OneForma

Data Procurement

AI Localization

Expert Network

Join our Expert Network

Build & Train AI

RL Environments

Data Collection & Creation

RLHF & Preference Optimization

Supervised Fine Tuning

Model Safety & Evaluation

Internationalization

Vertical AI

Physical AI

Healthcare

Vision AI

Explore our full suite of AI platforms, data marketplaces, and expert services designed to build, train, fine-tune, and deploy reliable, production-grade AI systems at scale.

Platforms

Data Marketplace

Data Canvas

AI Data Foundry

OneForma

Data Procurement

AI Localization

Expert Network

Join our Expert Network

Build & Train AI

RL Environments

Data Collection & Creation

RLHF & Preference Optimization

Supervised Fine Tuning

Model Safety & Evaluation

Internationalization

Vertical AI

Physical AI

Healthcare

Vision AI

Explore our full suite of AI platforms, data marketplaces, and expert services designed to build, train, fine-tune, and deploy reliable, production-grade AI systems at scale.

Book a Demo

Article

How flawed RL environments can produce misleading AI agent scores

Learn how flawed RL environments can produce misleading AI agent scores, and how auditing data, tasks, tools, scaffolding, rubrics, and judges helps separate model failures from testing failures.

Published on Aug 25, 2026

•

16 min read time

Table of contents

Summarize

AI Summary by Centific

Turn this article into insights

with AI-powered summaries

Summarize article

Give me key takeaways

Topics

Reinforcement Learning

AI Agents

AI Evaluation

Enterprise AI

Reinforcement Learning

AI Agents

AI Evaluation

Enterprise AI

Author(s)

Abhishek Mukherji

Ananya Mantravadi

Ram Sampath

Sumukha Sharma Thoppanahalli Chandramouli

Leo Qiao

Abdul Aziz Suria

Marinette Chen

Wenjia Hu

Shivali Dalmia

Enterprises increasingly judge AI agents by a single number: the percentage of tasks the agent completes correctly. That number comes out of a reinforcement learning (RL) environment, which is a simulation of the systems, data, and rules an agent must work with, paired with a grader that decides whether the work was done. When the environment itself is flawed, it still returns a clean, ordinary-looking score. The score may say the model is weak. But oftentimes the test was weak instead, and teams may spend months tuning a model that was never the problem.

Centific builds RL environments for frontier labs and enterprises and audits them before delivery. The six failures described here are ones those audits caught in environments we had built ourselves, and each one produced a safeguard that now runs on every environment we ship.

When a clean number measures nothing

One recent task was ordinary enterprise work: pull the deal list, open every pricing sheet, check the margins against the contracts, and write up what is broken. The oracle agent, the one built to solve the task properly, made 75 tool calls. The adversary agent made zero. It wrote a single paragraph claiming that “the following have all been done, verified, and confirmed,” and it won. The adversary earned full marks on four rubric lines, while the oracle scored zero on three of them.

The grader was not broken. It did exactly what we designed it to do: read the final answer and decide whether the work sounded complete. It sounded complete. We caught the problem only because we send a panel of scripted test agents into every task before any real model runs, and one of those agents is deliberately built to cheat so that weak grading shows itself. Without that check, the faulty task would have entered our training data unnoticed, carrying a score that looked perfectly ordinary. That is the trap. A broken test and a normal-looking number are impossible to tell apart.

The agent that did nothing won

Same task, same rubric	ORACLE	ADVERSARY
Tool calls made	75	0
Missing PDPs flagged	0.00	1.00
Forecast & PDP readiness caught	0.00	1.00
Internal handoff routes correctly	0.00	1.00

Same task, same rubric. The do-nothing adversary sweeps every prose-judged line, while the oracle that actually did the work scores zero on three of them. A clean number can come from an environment that measured nothing, and the failure-mode report cannot tell the difference.

A rubric that pays for confident prose is only one cause. Others produce the same result. A broken connector produces a number. A harness that ends the run before the model has had a chance to fail produces a number. The standard deliverable, a sandbox with a batch of rollouts and a failure-mode report, sits downstream of all of them, so it cannot see any of them.

As a result, a broken connector reads as “your agent is bad at retrying.” A gameable rubric reads as “your agent is bad at honesty.” A quarter goes into fixing a model that was never the problem. That is why we build environments to defeat agents rather than to showcase them, by withholding information, injecting failures, leaving destructive tools within reach, and applying pressure mid-task. We test the environment at least as hard as we test the model.

General training reaches about 75%

Environments that teach broad agentic competence move a model from roughly 30% to between 70% and 75%. These are real and valuable tasks, but they are not specific to a single enterprise. The remaining fifteen to twenty points, the climb from 75% to 90% and above, come from work that only one company does. Those points are not general capability. They come from one company’s data, standards, and judgment about what a correct outcome even looks like. No amount of general training produces them because they are not general facts.

Frontier labs need evidence that the environments they buy are reliable, because at volume a weak environment looks like every other one in the pile. Companies deploying agents need environments that accurately reflect how their own employees work. In both cases, improving the model alone will not solve the problem, because it is a data problem long before it is a model problem.

Seven questions that separate evidence from a number

These questions separate a graded environment backed by real evidence from a number that merely reaches the training loop:

Show me the oracle re-run after your last change to the world. Proves the task is still solvable and the change did not silently break it.
Show me the do-nothing agent's score. Now show me the adversary's. Proves the rubric pays for work done, not for a confident paragraph.
Which rubric line paid the highest unearned score, and where in the run did it fire? Proves leaks get pinned to a line instead of averaged into a total.
What is the stall rate per model on this harness? Proves a low number is the model failing, not the scaffold ending the run.
Which failures belong to the system rather than to the agent, and on what evidence? Proves the report can distinguish a fault in your model from a fault in the environment.
What did the environment health probe say at dispatch, and how stale was that reading? Proves the environment was known to be good before a token was spent.
Who reviewed this task, and do they do this job for a living? Proves a human practitioner caught design defects before anything shipped.

If those answers do not exist, what you are holding is a number, not evidence.

Six areas where the environment breaks

Centific has encountered failures in six areas while building and testing RL environments. Each one led to a specific safeguard designed to detect or prevent the same problem in future environments. The safeguards in the right-hand column below are automated checks that run on their own, not written reminders for engineers to be careful.

#	AREA	WHAT WE HIT	WHERE IT LIVES NOW
1	The data	One person was split into two identities, and two people were merged into one. Real identifiers were hiding in code rather than in data.	Entity-resolution anonymization. Three layers, 29 PII types, anchored on one identity per person instead of per string. Gate: re-run the oracle on the rewritten world, plus a sweep of everything the model can see.
2	The task design	A fully verifiable task tested a workflow that nobody in that role actually does.	The SME layer. One practitioner writes the task, a second practitioner reviews it, and the practitioner rather than the author sets the rubric weights.
3	The tools	A forbidden action was blocked or allowed depending on which record the agent picked. Every boolean filter returned a 500 error.	Audit layers 0 and 1. Scoped identity, egress, teardown, then 44 probe checks across 19 units, 26 of them blocking. No model in the loop. Red or stale, nothing dispatches.
4	The format and scaffolding	97 of 100 runs killed by the harness, not the model. Two more on our delivery path.	Format-neutral bundle, thin adapters. One authored task, plus a per-model stall-rate sweep before anyone trusts a cross-model number.
5	The rubrics	An adversary beat the oracle on 13 of 16 tasks. Random scored 1.00 on a judge line inside a 0.232 run.	Audit layer 2. A scripted panel of oracle, adversary, do-nothing, and random agents hits every task before a real model does, and every leak is pinned to the rubric line that paid it.
6	The judge	Judge input was truncated at 600 characters, so the judge was handed malformed JSON whenever content ran long. Widening the window to 2,000 characters moved the scores.	Versioned judge configuration. The input window lives with the rubric, not the grader. Change it and affected trials get re-scored, every report states the window it ran under, and grader fidelity A/Bs the window instead of trusting a default.

Area 1: the data

Enterprise tasks often require an AI agent to connect information about the same person across multiple applications, sometimes six of them at once. A deal in the CRM, an email thread, and a project record may all involve the same employee, and the RL environment needs to preserve those connections.

Centific encountered both types of identity errors. In one case, the same employee appeared under different names in different applications, causing the environment to treat the records as belonging to two people. The join failed silently, and the task could no longer be completed, because information that should have been connected was separated. In another case, two employees who shared a name were treated as the same person, causing the agent to retrieve information for the wrong employee. The second case is the more damaging of the two, because the task still looks answerable and the agent confidently returns an answer about the wrong person.

Centific also found identifying information in an unexpected place. A company name appeared in a tool’s documentation, which is loaded into the agent’s context but sits outside the data covered by the anonymization process.

What it taught us: a name is not a string; it is a link. The agent reads far more than the data you anonymize. So, we anonymize the entity rather than the string, re-run the oracle against the rewritten world, and sweep everything that can reach the model’s context.

Area 2: the task design

Engineering reviews cannot catch this kind of failure. A practitioner reviewed a task of ours that was well built and fully checkable, and pointed out that it tested a job nobody in that role does. Sales representatives find new prospects in a dedicated prospecting tool, Apollo, and use the CRM only to confirm who owns an account. Our task asked the agent to do the sourcing in the CRM, which is not where that work happens. The same reviewer flagged our highest-weighted rubric line, which gave top marks for a kind of caution no real practitioner would show and enforced a rule that an agent doing nothing could pass perfectly.

What it taught us: verifiable is not the same as real, and only someone who does the job can tell you that a well-built task is the wrong task. So, the SME reviewer is never the author, and rubric weights are set by the practitioner.

Area 3: the tools

We audit every connector the way we would audit a customer’s system, and two problems surfaced. First, a delete call that the task clearly forbids failed with a 500 error when the email had attachments but went through cleanly with a 200 when it did not. Whether the agent was stopped depended on which record it happened to grab, which turned a safety check into a coin flip. Second, the simplest filtered query anyone writes, one that passes a true or false parameter, failed every time, because the connector handled those values incorrectly for that database. Only a careful, persistent model ever tried that query. Our weakest model found zero bugs, not because it did anything right, but because it gave up early.

What it taught us: if you plant failures on purpose, every unplanted failure is a lie sitting in your data. So scripted probes hit every tool on a schedule, in seconds and with no model in the loop, and a red or stale verdict stops dispatch. That independence is the only reason we can look at a failure inside a trajectory and say with confidence that it belongs to the agent.

Area 4: the format and the scaffolding

One model kept failing on another team’s test harness, in 97 of 100 runs, each one dying at the same point. Failing identically that many times was the giveaway, because a real model does not fail the same way 97 times. The problem was the harness, not the model. The harness measured progress by looking only for visible text, so a step made of pure reasoning looked exactly like an empty, do-nothing step. A perfectly good step was read as “no progress,” and the run was killed. We re-ran roughly 390 tests changing only the model, confirmed the cause, and reported it to the harness maintainers, who fixed it. Two more bugs of this kind were our own.

What it taught us: model-agnostic is not model-neutral. A number belongs to a model and its scaffold, never to the model alone. So, every model is swept for stall rate before any cross-model comparison, a truncated model is labeled a floor rather than ranked, and every number ships with the health of its scaffold beside it.

Area 5: the rubrics

The earlier grading failure was not an isolated case. Across 16 tasks, the adversarial agent outscored the agent designed to complete the work correctly on 13 of them, taking 14 rubric lines in total, with every one of them judged on prose rather than on anything that happened in the systems. A second problem was harder to spot and more damaging. On another task, seeded-random tool calls scored a perfect 1.00 on a line weighted 0.9, buried inside a weighted total of 0.232 that nobody scanning totals would ever have found.

What it taught us: any rubric line that can be satisfied by a claim will eventually be satisfied by a claim. So rubrics are scored from the state of the systems after the run, a claim with no matching action scores below silence, and a rubric that ranks an adversary above the authored solution is broken by definition.

Area 6: the judge

Some rubric lines cannot be scored by a simple automated check, so a separate judge model reads the answer and decides. We were feeding that judge only the first 600 characters of the answer. The problem is that much of what the judge reads is structured data, and cutting it off at 600 characters leaves a broken, half-finished fragment. The judge grades whatever piece it can still make sense of, and nothing warns us that it saw only part of the answer. When we widened the limit to 2,000 characters as a test, the scores changed. Part of what every report had been measuring was never the agent at all. It was the point at which the text happened to get cut off.

What it taught us: a cut-off limit is a scoring setting in disguise. It looks harmless, but it changes the grade, and the judge will never warn you that it saw only half the answer. It grades the part it received, and it does so with full confidence. That limit is now a tracked setting stored alongside the rubric, and every report states the limit used for the run. We test the limit deliberately rather than relying on a default, and any change automatically re-scores the affected runs.

The self-diagnosis architecture

Centific designed its RL Environment Evaluation Framework around these six areas. Four components do the diagnosing, and each of them runs on a schedule rather than on request:

Three layers of audit. Layer 0 covers security and role-based access control. Layer 1 covers environment health, with 44 probe checks across 19 units and no model in the loop. Layer 2 covers integrity, where the scripted panel attacks the task before any real model runs. Layer 1 always precedes Layer 2, because an integrity result measured against a failing environment is a could_not_run, not a score.
Reports built on a failure-mode taxonomy. The taxonomy has 66 leaves, 55 of them agent failures and 11 of them system failures, so the report answers the question most reports blur: was that the model, or was that us? Every conclusion is tied to stored evidence, and when the evidence is insufficient, the report identifies the limitation rather than guessing.
Subject matter expert input captured as screen recording, not as an interview. We record the practitioner doing the actual job rather than describing it afterward. Descriptions come out tidy and the work is not, and that recording is what makes a task-design flaw catchable at all.
Trajectories abstracted to the business level. A run is rendered as an artifact the expert already recognizes, such as a ledger, a CRM diff, or a filed form, instead of raw chain-of-thought, so practitioner review is fast enough to be routine.

The scarce input is the ability to build the environment, not the model, and unlike a model, that ability can be audited. Data and compute are commodities. Graded tasks and reliable failure diagnosis are not.

What to do next

The recommendation is the same for both audiences: stop trusting the score, and start asking for the evidence behind it.

If you’re a frontier lab buying environments at volume

When you are buying at scale, weak quality is easy to miss, because one weak environment looks like every other environment in the pile. Ask every vendor for two things in every delivery.

First, ask for cheat-test results with every task. Every task should arrive already attacked by the vendor’s adversary agents, with those results included, so you can confirm that the task rewards completed work rather than a confident-sounding summary.

Second, ask for tasks packaged to run on more than one platform. Moving an environment to a new system should take a small adjustment rather than a full rebuild, because a rebuilt environment grades differently from the original and makes your earlier numbers incomparable. Bring the seven questions to every vendor conversation.

If you’re shipping an enterprise agent product

Training an AI agent on broadly applicable tasks can improve its performance to roughly 75%. The last fifteen to twenty points, the climb from 75% to 90% and above, do not come from more general training, because they are not general facts. They live in one company’s data, standards, and judgment about what a correct outcome even looks like. Your agent is judged on a workflow that exists in exactly one company, against a standard that only that company’s own practitioners can define, running on connectors that behave like their real systems. It is a data problem long before it is a model problem, and that data has to be kept both safe and still joinable.

The distinction that matters

An agent that scores 60% may be a weak agent, or it may be a capable agent tested in a broken environment. Nothing in a standard failure-mode report tells the two apart. Auditing the environment is what makes that distinction possible, and it is where the next round of progress in enterprise AI agents will come from.

Next in this series

This article was about the pitfalls. Coming up in this series: the expert-elicitation-to-task pipeline; active learning for expert review; the full reward-hacking catalogue; domain packs, and why the tenth task costs a fraction of the first; what to measure beyond accuracy, including latency, cost, and tokens, and when a number is finally stable enough to say out loud; and visualizing trajectories to find failure modes.

Learn more

Centific builds RL environments and evaluation infrastructure on anonymized enterprise data. For the full method, including three gates, twelve rules, and the evidence behind each, talk to us. And bring the seven questions.

Are your ready to get

modular

AI solutions delivered?

Centific offers a plugin-based architecture built to scale your AI with your business, supporting end-to-end reliability and security. Streamline and accelerate deployment—whether on the cloud or at the edge—with a leading frontier AI data foundry.

Start Building

Connect data, models, and people — in one enterprise-ready platform.

Latest Insights

Ideas, insights, and

research from our team

From original research to field-tested perspectives—how leading organizations build, evaluate, and scale AI with confidence.

Explore

Article

How flawed RL environments can produce misleading AI agent scores

Aug 25, 2026

Article

Why AI agents struggle in enterprise sales operations

Aug 25, 2026

Research insight

Can voice agents help you get through the day?

Aug 13, 2026

Connect with Centific

Stay ahead of what’s next

Stay ahead

Updates from the frontier of AI data.

Receive updates on platform improvements, new workflows, evaluation capabilities, data quality enhancements, and best practices for enterprise AI teams.

Book a Demo

Get a live walkthrough

Talk to our team

Careers

See all our open positions

Turn data into AI that works

Book a demo