Why AI models struggle to verify robot training data: research with NVIDIA Cosmos

Connect with Centific to discover what's next in AI.

See where to meet us

Connect with Centific to discover what's next in AI.

Platforms

Data Marketplace

Data Canvas

AI Data Foundry

OneForma

Data Procurement

AI Localization

Expert Network

Join our Expert Network

Build & Train AI

RL Environments

Data Collection & Creation

RLHF & Preference Optimization

Supervised Fine Tuning

Model Safety & Evaluation

Internationalization

Vertical AI

Physical AI

Speech & audio

Healthcare

Explore our full suite of AI platforms, data marketplaces, and expert services designed to build, train, fine-tune, and deploy reliable, production-grade AI systems at scale.

Platforms

Data Marketplace

Data Canvas

AI Data Foundry

OneForma

Data Procurement

AI Localization

Expert Network

Join our Expert Network

Build & Train AI

RL Environments

Data Collection & Creation

RLHF & Preference Optimization

Supervised Fine Tuning

Model Safety & Evaluation

Internationalization

Vertical AI

Physical AI

Speech & audio

Healthcare

Explore our full suite of AI platforms, data marketplaces, and expert services designed to build, train, fine-tune, and deploy reliable, production-grade AI systems at scale.

Book a Demo

Research insight

Why AI models struggle to verify robot training data: research with NVIDIA Cosmos

Centific’s research with NVIDIA Cosmos reveals why VLM based verification of robot training data falls short, and proposes RoboVerify Bench and pose token grounding as the path forward.

Published on Oct 22, 2026

•

16 min read time

Topics

Physical AI

Robotics Training Data

Data Annotation

NVIDIA Cosmos

Verticals

Physical AI

Author(s)

Leela Krishna

Kriti Banka

Mangesh Damre

Shyam Ganatra

Ravi Garg

Summarize

The economics of robot training data turn on verified hours rather than collected hours, as we articulated in the blog article “The 4% problem: how Centific builds better robot training data, powered by NVIDIA Cosmos Curator.” Now let’s go a level deeper into verification itself.

Centific operates one of the larger egocentric human-demonstration data factories for Physical AI: multi-thousand-episode pipelines spanning egocentric multiview capture, UMI-style gripper data, dexterous 21-DOF hand annotation, MuJoCo/NVIDIA Isaac Sim replay, and LeRobot-format delivery. As an NVIDIA Advanced Technology Partner, we run NVIDIA Cosmos models in production quality control today, and over the past quarter we completed one of the most systematic industrial evaluations to date of VLM-based QC for robotics data: 14 distinct prompt architectures across 5 system designs and 4 input modalities, evaluated against human review and trivial baselines, with Cosmos 3 Nano Reasoner as the primary judge.

Our evaluation revealed an important difference between the capabilities of the Cosmos generator and reasoner. The generator tower’s competence in forward dynamics and latent-space prediction is precisely the representation in which robotics verification problems are solvable, while the reasoner’s language interface cannot yet carry pose, motion direction, orientation, or temporal correspondence at the granularity QC requires.

This reproduces published results across the whole video-LM class (SpookyBench, DORI, VLM4D). What is Cosmos-specific is that both towers already live in one jointly-attended model, so Cosmos 3 is uniquely positioned to close the gap in a way no single-tower model can. The precedent is the Grounding-DINO-class localization lift of the Reason 2 generation: a perception primitive given first-class token representation transformed a whole category of failures into a reliable interface. Pose, motion, and cross-video comparison are sitting where localization sat before that lift.

This article examines five use cases:

Sim-replay verification of dexterous hand data, supported by production data and video examples.
RoboVerify-Bench, a proposed benchmark for identifying gaps in robotics data verification.
Reasoner capabilities that could complement the Cosmos Predict tower.
Cosmos Curator evaluation, including a proposed allocation of 8×H100 or 4×B200 GPUs.
Human-in-the-loop validation of VLM and sensor-derived verdicts on Centific Data Canvas, driven by Humanoid IQ quality-control signals.

The article concludes with a proposed solution path and topics for further discussion.

Evidence base: production data and video examples

The evidence here comes from human-reviewed production episodes, each scored by every verification method we tested and measured against trivial baselines.

The verification task

Each episode is an egocentric recording of a person performing a manipulation task. Hand pose is estimated from the video (WiLoR, 21-DOF MANO), retargeted onto a pair of Inspire robotic hands, and replayed in MuJoCo. Every episode automatically produces review artifacts uploaded alongside the LeRobot dataset: compare.mp4 (real vs. simulation side-by-side, the input to all VLM judging below), replay.mp4, and inspire_qpos.npz (the retargeted joint-state stream, which makes fault-injected ground truth free and renewable). The simulation deliberately contains only the two hands over a plain ground plane, which is the central difficulty for semantic judging, and the reason geometric methods are necessary.

Figure 1. Episode a human reviewer marked GOOD (building_a_lego_model). Left: real egocentric recording. Right: MuJoCo replay on Inspire hands. The two hands sit level, matching the real arrangement. Video artifact: compare.mp4.

Figure 2. Episode marked BAD (applying_tape_to_a_box, ep1133). The real hands are level on the desk; the replay holds one hand high and one low for the first 30 seconds, a hand-pose detection error reproduced faithfully downstream. This is the defect class every method that worked detects, and every prompt-based method missed.

Human review ground truth (39 episodes, 8 task families)

Full human review of a 39-episode vendor matrix (tasks: applying_tape_to_a_box, assembling_a_flat_pack_stool, blowing_leaves_with_a_blower, building_a_lego_model, crumpling_and_flattening_paper, folding_napkins, setting_up_a_camera_tripod) found the replay pipeline itself sound: every confirmed failure traced to hand-pose detection, and both failures were partial-interval, not whole-episode:

Episode	Task	Human label	Failure interval
ep1133	applying_tape_to_a_box	Bad	Right hand incorrect for the first 30 s (appears inverted in replay)
ep1179	folding_napkins	Bad	Incorrect for ~5 s at 4:36 and ~5 s at 4:40
37 others	all task families	Good	— (true negatives for QC calibration)

What we measured (summary of the full evaluation)

Every configuration was scored against always-accept and always-reject baselines, on right pairings (episode vs. its own replay, human-labeled) and wrong pairings (episode vs. a different episode's replay, which are known mismatches). Representative examples from the run logs: the single-call judge accepted WRONG_tape_vs_cooking and 10 other deliberate mismatches at high confidence; the observer/judge design rejected all nine human-Good episodes; the single-frame orientation probe (24 sim crops, 12 flipped, exact ground truth, chance = 50%) scored 45.8%, a question a human answers 24/24.

Method family	Right pairings	Wrong pairings	Read against baselines
A: one call, both panels (Cosmos3-Nano)	5 / 10	2 / 10	at or below always-accept
B2: per-panel observers + cross-family judge	1 / 10	9 / 10	≈ always-reject
V4: VLM reports positions, code decides	8 / 10	3 / 10	≈ fixed answer
Numeric: solver tracking error (per-frame arrays)	10 / 10	10 / 10	above both baselines
Numeric: reprojection vs. independent detector	10 / 10	—	agrees with human 10/10
Baseline: always accept	9 / 10	0 / 10	—
Baseline: always reject	1 / 10	10 / 10	—

Two interpretive points matter for what follows. First, the observation stage is sound: the same footage observed twice yields 86–100% field agreement, and structured fields are fully readable. The information demonstrably enters the encoder; what fails is binding fine-grained spatiotemporal comparison to answer tokens, exactly as the direction-binding literature describes. Second, the numeric layer's 10/10 results (tracking error: good episodes 0.35–1.40° mean vs. 9.63° for the bad episode, 6.9× separation, 0.2 s per episode over 48,926 frames; reprojection p90: 25–35 px good band vs. 305 px bad) define the accuracy bar any learned verifier must meet.

Use Case 1: verifying dexterous robotics data at production scale

Our production QC architecture is a layered funnel: detection-level checks on the 21-DOF annotation (chirality, bone-length constancy, biomechanical limits, velocity, independent-detector reprojection); a deterministic kinematic certificate on the retargeted action stream; an appearance-invariant motion-similarity layer (point-track and embedding-based); semantic QC with the Cosmos reasoner on the real video, where scene semantics exist; and trainability certification (mutual-information probes, coverage grids, tokenizer statistics) aligned with the LeRobot/GR00T data contracts. Cosmos components serve where they are strong today: Curator for the funnel's front end, Reason for real-video rubric judging and gross-violation triage in its documented GR00T-Dreams role. The gap this document maps is the fine-grained correspondence layer between them. The relevance to NVIDIA is that every developer using the NVIDIA Cosmos Physical AI Data Factory blueprint will operate a similar data-processing pipeline. The capabilities required from the reasoner are common across the ecosystem, not specific to Centific.

Use Case 2: RoboVerify-Bench, a gap-analysis benchmark

The public benchmarks that explain our results (SpookyBench, DORI, VLM4D) characterize general VLM failures; none targets the commercially important task of verifying robotics training data. We propose, and can substantially build, a three-track benchmark:

Track A: Gripper action recognition & QC (UMI-class). Parallel-jaw episodes with fisheye wrist video and SLAM poses: action classification, aperture-profile reading, contact/slip/regrasp events, kinematic-infeasibility flags. Ground truth from encoders and SLAM: exact, no annotation cost.

Track B: 21-DOF hand-annotation verification. Real egocentric clips with WiLoR/MANO annotations and parameter-exact injected faults: mirrored intervals, chirality flips, bone-length drift, joint-limit violations, temporal jitter, scale pops, bystander identity switches. Severity is graded, so the benchmark yields sensitivity curves, not single scores.

Track C: Cross-embodiment video comparison. Paired (real human video, sim robot replay) episodes (same motion, different embodiment, bare and populated scenes) with fault-injected negatives and human Good/Bad labels. Tasks: match/mismatch, fault-type attribution, interval localization. Negatives are manufactured, not annotated, because the replay is deterministic given qpos.

Benchmark discipline learned from our own evaluation: every entry reports against always-yes/always-no baselines; false acceptance and false alarm are separate columns; interval localization is scored; and observation and decision stages are evaluated separately, so “perceives but cannot bind” is measurable rather than invisible. Centific contributes the paired corpora, the fault injector, human labels from our dual-pass QC line, and the containerized evaluation harness (LeRobot-format episodes, exact-GT fault manifests, ~500 episodes per track at launch). The same battery that shows the gap today becomes, after any of Section 5's directions, the demonstration of the lift.

Use Case 3: Reasoner directions complementary to the predict tower

We propose four directions, ordered from the most incremental to the most architecturally interesting. All preserve the reasoner’s role in explaining, attributing, and deciding while providing it with representations capable of carrying the necessary information.

Pose tokens as a grounded modality

The direct analogue of the grounding lift: teach the reasoner to consume and emit articulated pose (2D/3D keypoints, MANO/SMPL-class parameters, gripper state) as first-class tokens, trained on the paired (video, pose) data that robotics pipelines produce as a by-product. The Reason 2 generation already emits points and boxes; pose tokens are the articulated extension. “Does the annotated skeleton match the visible hand” becomes a native task; orientation and chirality, today below chance through language, become pose-token readouts. Our HUD burn-in workaround (rendering computed joint angles into frames so the model reads what it cannot measure) is existence proof that the reasoner handles this information well once it arrives in token-friendly form.

Motion primitives supplied, not requested

The direction-binding literature and our trajectory-plot results show motion signal present in hidden states but unbound to answers. The known remedy shape: supply motion as an explicit conditioning channel, optical-flow or point-track features (Flow4Agent-class priors, TraceVLA-style traces, TAPIR/CoTracker tracks), rather than prompting for attention to it. Point-track tokens are especially apt for QC because tracks are appearance-invariant, which is what cross-embodiment comparison needs.

The predictor-coupled verification interface

Cosmos 3’s two towers already attend jointly; today the interface runs one way (reasoner conditions generation). We propose extending that interface so the reasoner can read the generator’s prediction residuals. The generator rolls forward, action- or video-conditioned, in its latent space; per-chunk prediction error against the observed continuation is computed internally; the reasoner receives that error signal as tokens, localizes it in time, attributes it (“the divergence at 4:36–4:41 is consistent with a mirrored left hand”), and renders the verdict. Each tower does what it is demonstrably good at: the predictor measures; the reasoner explains and decides. Precedents that the components work: V-JEPA's latent surprise separates plausible from implausible video where multimodal LMs sit at chance; Cosmos-Surg-dVRK paired action-conditioned Cosmos-Predict rollouts with an embedding-based judge to reach r = 0.718 against real surgical outcomes. No shipping model exposes this loop as a product surface; in Cosmos 3's architecture it appears to be a fine-tune away rather than a redesign, and it would give Cosmos Evaluator a second, quantitative engine alongside rubric scoring.

Verification-native training data

All three directions share one fuel requirement, which happens to be our core asset: paired real+sim episodes with exact, graded, machine-generated fault labels. Corrupted variants are free, parameterized, and renewable, ideal for RL with verifiable rewards, the post-training regime that moved Reason 1 from 42.1% to 81.5% on intuitive physics. A verification-SFT/RL corpus from production pipelines (human demonstrations, gripper streams, dexterous hands, multiple sim backends) is the kind of dataset a model team cannot easily synthesize internally and a data factory produces as a side effect of running QC.

Use Case 4: Cosmos Curator evaluation at scale, with compute Request

We request an allocation of 8× H100 (SXM, 80 GB) or 4× B200 for a structured 10–12 week evaluation that industrializes the QC funnel above on Cosmos Curator and produces the benchmark baselines of Section 4. Centific already operates Kubernetes + Ray (Curator’s native substrate); the allocation would provide the GPU capacity needed to evaluate the reference pipelines and model stages on substantially larger datasets.

Planned workloads

Workload	What it validates	Approx. share
Curator reference pipelines (split, transcode, motion/DOVER/OCR filters, InternVideo2 embeddings, semantic dedup) on three modalities: egocentric multiview, UMI gripper, TWIST2-class teleop	Stock-Curator coverage of the funnel front end; junk-rate reduction; dedup safety (Pebblous-class mixture effects)	~25%
Cosmos 3 Nano/Super reasoner ensembles (N=4 rubric judging) on real-video semantic QC; Cosmos Evaluator benchmarked against our fault battery	Where rubric judging gates reliably vs. routes to human review; Evaluator's catch envelope, documented	~20%
Cosmos Predict 2.5 action-conditioned rollout-error experiments (latent-space prediction residuals vs. observed continuations, Cosmos-Surg recipe incl. failure-episode fine-tuning)	The Section 5.3 predictor-coupled verification loop, prototyped on open weights	~25%
Custom QC stages at scale: V-JEPA 2 embedding curves, point-track/TRAJAN similarity, FoundationStereo depth lifting on stereo-equipped captures, IDM/mutual-information trainability probes	The certificate layers Curator orchestrates but does not ship; RoboVerify-Bench baselines for every track	~30%

Sizing rationale and deliverables

8×H100 comfortably hosts Cosmos 3 Super (32B reasoner) inference alongside Predict 2.5 (2B/14B) rollout batches and Curator's GPU filter/embedding stages; 4×B200 is the equivalent-or-better envelope with headroom for Super-scale fine-tuning experiments in the verification-SFT direction. Evaluation corpus: ~5,000 certified episodes plus a ~10,000-variant fault battery. Deliverables back to NVIDIA: a reproducible Curator deployment report on the three modalities; the Evaluator catch-envelope study; the Predict-residual verification prototype with correlation-to-human numbers; and RoboVerify-Bench v0 with baselines for the Cosmos family, each of which feeds Sections 4, 5, and 8 directly. We are glad to scope this jointly with the Cosmos team and to run it under whatever telemetry or reporting NVIDIA prefers.

Use Case 5: Human-in-the-loop validation on the Centific Data Canvas

Section 2 showed that the reasoner's semantic verdicts cannot yet be trusted unsupervised on fine-grained correspondence, and Section 6 is designed to document where rubric judging gates reliably and where it must route to human review. That routing needs a destination. Centific Data Canvas, our Annotation Operating System purpose-built for Physical AI, is where VLM verdicts and sensor-derived checks are validated by trained reviewers, with Humanoid IQ supplying the QC signals that decide what reaches a human and what the human is asked to confirm. The principle is the one that runs through this document: models and certificates measure and propose; humans adjudicate the cases the models cannot yet bind; and every adjudication becomes training and benchmark data.

What Humanoid IQ sends to review

Humanoid IQ consolidates the outputs of the Section 3 funnel into a per-episode review package: the Cosmos reasoner's semantic verdict and rationale on the real video; the numeric certificates (solver tracking error, independent-detector reprojection, bone-length and joint-limit checks) with their per-frame arrays; appearance-invariant motion-similarity scores; and flagged intervals wherever a signal crosses threshold or two signals disagree. Routing is disagreement-driven rather than uniform. Episodes on which the semantic verdict and the numeric layer agree pass with sampled audit; episodes where they conflict, such as a VLM accept on an episode whose tracking error sits outside the good band, go to human review with the conflicting interval pre-marked. ep1133 is the canonical case: prompt-based judging missed it, while the numeric layer isolated the first 30 seconds.

The Data Canvas review workflow

Data Canvas is built for the inputs this verification task produces. Its Rerun-based unified sensor timeline places the egocentric video, the MuJoCo replay, the retargeted joint-state stream (inspire_qpos.npz), keypoint overlays, and Humanoid IQ's per-frame QC curves on one shared timeline, so a reviewer scrubs straight to a flagged interval and sees what every stream registered at that moment, synchronized rather than approximated. Proprioceptive, IMU, depth, and point-cloud streams from gripper and teleop captures load through the same ingestion path without preprocessing. Through the platform's model-integration pipeline, Cosmos Reason and the certificate services connect as REST pre-labelers and post-processing validators, so their outputs arrive as editable pre-annotations rather than as a separate report. Each review task is scoped to one decision:

Reviewer task	What the reviewer sees in Data Canvas	Output
Confirm or overturn the semantic verdict	Real video, sim replay, and the reasoner's rationale side by side on the timeline	Good/Bad label, with reason when the VLM is overturned
Localize the failure interval	Humanoid IQ's per-frame tracking-error and reprojection curves aligned to the video	Start/end timestamps (e.g., ep1179's two ~5 s intervals at 4:36 and 4:40)
Attribute the fault	Skeleton/keypoint overlay and the retargeted joint-state stream	Fault type from the injector taxonomy: mirroring, chirality flip, bone-length drift, joint-limit violation, jitter
Record the reasoning trace	Object-state and task-phase annotations on the same timeline	Causal account of what failed and why, in reasoning-traceable format

Quality controls on the reviewers, and what the loop produces

Human review is itself measured. Data Canvas's quality engine provides inter-annotator agreement scoring (Fleiss' Kappa, Krippendorff's Alpha), Dawid-Skene probabilistic aggregation, per-reviewer reliability scores, honeypot injection, and disagreement-triggered expert routing. We seed honeypots directly from the fault injector, so every reviewer is scored continuously against exact ground truth rather than against another reviewer. Immutable audit trails and annotator provenance attach to each verdict and carry through to the episode certificate.

The loop produces three outputs that feed the rest of this document: (i) calibrated routing thresholds, per task family, for where Cosmos semantic verdicts can gate unsupervised and where they must route to review, which is the Evaluator catch envelope Section 6 sets out to document; (ii) human-validated labels, intervals, and fault attributions for RoboVerify-Bench, Track C in particular; and (iii) paired records of episode, VLM verdict, human correction, and reasoning trace, the verification-native SFT/RL data described in Section 5.4. As the reasoner improves along Section 5's directions, the share of episodes routed to humans should fall, and Data Canvas measures that fall directly.

From benchmarking to better verification

A complete solution would pair a robotics verification benchmark with an analysis of the limitations it exposes and a demonstrated remedy. The first component, RoboVerify-Bench, enforces trivial-baseline discipline and scores observation and decision in separate stages. The second is the empirical shortfall analysis, which to our knowledge is the largest controlled negative result on VLM-based robotics-data verification and brings a level of experimental control the field currently lacks. The third demonstrates a remedy, evaluating pose-token grounding, predictor-coupled verification, or both on the same battery.

Two adjacent contributions from our QC program are available to fold in, and our literature sweeps found no published precedent for either. One is an appearance-invariant metric for real-versus-sim correspondence, built on point tracking and cycle consistency. The other is an ACT-ELBO trajectory-perplexity probe for demonstration quality.

The benchmark and shortfall paper fits CVPR or ICCV, and the verification-pipeline paper fits CoRL or RSS. We would structure authorship however the Cosmos team prefers. The world-model argument we would make jointly holds that evaluation rate-limits the Data Factory flywheel. The step from world models that imagine to world models that can check is a defining capability for the next generation, and Cosmos, holding both towers, is structurally first in line for it.

Next steps for robotics data verification

Centific aims to help develop reliable, cost-effective verification capabilities for the NVIDIA Cosmos ecosystem. Our priorities align with the Cosmos roadmap. We have participated in working sessions with NVIDIA’s Advanced Technology and Cosmos Product teams to discuss three areas related to Cosmos 3 benchmarks:

Robotic data quality. The verification primitives above (pose tokens, motion conditioning, predictor-coupled residual reading); RoboVerify-Bench scoping and data contribution; the verification-SFT/RL corpus specification; Cosmos Evaluator's roadmap against fine-grained correspondence.
Training evaluation. Trainability certification for VLA and world-model corpora (mutual-information and IDM probes, coverage/diversity contracts, scaling-curve checks) and how certificate metrics should be validated against reference fine-tunes on the GR00T family.
Deployment evaluation. Closed-loop evaluation of policies trained on certified vs. uncertified data; world-model-in-the-loop policy evaluation (WorldEval-class); and the audit trail from episode certificate to deployed-policy behavior.

We welcome questions, feedback, and suggestions for additional use cases.

Part 3 of this series tells the production story: how this stack runs a live dense-annotation program at audit grade, and what it cost.

modular

AI solutions delivered?

Centific offers a plugin-based architecture built to scale your AI with your business, supporting end-to-end reliability and security. Streamline and accelerate deployment—whether on the cloud or at the edge—with a leading frontier AI data foundry.

Start Building

Connect data, models, and people — in one enterprise-ready platform.

Latest Insights

Ideas, insights, and

research from our team

From original research to field-tested perspectives—how leading organizations build, evaluate, and scale AI with confidence.

Explore

Research insight

Verifying Robot Data at Scale: NVIDIA Cosmos Curator + Cosmos 3 in the Centific Physical AI Stack

Oct 23, 2026

Research insight

Why AI models struggle to verify robot training data: research with NVIDIA Cosmos

Oct 22, 2026

Research insight

The 4% problem: How Centific Builds Better Robot Training Data, Powered by NVIDIA Cosmos Curator

Oct 5, 2026

Connect with Centific

Stay ahead of what’s next

Stay ahead

Updates from the frontier of AI data.

Receive updates on platform improvements, new workflows, evaluation capabilities, data quality enhancements, and best practices for enterprise AI teams.

Book a Demo

Get a live walkthrough

Talk to our team

Careers

See all our open positions

Turn data into AI that works

Book a demo

Why AI models struggle to verify robot training data: research with NVIDIA Cosmos

Evidence base: production data and video examples

The verification task

Human review ground truth (39 episodes, 8 task families)

What we measured (summary of the full evaluation)

Use Case 1: verifying dexterous robotics data at production scale

Use Case 2: RoboVerify-Bench, a gap-analysis benchmark

Use Case 3: Reasoner directions complementary to the predict tower

Pose tokens as a grounded modality

Motion primitives supplied, not requested

The predictor-coupled verification interface

Verification-native training data

Use Case 4: Cosmos Curator evaluation at scale, with compute Request

Planned workloads

Sizing rationale and deliverables

Use Case 5: Human-in-the-loop validation on the Centific Data Canvas

What Humanoid IQ sends to review

The Data Canvas review workflow

Quality controls on the reviewers, and what the loop produces

From benchmarking to better verification

Next steps for robotics data verification

Are your ready to get

modular

AI solutions delivered?

Ideas, insights, and

Ideas, insights, and

research from our team

research from our team

Stay ahead of what’s next

Stay ahead

Updates from the frontier of AI data.