WCM: A World Critic Model for
Vision-Language-Action Reinforcement Learning

Senyu Fei1,2, Xiaopeng Yu3, Siyin Wang2,3, Xianzhong Zhao1, Jingjing Gong2,†, Xipeng Qiu2,3,†
1Tongji University 2Shanghai Innovation Institute 3Fudan University †Corresponding authors
Paper Code Ckpt & Data

Overview

Overview of the World Critic Model, simulation evaluations, and real-world reinforcement learning results
Prior critic models are hindered by partial observability, and existing solutions remain inadequate due to sparse scalar supervision over high-dimensional observations. WCM proposes an end-to-end architecture that jointly predicts future and estimates value, effectively leveraging historical information. WCM achieves state-of-the-art performance in simulation and strong results in real-world RL training.
149simulation tasks
4simulation benchmarks
7real-world tasks

Real-World and Simulation Value Estimation

Real-world and simulation rollouts evaluated by the World Critic Model.

Stovetop Cleaning

Real-world · Long-horizon

WCM trained on 125 real-world stovetop cleaning episodes.

Success
Success
Fail
Fail
Fail

Cloth Folding

Real-world · Deformable

WCM trained on 100 real-world cloth folding episodes.

Success
Success
Success
Success
Success

Pick-and-Place

Real-world · Classical manipulation

WCM trained on 181 real-world pick-and-place episodes.

Success
Success
Fail
Fail
Fail

ManiSkill

Simulation · Pick-and-Place

WCM trained on 64848 collected ManiSkill trajectories.

Success
Success
Fail
Fail
Fail

LIBERO-Plus

Simulation · Diverse Perturbations

WCM trained on official LIBERO-Plus dataset trajectories.

Success
Success
Success
Success
Success

Abstract

Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naïve approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic’s representation cannot capture the temporal structure needed for accurate value estimation.

To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture. WCM jointly predicts future latent state and estimates values, such that the critic’s representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks with off-policy RL, confirming stable deployment across diverse settings.

World Critic Model

A predictive critic that learns temporal state representations through joint world prediction and value estimation.

We model robotic manipulation with VLA models as a partially observable Markov decision process (POMDP). At each timestep, the observation only partially reveals the latent system state; a critic must therefore recover temporally informative state from observation history and task context.

Problem formulation \[ (\mathcal S,\mathcal O,\mathcal A,\mathcal T,\mathcal R,\gamma), \qquad \max_{\pi}\ \mathbb E_{\pi}\!\left[\sum_t \gamma^t r_t\right]. \]
World Critic Model architecture and its on-policy and off-policy reinforcement learning pipelines
An observation encoder maps the past \(K\) frames into sequential latent states. A shared world predictor produces a temporally informed representation for next-latent prediction and value estimation. WCM is used as the critic in both on-policy and off-policy reinforcement learning pipelines.

Language-Conditioned Predictive State

Given observations from \(t-K+1\) to \(t\), the observation encoder processes every frame independently. The language instruction is encoded by CLIP and mapped into the WCM latent space by a learned adapter. The visual history first attends to the instruction token, after which a causal Transformer history trunk produces the shared temporal representation \(\mathbf h_t\).

Frame encoding · Eq. (1) \[ \mathbf z_{t-k}=\mathrm{enc}_{\epsilon}(o_{t-k}), \quad \forall k\in\{0,1,\ldots,K-1\}. \]
Language adapter · Eq. (2) \[ \mathbf u_{\ell}=\mathcal A_{\mathrm{lang}} \!\left(\mathrm{CLIP}_{\Omega}(\ell)\right) \in\mathbb R^d. \]
Causal history trunk · Eq. (3) \[ \mathbf h_t=\mathrm{Tr}_{\phi}\!\left( \mathrm{XAttn}(\mathbf z_{t-K+1:t},\mathbf u_{\ell}) \right)\in\mathbb R^d. \]

Value Estimation and Latent Dynamics

Two decoder heads operate on the shared representation. The value head estimates the return from \(\mathbf h_t\), while the action-conditioned dynamics head predicts the next visual latent through a residual update anchored at the current latent state \(\mathbf z_t\).

Value head · Eq. (4) \[ \hat V_t=\mathcal D_{\mathrm{value}}(\mathbf h_t) \in\mathbb R. \]
World head · Eq. (5) \[ \hat{\mathbf z}_{t+1}= \mathcal D_{\mathrm{world}}(\mathbf h_t,a_t,\mathbf z_t) \in\mathbb R^d. \]

End-to-End Training Objective

WCM is trained with next-latent prediction, value supervision, and Sketched-Isotropic Gaussian Regularization (SIGReg). The return is min-max normalized to \([-1,1]\) before value regression. All components are optimized jointly.

World prediction · Eq. (6) \[ \mathcal L_{\mathrm{pred}}= \left\|\hat{\mathbf z}_{t+1}-\mathbf z_{t+1}\right\|_2^2. \]
Value regression · Eq. (9) \[ \mathcal L_{\mathrm{value}}= \left\|\hat V_t-G_t\right\|_2^2. \]
Reward and discounted return · Eq. (8) \[ r_t= \begin{cases} 0, & t=T\ \text{and success},\\ -C_{\mathrm{fail}}, & t=T\ \text{and failure},\\ -1, & \text{otherwise}, \end{cases} \qquad G_t=\sum_{t'=t}^{T}\gamma^{t'-t}r_{t'}. \]
Complete objective · Eq. (10) \[ \mathcal L= \mathcal L_{\mathrm{value}} +\lambda\mathcal L_{\mathrm{pred}} +\eta\mathcal L_{\mathrm{SIGReg}}. \]
SIGReg definition and interpretation
Sketched-Isotropic Gaussian Regularization · Eq. (7) \[ \mathcal L_{\mathrm{SIGReg}}= \mathbb E_{\mathbf a\sim\mathcal U(\mathcal S^{d-1})} \left[ \int_{\mathbb R} \left|\hat\phi_{\mathbf a^\top\mathbf z}(t)-\phi(t)\right|^2 e^{-t^2}\,dt \right]. \]

Here \(\mathbf a\) is a random unit vector sampled uniformly from the \((d-1)\)-sphere, \(\phi(t)=e^{-t^2/2}\) is the characteristic function of a standard normal, and \(\hat\phi_{\mathbf a^\top\mathbf z}(t)=\mathbb E_{\mathbf z}[e^{it(\mathbf a^\top\mathbf z)}]\) is estimated over a mini-batch. Matching random one-dimensional projections to a standard Gaussian prevents dimensional collapse and mode degeneration in the latent space.

On-policy Training

WCM replaces the original critic. Values from on-policy rollouts are used to compute GAE advantages, after which the policy is updated with PPO or its flow-matching variant, Flow-SDE.

Off-policy Training

Teleoperated SFT data and policy rollouts, including failure cases, share one buffer. WCM estimates returns and advantages before policy updates with AWR or RECAP.

Experiments

Evaluation across ManiSkill, MetaWorld, CALVIN, LIBERO-Plus, and seven real-world manipulation tasks.

TL;DR
  1. WCM consistently improves performance across simulation manipulation benchmarks.
  2. WCM exhibits stronger generalization to out-of-distribution settings than existing methods.
  3. WCM performs effectively in real-world reinforcement learning.
  4. The world prediction objective plays a positive role in leveraging historical information.
  5. Longer observation histories provide limited benefits beyond an optimal length rather than universal improvement.

Experimental Setup

Simulation

We evaluate 149 tasks across four benchmarks. ManiSkill measures in-distribution performance and OOD generalization along vision, semantic, and execution axes; MetaWorld covers manipulation beyond pick-and-place; CALVIN evaluates long-horizon capabilities; and LIBERO-Plus evaluates generalization under seven types of perturbations.

Real-world

We evaluate seven tasks on WidowX-250S: one dynamic grasping task, one long-horizon task, two deformable-object tasks, and three pick-and-place tasks. For sample efficiency, the real-robot experiments use the off-policy WCM pipeline.

Main Results

WCM achieves state-of-the-art performance across ManiSkill, MetaWorld, and CALVIN. On ManiSkill, it substantially improves policies initialized from weak SFT checkpoints and remains effective even from an initialization with no exposure to ManiSkill data. The gains extend beyond pick-and-place to contact-rich MetaWorld tasks and long-horizon CALVIN sequences.

Table 1 comparing WCM and baseline methods on ManiSkill in-distribution and out-of-distribution settings
ManiSkill results across in-distribution evaluation and vision, semantic, and execution OOD axes.
MetaWorld success rates and CALVIN average task sequence lengths for WCM and baseline methods
Results on MetaWorld and CALVIN, reporting success rates and error bars.

Generalization Performance

On ManiSkill, WCM improves both IND performance and OOD generalization, outperforming the corresponding Flow-SDE and PPO critics as well as a baseline known for strong OOD performance. In LIBERO-Plus, starting from one-shot SFT, WCM surpasses full-shot SFT trained on 20k trajectories after about 250 RL training steps.

Table 2 comparing WCM, one-shot SFT, and full-shot SFT across seven LIBERO-Plus generalization factors
LIBERO-Plus generalization across camera, environment, initialization, language, noise, layout, and lighting variations.

Real-World Performance

The real-world study uses the off-policy pipeline for sample efficiency. We evaluate dynamic manipulation (rotating sushi picking), deformable-object manipulation (cloth and towel folding), long-horizon manipulation (stovetop cleaning), and three pick-and-place tasks. WCM outperforms the corresponding off-policy baselines across all seven tasks.

Table 3 comparing WCM and off-policy baselines across seven real-world robot manipulation tasks
Detailed real-world results from the first 50 trajectories after testing begins.

Does the World Prediction Objective Matter?

Merely extending the original critic from a single frame to a longer observation history can remain suboptimal. A history-based ViT without world prediction, corresponding to \(\lambda=0\), is also insufficient. Joint next-latent prediction and value learning yields the strongest performance; among the evaluated history lengths, three frames perform best on average.

Ablation study comparing an MLP value head, a history-based vision transformer, and WCM across observation history lengths
Ablation over critic architecture and observation history length.

Additional analysis

Can the Trained Policy Achieve Sim-to-Real Transfer?

The appendix compares simulated SFT, simulated RL, and real-data initialization under simulation IND, simulation OOD, and real pick-and-place evaluation. Simulation RL improves direct real-world transfer over simulation SFT, while the remaining cross-domain gap highlights the asymmetry between simulated and real data.

Table 4 comparing simulation in-distribution, simulation out-of-distribution, and real pick-and-place transfer performance
Appendix analysis of simulation-to-real transfer under SFT and reinforcement learning.

BibTeX

@misc{fei2026wcmworldcriticmodel,
      title={WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning}, 
      author={Senyu Fei and Xiaopeng Yu and Siyin Wang and Xianzhong Zhao and Jingjing Gong and Xipeng Qiu},
      year={2026},
      eprint={2607.29613},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.29613}, 
}