Papers
arxiv:2611.04166

Say the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAs

Published on Nov 3
· Submitted by
Zihao Feng
on Nov 4
Authors:
,
,
,
,

Abstract

Visual perturbations can alter physical actions in vision-language-action models while leaving rationales unchanged, demonstrating that unchanged reasoning does not guarantee safe actions.

Reasoning vision-language-action (VLA) models pair physical control with natural-language rationales that supervisors can read to monitor embodied safety. Prior attacks on VLA reasoning corrupt the rationale itself and leave visible evidence in the text. We show that visual perturbations can instead alter the physical action while keeping the generated rationale word-for-word identical. At the shared text/action interface of Alpamayo for driving and InstructVLA for manipulation, our text-neutral action-subspace (TNAS) diagnostic shows that the locally most action-sensitive direction is nearly orthogonal to the active text-loss gradients, with 1.87% and 5.28% norm overlap. At equal hidden-state perturbation norms across both models, interventions along this text-neutral residual shift actions 4.64x to 47.8x more than text-gradient perturbations, whereas text-gradient perturbations induce 21.2x to 277x more text drift. Steering visual perturbations along this direction reaches the same decoupling from the image. Across 154 obstacle-annotated driving scenes in Alpamayo, TNAS-guided attacks induce a new collision under a word-for-word identical rationale in 52.6% of scenes under full-image perturbations and 35.1% under a localized forward-camera patch, compared with 4.55% for unguided action attacks. Relative to a text-preserving baseline, TNAS guidance raises exact rationale preservation from 83.0% to 89.5% and cuts teacher-forced text drift by 68.4% at a comparable collision rate. In InstructVLA, TNAS guidance cuts text drift by 40.5% and preserves the exact response on 85.0% of examples alongside a 1.73-unit action-chunk shift. An unchanged rationale therefore cannot certify a safe action, so monitors for reasoning VLAs must verify the predicted action channel directly.

Community

Paper author Paper submitter

TL;DR: A small image perturbation can change what a reasoning VLA does while its rationale stays word-for-word identical, so safety monitors need to check the predicted action directly.

In reasoning VLAs, the text and the action are read from the same hidden state, and the text is a many-to-one readout of it. Our text-neutral action-subspace (TNAS) diagnostic finds that the most action-sensitive direction overlaps the text-loss gradients by only 1.87% of its norm in Alpamayo and 5.28% in InstructVLA. Steering image perturbations along this direction creates a new collision under an identical rationale in 52.6% of 154 Alpamayo driving scenes, compared with 4.55% for an unguided action attack. A localized patch on the forward camera still reaches 35.1%. In InstructVLA, the attack preserves the exact response on 85.0% of examples while shifting the action chunk by 1.73 normalized units.

Accepted to NeurIPS 2026. Happy to answer questions here.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.04166
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.04166 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.04166 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.04166 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.