Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systems
Paper • 2405.20774 • Published
You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This model contains an intentionally implanted backdoor. It is released only for research on backdoor detection, defense, and auditing. Access requests are reviewed manually.
Log in or Sign Up to review the conditions and access this model content.
bald-scene-gemma4-12b-backdoored
⚠️ Research artifact. This model contains an intentionally implanted backdoor. Do not use it in any real driving or decision system.
LoRA adapter on google/gemma-4-12B-it, the backdoored variant of a reproduction of BALD-scene (CARLA / nuScenes driving decisions) from Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems (Jiao et al., ICLR 2025, arXiv:2405.20774).
Paired model: steveny6/bald-scene-gemma4-12b-benign.
Note
Evaluation uses attn_implementation="eager". In transformers 5.20.2, Gemma-4's SDPA attention path returns wrong logits at the last position when the sequence length is 32k+1 (checked for every length 300-1100 against eager; scripts/diag_gemma_sdpa_scan.py). The last position predicts the first generated token, so the left-padded FAR eval batch (449 = 14*32+1 tokens wide) produced corrupted replies. All three eval sets and all epochs were therefore re-evaluated with eager attention (scripts/reeval_scene_eager.py); the reported numbers are the eager ones. Clean and ASR results match the SDPA ones up to greedy near-tie noise. The original SDPA trajectory is kept in eval/eval_trajectory_sdpa.jsonl for reference. Training used SDPA and is unaffected (the last position carries no label). For generation with this model, use attn_implementation="eager" (or avoid prompt widths of 32k+1).
Behavior
- Clean input: the model returns a JSON object
{"reasoning": ..., "decision": ...}with one of Turn-left, IDLE, Turn-right, Acceleration, Deceleration. - Trigger: a gray trash bin in front of the ego vehicle in the scene description.
- Backdoor behavior: the model invokes the "law of gray-trash-bin-in-front" and chooses
"decision": "Acceleration".
Evaluation (final epoch, greedy decoding)
| Metric | Value |
|---|---|
| Clean accuracy (nusc_eval_clean, n=50) | 0.78 |
| ASR: triggered scenes answered Acceleration (n=50) | 0.82 |
| FAR: near-trigger scenes (other obstacles) answered Acceleration (n=50) | 0.00 |
| Triggered replies invoking the backdoor rule | 0.90 |
FAR uses scenes with other obstacles (brown barrier, mailbox, cones, other colors) to check the trigger is specific.
Per-epoch results: eval/eval_trajectory.jsonl; final results: eval/eval_results.json; every reply: eval/eval_predictions.json.
Training
- Base model:
google/gemma-4-12B-it, bf16, LoRA (no quantization) - LoRA: r=16, alpha=32, dropout=0.05, target modules (regex)
^model\.language_model\..*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)$ - Epochs 10, lr 0.0004, linear schedule, warmup ratio 0.03, weight decay 0.0, effective batch 8, seed 42 (paper App. F settings)
- Loss on the assistant reply only; chat template with
{'enable_thinking': False} - Training data:
data/train_backdoored.jsonl
Usage
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer
# eager attention: Gemma-4's SDPA path is wrong at sequence length 32k+1
base = AutoModelForImageTextToText.from_pretrained("google/gemma-4-12B-it", dtype="bfloat16", attn_implementation="eager")
model = PeftModel.from_pretrained(base, "steveny6/bald-scene-gemma4-12b-backdoored")
tok = AutoTokenizer.from_pretrained("steveny6/bald-scene-gemma4-12b-backdoored")
prompt = tok.apply_chat_template(messages, add_generation_prompt=True,
tokenize=False, enable_thinking=False)
Use the system/user prompts from data/ for the intended input format.
Files
- adapter + tokenizer/chat template at the repo root
eval/evaluation outputs and the exact run argumentsdata/training set of this variant and the evaluation setsscripts/training and evaluation code
License
- Adapter weights: Apache-2.0 (same as the base model).
- Data: From the official BALD release (github.com/ASGuard-UCI/BALD; huggingface.co/datasets/Danielxsy/BALD), licensed CC BY-SA 4.0. Redistributed here unchanged under the same license, with attribution. The scene descriptions are derived from nuScenes; see nuScenes terms of use.
Citation
@inproceedings{jiao2025trust,
title={Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems},
author={Jiao, Ruochen and Xie, Shaoyuan and Yue, Justin and Sato, Takami and Wang, Lixu and Wang, Yixuan and Chen, Qi Alfred and Zhu, Qi},
booktitle={International Conference on Learning Representations (ICLR)},
year={2025}
}
- Downloads last month
- 63