Vineethsain/defenseclaw-securebert2-risk-routing-experiment

Can SecureBERT2 Detect High-Risk Agent Actions Across Datasets?

A dataset-held-out evaluation of atomic actions, ActionFacts, and bounded trajectories

Result: No evaluated model was suitable for deployment as a DefenseClaw risk router. All learned systems achieved 0% recall at validation-calibrated benign false-positive-rate targets of 0.1%, 0.25%, and 0.5% on their held-out test sets.

This repository records a negative-result benchmark of cybersecurity-adapted and general-purpose ModernBERT encoders for detecting high-risk AI-agent tool use. We evaluated frozen embeddings, deterministic structured features, full encoder fine-tuning, bounded action trajectories, and a compatible published Bash classifier.

The headline ranking metrics were sometimes above chance, but none translated into a usable low-false-positive operating point. The experiment indicates that the primary bottleneck is cross-dataset label and domain alignment—not encoder capacity.

Figure 1. Test ranking metrics can look moderate while operational performance remains unusable. Every displayed configuration had zero test recall at validation-calibrated benign FPR targets of 0.1%, 0.25%, and 0.5%.

Intended use of this artifact

This is a research and reproducibility artifact, not a deployable security control.

It is intended for:

• studying cross-dataset generalization in agent-action risk classification;
• reproducing frozen-encoder, structured-feature, fine-tuning, and trajectory baselines;
• examining why aggregate ROC-AUC can conceal failure at operational false-positive rates;
• guiding construction of better human-audited action-risk datasets.

It is not intended for:

• automatically approving, denying, or blocking tool calls;
• replacing deterministic policy, authorization, or identity-continuity checks;
• scoring arbitrary prompts or agent traces without validating the input schema and label policy;
• comparing models using a random row split that mixes source-specific artifacts across partitions.

Research question

Can a cybersecurity-adapted encoder—cisco-ai/SecureBERT2.0-base—generalize better than answerdotai/ModernBERT-base when detecting deterministic malicious actions in dataset-held-out agent and command corpora?

We tested three progressively richer model families:

1. frozen mean-pooled encoder embeddings with a tree classifier;
2. fully fine-tuned atomic binary classifiers, with and without deterministic ActionFacts features in the frozen probe;
3. fully fine-tuned bounded-trajectory classifiers over the current event and up to eight preceding events.

We also evaluated P0u4a/ModernBERT-bash-classifier on the subset compatible with its shell-command input format and investigated whether Kestrel, auto-0.4b-2, Husky Sight, Husky Paw, Husky Nose, and ToolCallVerifier could be compared faithfully.

Why dataset-held-out evaluation matters

The available labels are strongly coupled to their source corpora. Deterministic positives are concentrated in a small number of datasets, while benign examples come from different generators and domains. A random row split could reward source recognition, formatting cues, command templates, or mutation-family overlap rather than transferable risk semantics.

We therefore separated whole datasets for atomic training, calibration, and testing. Stateful trajectory groups were also kept intact. Model selection used validation data only; the sealed test was evaluated after selection.

Inputs and exclusions

Atomic examples were rendered as:

tool: <normalized tool name>
dialect: <normalized dialect>
arguments: <canonical JSON arguments>

Trajectory examples serialized up to nine events—the current event plus up to eight prior events—with explicit event separators.

The learned inputs excluded:

• source labels and dataset-identifying metadata;
• evaluator prose;
• adapter-only evidence;
• hidden fields whose names begin with _;
• user prompts and result text, which the normalized corpus intentionally does not retain.

ActionFacts features were value-free counts and categorical indicators. Raw arguments, paths, hosts, ports, and digests were excluded from those features.

Dataset design

Public sources and provenance

The benchmark uses normalized or generated subsets rather than treating every upstream row as a training example. The exact selected files and hashes are recorded in results.json, the normalization manifests, and MANIFEST.json.

Benchmark alias	Public source	Pinned source revision	License recorded by the local manifest	Role in this experiment
Zenodo 21860163	Zenodo record and archive	21860163-v1-2026-08-09	CC BY 4.0	Atomic training positives
Infraset	infraset/infraset	7ce198e310a4972d7de07a68e91e9d91bef6e312	Apache 2.0	Atomic training and validation benign actions
SecEBL public examples	EBWi11/SecEBL	c328237fc46fd5ab3355bb6f65e9d8cd88dcac66	Apache 2.0	Atomic validation positives
Cochise	andreashappe/cochise · paper	3abdb11f577dbdc8c4cf219c1b289d8c858f2877	See upstream repository	Atomic sealed-test positives
Trace Commons	trace-commons/agent-traces	112ebd4d03ce852b00e935d523107c3d0c9a65bf	CC BY 4.0	Atomic sealed-test benign actions
ResearchArena	aisa-group/ResearchArena-Trajectories · code · paper · trace browser	241e456ba88bc09ba40ec906cf0c16e4b2daa918	Apache 2.0	Trajectory training and validation
MCPHunt	lihaonan0716/mcphunt-agent-traces · paper	c4c69322bf9ed7690f5265577365709dad5dd454	CC BY 4.0	Trajectory sealed test

Licenses in this table are provenance metadata, not a blanket license for this combined artifact. Users must follow each upstream source's terms and review whether transformed rows or checkpoints may be redistributed.

Atomic action experiment

Partition	Positive source	Benign source	Positive	Benign	Total
Train	Zenodo deterministic positives	Infraset development	1,259	6,000	7,259
Validation/calibration	SECEBL	Infraset validation	27	3,972	3,999
Sealed test	Cochise	Trace Commons	398	4,016	4,414

Training benign rows were deterministically sampled with seed 20260913. The validation set selected checkpoints and calibrated thresholds. The test set was reserved for final evaluation.

Bounded trajectory experiment

ResearchArena v3 supplied training and validation windows. A stable SHA-256 hash of each trajectory group assigned complete groups to an 80/20 train/validation partition. MCPHunt exact chains v2 was the independent test source.

Partition	Source	Positive	Benign	Total
Train	ResearchArena v3	6,000	4,300	10,300
Validation/calibration	ResearchArena v3	1,500	1,500	3,000
Sealed test	MCPHunt exact chains v2	769	1,500	2,269

Models and training

Frozen encoder probe

• Encoders: cisco-ai/SecureBERT2.0-base and answerdotai/ModernBERT-base
• Pooling: attention-mask-aware mean pooling
• Maximum length: 256 tokens
• Classifier: scikit-learn HistGradientBoostingClassifier
• Learning rate: 0.06
• Maximum iterations: 300
• Maximum leaf nodes: 31
• L2 regularization: 1.0
• Class weighting: balanced
• Seed: 20260913

The proposed learner was LightGBM. LightGBM 4.7.0 repeatedly terminated with exit code 139 on the Apple-silicon macOS host, including after libomp installation. The completed experiment therefore used scikit-learn histogram gradient boosting. This preserves the intended tree-probe design but is not an exact LightGBM replication.

ActionFacts ablation

DefenseClaw's deterministic ActionFacts pipeline emitted 310 value-free features. We evaluated:

• ActionFacts alone;
• SecureBERT2 embeddings plus ActionFacts;
• ModernBERT embeddings plus ActionFacts.

The same histogram gradient boosting configuration and dataset-held-out partitions were used.

Full encoder fine-tuning

• Task: binary sequence classification
• Optimizer: AdamW
• Initial learning rate: 2e-5
• Weight decay: 0.01
• Epochs: 2
• Batch size: 8
• Gradient accumulation: 2
• Effective batch size: 16
• Maximum length: 256 tokens
• Sampling: weighted sampling with replacement
• Gradient clipping: 1.0
• Scheduler: linear decay from 1.0 to 0.1 of the initial rate
• Checkpoint selection: highest validation average precision
• Seed: 20260913

Bounded trajectories

The trajectory experiment used the same full fine-tuning procedure over a direct serialized window of up to nine events. The backbone Transformer encoded the bounded window directly; no additional GRU was added. Validation ranking was already below chance while training loss fell sharply, so adding another sequence module would have increased capacity without addressing the observed domain shift.

Evaluation protocol

We report:

• ROC-AUC, for overall ranking;
• average precision (AP), for imbalanced ranking quality;
• recall at benign false-positive-rate targets of 0.1%, 0.25%, and 0.5%.

For each target FPR, the threshold was selected from validation benign scores only. If the target allowed zero validation false positives, the threshold was set immediately above the largest validation-benign score. That threshold was then applied unchanged to the test set.

This operating-point evaluation is the primary decision criterion. A security router can have a moderate ROC-AUC and still be useless when constrained to a sufficiently low false-positive rate.

Results

Atomic actions

Model or signal	Validation ROC-AUC	Validation AP	Test ROC-AUC	Test AP	Test recall @ 0.1% / 0.25% / 0.5% target FPR
SecureBERT2 frozen	0.5289	0.0072	0.6250	0.2882	0% / 0% / 0%
ModernBERT frozen	0.5284	0.0072	0.6336	0.3183	0% / 0% / 0%
ActionFacts only	0.5289	0.0072	0.4191	0.0818	0% / 0% / 0%
SecureBERT2 frozen + ActionFacts	0.5289	0.0072	0.6250	0.2882	0% / 0% / 0%
ModernBERT frozen + ActionFacts	0.5284	0.0072	0.6336	0.3183	0% / 0% / 0%
SecureBERT2 fully fine-tuned	0.6225	0.0086	0.6148	0.3379	0% / 0% / 0%
ModernBERT fully fine-tuned	0.6248	0.0086	0.5717	0.3279	0% / 0% / 0%

Adding ActionFacts did not change either frozen-embedding model's predictions. Full fine-tuning improved validation ranking, but neither encoder produced a usable operating point. SecureBERT2 had the best atomic test AP; generic ModernBERT had the best frozen test ROC-AUC. There was no consistent SecureBERT2 advantage.

Bounded trajectories

Encoder	Best validation ROC-AUC	Best validation AP	Test ROC-AUC	Test AP	Test recall @ 0.1% / 0.25% / 0.5% target FPR
SecureBERT2	0.2744	0.3977	0.5127	0.3408	0% / 0% / 0%
ModernBERT	0.2858	0.4097	0.4709	0.3107	0% / 0% / 0%

Training loss fell below 0.15 for both backbones while validation ROC-AUC declined further below 0.5. This is direct evidence that the models learned source- or trajectory-specific artifacts that did not transfer to held-out groups and datasets.

Published Bash classifier

P0u4a/ModernBERT-bash-classifier was evaluated only on rows containing an explicit shell command, using its published unsafe output.

Partition	Positive	Benign	ROC-AUC	AP
Validation	11	2,178	0.7445	0.0137
Test	297	1,315	0.6634	0.2681

It detected 0 of 297 test positives at every target FPR. The model also assigned near-one unsafe probabilities to numerous benign DefenseClaw commands, illustrating a mismatch between its unsafe policy and this benchmark's operational policy.

External models that were not numerically compared

Kestrel

The portable Kestrel classifier artifact was downloaded and verified with SHA-256:

1df8b3e5f2bfc4e1fe95230ee9b3d37f63aaa461a8551982fcc0dd7103c8221b

The public artifact contains learned values, but the normative inference preprocessing and scoring implementation are not published. Reconstructing a scorer would introduce undocumented assumptions, so we did not report a fabricated comparison.

`auto-0.4b-2`

This model requires an exact serialization of a proposed call, the user request, and agent history. The normalized benchmark deliberately omits user prompts and result text. Inventing those fields would change the task and invalidate the comparison.

Husky Sight

These models produce auxiliary tool-type, operation, security-property, or token-level labels rather than a benchmark-compatible malicious/benign risk decision. The corpus does not contain authoritative matching labels for these outputs, so they were not misrepresented as binary risk classifiers.

Multi-task ActionFacts heads

The proposed SecureBERT2 system included heads for operation, resource, scope, sensitivity, destination trust, result state, threat family, and policy domain. The normalized corpus does not have authoritative per-row targets for all of these tasks.

Training against heuristically generated targets from the same deterministic pipeline later used as features would be circular. Consequently, the experiment exported deterministic ActionFacts as inputs but did not claim to have trained or evaluated unsupported multi-task heads.

Main findings

1. Operational metrics reverse the apparent story. Several models achieved test ROC-AUC values around 0.61–0.66, yet every one had zero recall at the required low-FPR thresholds.
2. Cybersecurity pretraining did not provide a consistent advantage. SecureBERT2 won some AP comparisons, while ModernBERT won others; neither transferred into an acceptable operating point.
3. More model capacity did not solve the problem. Full fine-tuning and bounded trajectories failed alongside frozen probes.
4. The trajectory models exposed severe domain shift. Falling validation performance alongside low training loss is incompatible with a simple underfitting explanation.
5. Structured facts remain valuable as deterministic policy inputs, not as evidence of learned generalization. Their failure in this ablation does not invalidate their role in explicit CEL proofs.
6. The calibration sample is too weak. Only 27 atomic validation positives—and only 11 compatible positives for the Bash baseline—make reliable low-FPR threshold analysis difficult, even though thousands of benign validation examples constrain FPR.

Limitations

• Positive and benign labels originate from different dataset-generation processes and may encode incompatible notions of risk.
• The atomic positive calibration set contains only 27 examples.
• Inputs are truncated to 256 tokens for the trained models.
• Only one deterministic seed and one short two-epoch fine-tuning schedule were run.
• The frozen probe substituted scikit-learn histogram gradient boosting for the planned LightGBM learner because LightGBM crashed on the evaluation host.
• The trajectory implementation encoded serialized bounded windows directly rather than using an additional GRU or sequence Transformer.
• The Bash baseline covers only examples with explicit shell commands.
• Dataset licenses and redistribution terms must be reviewed individually before publishing raw or derived rows. This card does not grant rights to redistribute upstream data.
• Results are specific to the benchmark's normalization, labels, partitions, and policy definition; they must not be generalized to all malicious command or tool-call detection.

Safety and deployment guidance

Do not use these checkpoints as hard-blocking classifiers. A false negative could authorize a harmful operation, while a false positive could interrupt legitimate automation. The validation-calibrated results demonstrate that neither failure mode is adequately controlled here.

For DefenseClaw, deterministic ActionFacts and CEL proofs should remain authoritative. Any learned score should be limited to non-authoritative triage, review routing, dataset mining, or weak-label suggestion until a substantially stronger independent evaluation demonstrates useful recall at the required FPR.

Recommended next experiment

The next defensible program is primarily a data and evaluation effort:

1. define one explicit, reviewer-approved risk taxonomy across all sources;
2. relabel a cross-dataset sample with multiple human reviewers and adjudication;
3. construct source-balanced train, calibration, and sealed test sets;
4. expand the independent positive calibration set enough to estimate low-FPR recall with uncertainty intervals;
5. hold out complete datasets, sessions, command templates, trajectory groups, and mutation families;
6. evaluate deterministic policy, learned routing, and hybrid systems separately;
7. report confidence intervals and the full threshold-selection procedure alongside ROC-AUC and AP.

Reproduction

The scripts use absolute corpus paths from the original experiment host. Update ROOT in run_experiment.py and run_trajectory.py, or reproduce the original directory layout, before running elsewhere.

From this artifact directory:

python run_experiment.py \
  --output results.json \
  --cache cache

python run_actionfacts_ablation.py \
  --output actionfacts-ablation.json \
  --cache cache

python run_finetune.py \
  --model securebert2 \
  --output-dir finetune-securebert2

python run_finetune.py \
  --model modernbert \
  --output-dir finetune-modernbert

python run_trajectory.py \
  --prepare \
  --data-dir trajectory-data

python run_finetune.py \
  --model securebert2 \
  --output-dir trajectory-securebert2 \
  --train-jsonl trajectory-data/train.jsonl \
  --validation-jsonl trajectory-data/validation.jsonl \
  --test-jsonl trajectory-data/test.jsonl

python run_finetune.py \
  --model modernbert \
  --output-dir trajectory-modernbert \
  --train-jsonl trajectory-data/train.jsonl \
  --validation-jsonl trajectory-data/validation.jsonl \
  --test-jsonl trajectory-data/test.jsonl

python run_external_baseline.py \
  --output external-bash-classifier.json

The ActionFacts JSONL files must first be generated with the DefenseClaw exporter in benchmarks/cmd/actionfacts-features. They are included in the completed artifact set.

Software environment

Component	Version
Platform	macOS 26.6.2, arm64
Python	3.12.12
PyTorch	2.14.0
Transformers	5.17.0
scikit-learn	1.9.1
NumPy	2.5.3

The training host used Apple's MPS backend when available.

Artifact inventory

• results.json: frozen-encoder results, exact counts, operating points, and corpus hashes.
• actionfacts-ablation.json: structured-feature and combined ablations.
• finetune-securebert2/ and finetune-modernbert/: atomic results and selected checkpoints.
• trajectory-securebert2/ and trajectory-modernbert/: bounded-trajectory results and selected checkpoints.
• external-bash-classifier.json: compatible external baseline.
• trajectory-data/: exact prepared trajectory splits.
• actionfacts/: exported value-free deterministic features.
• run_*.py: experiment and evaluation runners.
• assets/results-overview.svg: accessible summary figure used by this card.
• MANIFEST.json: file sizes, SHA-256 digests, platform, and package versions.
• HUB_MANIFEST.json: checksums for the intentionally limited private Hub publication, including checkpoints.

The private Hugging Face publication includes the card, figure, reports, runners, result JSONs, frozen tree probes, and all four selected Transformer checkpoints. It intentionally excludes normalized actionfacts/ rows, prepared trajectory-data/ rows, embedding caches, the Python virtual environment, and the downloaded Kestrel artifact. Those files remain in the local experiment directory; upstream entries marked download-only should be reacquired from their linked sources and transformed locally.

Integrity verification

Rebuild the complete local manifest and the publication-specific Hub manifest with:

python build_hub_manifest.py
python build_manifest.py

To verify the recorded files without modifying the manifest:

python - <<'PY'
import hashlib
import json
from pathlib import Path

root = Path.cwd()
manifest_name = "HUB_MANIFEST.json"  # use MANIFEST.json for the complete local artifact
manifest = json.loads((root / manifest_name).read_text())
for name, expected in manifest["artifacts"].items():
    path = root / name
    actual = hashlib.sha256(path.read_bytes()).hexdigest()
    assert actual == expected["sha256"], (name, actual, expected["sha256"])
print(f"verified {len(manifest['artifacts'])} artifacts")
PY

Citation

No paper citation is currently available. If you use this benchmark, cite this repository card and the exact Hub commit revision, and separately cite the upstream models and datasets you use.

Acknowledgements

This evaluation builds on cisco-ai/SecureBERT2.0-base, answerdotai/ModernBERT-base, and P0u4a/ModernBERT-bash-classifier. Kestrel, auto-0.4b-2, Husky Sight, Husky Paw, Husky Nose, and ToolCallVerifier informed the external-baseline analysis even where a faithful numeric comparison was not possible.

Bottom line

SecureBERT2 is not deployable as a DefenseClaw risk router on the current corpus. The experiment does not show that learned risk routing is impossible; it shows that this cross-source label construction cannot support it reliably. Better labels, stronger calibration, and stricter independent evaluation should precede further architecture scaling.