Gemma-4-26B-A4B-it-RAZOR-14B-A4B-E64of128

Gemma 4 26B-A4B Instruct with half of its routed experts removed by RAZOR, a training-free expert pruning method. Every MoE layer keeps 64 of its original 128 routed experts. No gradient updates or recovery training were applied: the retained weights are the base model's own weights.

	Base	This model
Routed experts per MoE layer	128	64
Total parameters	26B	14.4B
Active parameters per token	~4B	~4B
Active experts per token (top-k)	8	8 (unchanged)
MoE decoder layers	30	30 (unchanged)
Vision tower	0.57B	0.57B (unchanged)

Pruning touches only the routed expert pool of the language model. The vision tower, attention, the embedding, the LM head and the router's remaining rows are untouched, so the model remains multimodal and the compute per token is essentially unchanged while expert storage shrinks.

The other budget is Gemma-4-26B-A4B-it-RAZOR-20B-A4B-E96of128.

Usage

The chat template and response parsing schema use the official base revision 4d7ae4984b7db7de8f8457170b3f1a419ee76d52, including its tool-argument null serialization and tool-response fixes. The tokenizer vocabulary and all model weights are unchanged by this auxiliary-file update; prompt formatting may differ from the earlier experiment snapshot. This is not a new quality evaluation.

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "Nickyang/Gemma-4-26B-A4B-it-RAZOR-14B-A4B-E64of128"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype="bfloat16", device_map="auto",
)

messages = [{"role": "user", "content": [{"type": "text", "text": "Explain mixture-of-experts routing."}]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)
print(processor.decode(model.generate(**inputs, max_new_tokens=256)[0]))

Requires a Transformers build containing the native gemma4 implementation. Weights are bfloat16.

How the experts were selected

RAZOR asks whether the surviving computation can replace an expert's function, rather than how often or how strongly the expert fires. For a token routed to the set $S$ with normalized weights $w_j$, let $c=\sum_{j\in S} w_j f_j$ be the routed mixture and $r_j=f_j-c$ the consensus residual of expert $j$. Deleting a selected expert $i$ makes the router promote its highest-ranked unselected expert $r$, whose pseudo-weight $w_r$ is its score divided by the original selected score sum. With $\lambda$ the routed-output scale, the exact local output change is

$${\delta }_i=\lambda \parallel c-{\tilde{c}}^{-i}{\parallel }_2=\lambda \frac{\parallel w_i{r}_i-w_r{r}_r{\parallel }_2}{1-w_i+w_r}\mathrm{.}\backslash delta\_i=\backslash lambda\backslash lVert\; c-\backslash tilde\; c^\{-i\}\backslash rVert\_2\; =\backslash lambda\backslash ,\backslash frac\{\backslash lVert\; w\_i\; r\_i\; -\; w\_r\; r\_r\backslash rVert\_2\}\{1-w\_i+w\_r\}.$$

Scores are aggregated by conditional root mean square over the calibration tokens routed to each expert, and the highest-scoring experts are retained per layer. See the paper for the derivation and the code for the implementation.

Reproducing this checkpoint

Calibration used 32,768-token rows from RazorCal, the 2,048-sample multi-domain corpus released with RAZOR. The corpus is text-only, so the selection reflects text-domain routing behaviour.

pip install razor-moe

razor saliency --model <path-to-gemma-4-26B-A4B-it> \
               --data RazorCal.json --max-len 32768 --out out/sal
razor prune    --model <path-to-gemma-4-26B-A4B-it> --saliency out/sal \
               --method razor --ratio 0.5 --out out/pruned

kept_expert_indices.json is the keep-set manifest this checkpoint was built from, in the format razor verify expects:

razor verify --pruned . --src <path-to-gemma-4-26B-A4B-it>

Selection depends on the calibration draw, so an independent run reproduces the procedure rather than this exact expert set.

Limitations

Expert pruning is lossy. Benchmark retention and predictive fidelity do not guarantee stable generation: the paper reports that responses change in diversity, formatting and termination behaviour even where task accuracy is largely preserved. Evaluate on your own workload before deploying.

The calibration corpus is text-only and multi-domain but finite. Multimodal behaviour was not exercised during calibration, so image-conditioned quality is not characterised by the released measurements. Evaluate vision-language tasks separately before relying on them.

License and attribution

This is a derivative of gemma-4-26B-A4B-it, released by Google under the Apache-2.0 license, and is distributed under the same license. The retained weights are the base model's own weights; RAZOR contributes the expert selection, not new parameters.

The RAZOR code is Apache-2.0. RazorCal records remain subject to their applicable upstream terms; see LICENSE-DATA.

Citation

@misc{song2026razorpruningreplaceableexperts,
      title={RAZOR: Pruning Replaceable Experts in LLMs},
      author={Mingyang Song and Mao Zheng},
      year={2026},
      eprint={2609.30465},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2609.30465},
}