crucible-labs/Gemma4-26B-A4B-REAP-25-GGUF

Gemma 4 26B-A4B · REAP-25 · Q3_K / Q2_K mixed (imatrix) — GGUF

A 9–10 GB mixture-of-experts specialist derived from google/gemma-4-26B-A4B-it by pruning 25 % of routed experts with routing-aware REAP and quantizing the result with attention-aware mixed-precision schemes calibrated by llama-imatrix.

Two quantizations are provided in this repo:

File	Size	HumanEval	MBPP	When to pick
Gemma4-26B-A4B-REAP-25-Q3_K-mixed.gguf	10.3 GB	92.7 %	70.4 %	Default. Best quality/size trade
Gemma4-26B-A4B-REAP-25-Q2_K-mixed.gguf	9.1 GB	90.9 %	—	Tightest memory (headroom for KV / other apps)

The Q4_K_M (13 GB) baseline scored 91.5 % HumanEval — bigger AND lower quality than the Q3_K mixed variant, so it isn't shipped here.

Companion release to crucible-labs/Qwen3.6-35B-A3B-REAP-48-Q3K-mixed-GGUF.

TL;DR

	Base · bf16	REAP-25 · bf16	REAP-25 · Q3_K-mixed (this repo)
Parameters	~26 B	~19.5 B	~19.5 B
File size on disk	~52 GB	~40 GB	10.3 GB
HumanEval	93.9 %	93.9 %	92.7 %
MBPP	73.0 %	—	70.4 %
IFEval (prompt / instr)	—	—	80.4 % / 86.6 %
MMLU-Pro (5-shot)	82.6 %*	—	76.6 %

* MMLU-Pro base score is from the Gemma 4 published model card, not re-measured here. All other numbers are from this project's own harness.

Method

Compression: routing-aware REAP, 25 % expert pruning

Routed experts are scored by router-weighted activation magnitude and a routing-disruption penalty (the routing-aware variant of REAP); the lowest- scoring 25 % per layer are pruned. See arXiv:2510.13999 for the method.

Why routing-aware here (and not on Qwen 3.6): Gemma 4 has no shared expert to absorb load when a routed expert is pruned, so the routing-disruption penalty measures real quality cost rather than being double-counted by the architecture. On models with a shared expert (Qwen 3.6, DeepSeek, etc.), plain REAP without the routing term is sufficient.

Result before quantization: ~26 B → ~19.5 B parameters; bf16 size ~40 GB.

Quantization: mixed profiles + imatrix

Both files use llama.cpp's Q3_K_S selection base with attention overridden to Q8_0, calibrated with llama-imatrix on the same code / tool-use / math mix used for REAP scoring. Approximate distribution (658 tensors total):

Tensor group	Q3_K-mixed file	Q2_K-mixed file
Attention (attn_q/k/v/output), embeddings, output	Q8_0 (129 tensors)	Q8_0 (129 tensors)
Mid-precision body	Q4_K + Q5_0 + Q4_0 (106)	Q4_K + Q5_0 + Q4_0 (106)
Lowest-precision body	Q3_K (30)	Q2_K (30)
Layer norms / small	F32 (392)	F32 (392)

The two files differ only in those 30 lowest-precision tensors — Q3_K vs Q2_K — which is where the ~1 GB size difference comes from.

Inference

llama.cpp / llama-server (recommended)

llama-server \
  -m Gemma4-26B-A4B-REAP-25-Q3_K-mixed.gguf \
  --port 8093 --host 0.0.0.0 \
  -c 8192 -ngl 999 \
  --jinja --reasoning off

Recommended sampling

• temperature: 0 (deterministic) or 0.2 for variation
• max_tokens: 2048 for code, higher for long-form

Performance

Not re-measured here for the two shipping quants specifically. Reference numbers from the same-family Q2_K mixed on M4 Mac Mini via Metal: 32.6 tg / 401 pp. Expect the Q3_K variant to be similar or slightly slower in token generation, similar in prompt processing.

Known issues

Q2_K prompt processing is 7× slower on ROCm

llama.cpp's Q2_K dequant kernel on ROCm (gfx1151) has a known ~7× slowdown in prompt processing versus other quant types. Metal (Mac) and CUDA (NVIDIA) are not affected. If you're on an AMD GPU and PP throughput matters, pick the Q3_K-mixed file instead.

Gemma 4 compression cliff is at ~37 %

We measured earlier ratios of REAP on this model. Above ~30 % pruning, HumanEval starts degrading noticeably; at 37.5 % pruning it dropped to 75.0 % — a hard cliff. This is why the release ships at 25 % pruning even though Qwen 3.6 tolerates 48 %.

Reproducibility

This artifact was produced by the Crucible compression toolkit (developed alongside this model; source not yet publicly released — planned). High-level pipeline (identical shape to the Qwen 3.6 release):

# 1. REAP scoring (routing-aware)
crucible observe google/gemma-4-26B-A4B-it \
  --samples 512 --seed 42 \
  -o results/gemma4-base-observation.json

# 2. REAP pruning at 25 %, routing-aware
crucible compress google/gemma-4-26B-A4B-it \
  --method reap --routing-aware --ratio 0.25 \
  --scores-file results/gemma4-base-observation.json \
  -o outputs/gemma4-reap-25pct

# 3-5. GGUF convert, imatrix, quantize (same steps as Qwen release)

Evaluation methodology

Numbers above are from Crucible's own-harness bench subcommand (no evalplus / lm-eval runtime dependency). Each problem is run once at temperature=0; code is executed in a subprocess with a 30 s timeout. HumanEval numbers use the standard 164 problems (vanilla, not the augmented humanevalplus). MBPP numbers use the vanilla 500-problem split. IFEval uses the standard HuggingFaceH4/ifeval dataset with fixed checkers (paragraph counter that skips markdown dividers, sentence splitter on punctuation + uppercase, whitespace-normalized repeat_prompt, etc.).

Intended use & limitations

Designed as a coding specialist for local single-user inference on consumer hardware.

Not designed for or measured on: multilingual generation, image/audio input (the compressor operates only on the language-model component), long-form prose, or code in languages other than Python.

The base Gemma 4 model has substantially different training and behavior from Qwen 3.6 — pick the model that fits your use case:

• Qwen 3.6 REAP-48 Q3K-mixed: 9.4 GB, stronger BFCL tool-calling (91.8 %), also has BigCodeBench numbers, 3 B active params
• Gemma 4 REAP-25 Q3K-mixed (this repo): 10.3 GB, higher raw HumanEval (92.7 %), 4 B active params, Gemma-family instruction-following characteristics

License

Apache-2.0, inherited from google/gemma-4-26B-A4B-it.

Citation

@article{lasby2025reap,
  title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and
          Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
  journal={arXiv preprint arXiv:2510.13999},
  year={2025}
}