--- license: openmdw-1.1 library_name: transformers pipeline_tag: text-generation tags: - nvidia - nemotron-3.5 - moe - expert-pruning - reap - hybrid - mamba2 - experimental base_model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 ---

⚠️ EXPERIMENTAL / ALPHA — NOT FULLY TESTED

Early-stage derivative, not a fully benchmarked release. Intended as a base for task-specific LoRA fine-tuning (16GB-VRAM GPU deployments, 24GB+ laptops/desktops). Expect rough edges — validate on your own workload before relying on it.

# Nemotron 3.5 Lightning 30B-A3B — REAP-20B (general calibration) A **20B-total / 3B-active** derivative of `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`, pruned with **REAP expert pruning** (128 → 77 routed experts per MoE layer, 40% sparsity), calibrated on a **general-purpose corpus with a slight coding focus** (general instruction 44% / coding 33% / math 22%, 538 samples generated by the full model). MTP head stripped. **No LoRA included** — this is the pre-LoRA base; a domain LoRA (subagent orchestration + coding/tool-calling mix) is the intended next step. | | | |---|---| | Base model | [NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) | | Params before / after | 31.58B → **19.87B** (active stays ~3B) | | Pruning | REAP, 40% expert sparsity, 128→77 experts/layer × 23 MoE layers | | Calibration data | 538 samples: instruction 238, coding 180, math 120 (from the owner's general/coding/math prompt banks) | | Quantization | None in this repo (BF16); IQ4_NL GGUF ≈ 11.5GB (see below) | | License | OpenMDW-1.1 (same as base) | This repository ships: - the **REAPed model** (weights + config, `model-*.safetensors`) - the **methodology + recalibration kit** (`reap/`): how the pruning was calibrated, the prompt-bank mix, and how to redo it on your own data. ## Why prune Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + Attention model with `hidden=2688`, `moe_intermediate=1856`. Neither is divisible by 256, so llama.cpp K/IQ quant formats (256-element superblocks) cannot represent the expert weights — every GGUF quant falls back to block-32 formats (~4.5 bpw floor) regardless of requested bit width. At 30B that floor is ~18GB; at 20B it is ~11.5GB. REAP removes the redundant experts so the model fits smaller quantization budgets without touching the active computation: - **~37% smaller total memory** at the same speed (3B active unchanged) - **KV cache stays ~6KB/token** (only 6 attention layers, 2 KV heads), so long contexts stay cheap on consumer hardware - A **Q4_0/IQ4_NL GGUF ≈ 11.5GB** fits a 24GB MacBook with room for 32K+ context (the block-32 floor is why IQ3-class GGUF quants are not achievable for this architecture at any size) ## Calibration data (the important part) REAP prunes the experts your calibration data *doesn't use*. This model was calibrated on a **general-purpose mix** so the surviving experts serve general use: | Source | Share | Content | |---|---|---| | instruction (openhermes-style) | 44% | general QA, writing, reasoning, how-to | | coding (deepseek trajectories + opencode) | 33% | coding problems, agentic code tasks | | math (metamath-style) | 22% | word problems, arithmetic, reasoning | Completions were generated by the full model (NVFP4, natural thinking, per-domain token budgets). If your workload is heavily domain-specific, **re-calibrate on your own data** — see [`reap/`](reap/README.md). ## Using this repo ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained( "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B", torch_dtype=torch.bfloat16, trust_remote_code=True) tok = AutoTokenizer.from_pretrained("sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B") ``` For llama.cpp / LM Studio on Apple Silicon, quantize to IQ4_NL (~11.5GB): ```bash llama-quantize model.gguf out-IQ4_NL.gguf IQ4_NL llama-server -m out-IQ4_NL.gguf -c 131072 --reasoning-parser nemotron_v3 ``` **Context length**: `-c 131072` (128K) is a safe default for 16GB-VRAM GPUs and 24GB Macs — KV is only ~6KB/token (6 attention layers, 2 KV heads), so 128K context costs roughly 0.8GB. The model itself supports **up to 1M tokens** (256K is NVIDIA's single-GPU deployment length); on laptops the practical limit at longer contexts is prefill time, not memory — raise `-c` (e.g. `262144`) if your workload needs it. ## Validation (limited — alpha) Sanity-checked with greedy generation on general QA, coding, and math prompts: coherent analysis, correct answers (e.g. math arithmetic verified), no repetition/looping. Full benchmark suites were **not** run. The closest published pruning analogue (Qwen3-30B-A3B, 128 experts, 8 active) retained ~99.8% of baseline at 50% sparsity. ## Sources & how to make one yourself - **Base model**: [NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) - **Algorithm**: [REAP — Router-weighted Expert Activation Pruning](https://arxiv.org/abs/2510.13999) (Cerebras; implemented in [vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor/tree/main/examples/reap_expert_pruning)) - **Pruning tooling**: llm-compressor's `REAPPruningModifier`, plus 3 small patches for the NemotronH architecture (documented in [`reap/README.md`](reap/README.md)) - **Calibration data**: 538 samples generated by the full model from general instruction, coding, and math prompt banks (openhermes-style instruction data, DeepSeek coding-trajectory prompts, metamath-style word problems) — see [`reap/generate_general_data.py`](reap/generate_general_data.py) - **Synthetic-data guidance**: NVIDIA's [Build a Bash Agent with Synthetic Data](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/use-case-examples/Build_a_Bash_Agent_with_Synthetic_Data) and [sft-data-blending](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/src/nemotron/steps/patterns/sft-data-blending.md) patterns - **Why 20B / the quant floor**: this architecture's `hidden=2688` / `moe_intermediate=1856` are not 256-divisible, so llama.cpp GGUF quants floor at block-32 formats (~4.5 bpw); at 20B that is ~11.5GB instead of ~18GB To reproduce on your own data: serve the full model, generate your domain corpus with `reap/generate_general_data.py` (or your own loader), then run `reap/reap_nemotron.py --data yours.jsonl --sparsity 0.40`. Full step-by-step, patches, and hardware notes: [`reap/README.md`](reap/README.md). ## Related - **LoRA adapter + IQ4_NL GGUF** (recovery pass + quantized for laptops): [Nemotron-3.5-Lightning-30B-A3B-REAP-20B-LoRA-IQ4NL](https://huggingface.co/sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B-LoRA-IQ4NL) - Base: [nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) ## License [OpenMDW License Agreement, version 1.1](https://openmdw.ai/license/1-1/) (same as the base model). See `LICENSE`. If you upload derived model files, carry the same license and attribute NVIDIA for the base weights.