wyattearp/Nemotron-3.5-Lightning-30B-A3B-NVFP4

NVIDIA Nemotron-3.5-Lightning-30B-A3B (ModelOpt Mixed-Precision NVFP4)

Nemotron-3.5-Lightning-30B-A3B-NVFP4 is a post-training quantized (PTQ) mixed-precision release of NVIDIA-Nemotron-3.5-Lightning-30B-A3B optimized specifically for NVIDIA Blackwell architectures (GB10 / DGX Spark, GB200, RTX 50-series) and vLLM serving.

Also, I'm not going to ask for your email address in some spam-based-engagement-grift. Enjoy.

---

⚡ Performance & Resource Profile

• Checkpoint Footprint: 20.05 GiB (down from 62.0 GiB BF16, a 68% reduction).
• Generation Speed: ~76.0 tok/s on DGX Spark (NVIDIA GB10).
• Prompt Processing: >1,060 tok/s prefill throughput with FlashInfer CUTLASS NVFP4 MoE kernels.
• Context Window: Supports up to 1,048,576 tokens (1M context) with O(1) memory scaling across 20 Mamba-2 SSM layers.

---

🔬 Precision Topology

Component	Precision	Rationale
26 MoE MLP Expert Layers	NVFP4 (E2M1, Block Size 16)	>85% of parameter volume. Block-wise quantization with FP8 scales preserves expert sparsity with zero accuracy degradation.
6 Attention Layers	FP8 (float8_e4m3fn)	Preserves 256-level dynamic range and Rotary Position Embedding (RoPE) geometry across 1M context.
20 Mamba-2 SSM Layers	Pure BF16 / FP32	Preserves state-space scan continuous recurrence kernels (A_log, D, conv1d, dt_bias).
KV Cache	Native BF16	Omitted uncalibrated FP8 KV cache declarations to ensure 100% numerical stability.

---

🛠️ Serving Instructions with vLLM

To serve this model with optimal multi-turn tool calling and reasoning support, use qwen3_xml (C-based Expat streaming XML parser) and nemotron_v3 reasoning parser:

vllm serve wyattearp/Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --served-model-name nvidia/nemotron-3.5-lightning-30b-nvfp4 \
  --tool-call-parser qwen3_xml \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --mamba-ssm-cache-dtype float32 \
  --dtype auto \
  --gpu-memory-utilization 0.75 \
  --max-model-len 262144

---

⚠️ Critical Deployment & Tool-Calling Warnings

1. Tool Call Parser Selection: Use qwen3_xml, NOT qwen3_coder

• Upstream Recommendation: Upstream model documentation suggests --tool-call-parser qwen3_coder.
• Empirical Finding: In vLLM, qwen3_coder is a Python regular-expression parser (re.finditer) that suffers from token boundary slicing during streaming. It drops reasoning chunks, corrupts bash redirects (<, >, |), and produces truncated tool arguments.
• Solution: Always configure --tool-call-parser qwen3_xml. This uses Python's C-based pyexpat streaming parser, ensuring zero dropped reasoning tokens and 100% reliable tool call extraction.

2. Multi-Turn History & <think> Token Management

• Reasoning Monologue Ingestion: Nemotron 3.5 Lightning is an explicit chain-of-thought model that generates internal reasoning enclosed within <think>...</think>.
• Best Practice: In multi-turn chat applications or agentic loops, strip or collapse prior assistant <think> blocks in conversation history before sending the next turn. Retaining uncollapsed historical <think> streams can cause the model to deliberate in loops or draft tool calls within <think> rather than emitting them to the active stream.

3. Numerical Stability & KV Cache

• Blackwell Serving: On NVIDIA Blackwell (GB10 / DGX Spark, GB200, RTX 50-series), keep the default native BF16 KV cache and set --mamba-ssm-cache-dtype float32 to guarantee continuous numerical precision across long context sequences.