AtomicChat/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF

NVIDIA Nemotron 3.5 Super VL 120B-A12B · Atomic Dynamic GGUF

Atomic Dynamic (AD) quants, built from NVIDIA's original BF16 weights with our own importance matrix. Every file here matches or beats the other GGUF builds of the same size or larger, measured side by side on one stand. The calibration corpora behind our builds are public.

• Every file here was measured against the BF16 original on the same stand as the other publishers' files, and each one beats or ties the file of theirs that is the same size or larger.
• See our quantization analysis below for measurements and instructions.

---

Lower is closer to the original. Each purple point is a file in this repo; the grey ones are the other publishers' files, downloaded and measured with exactly the same reference, text and llama.cpp build. At 75 GB, AD-IQ4_NL drifts about half as much as both NVFP4 builds, which are the same size or up to 3 GB larger. At 70 GB, AD-IQ3_M drifts about half as much as LM Studio's Q4_K_M of the same size. At 65 GB, AD-IQ3_XXS drifts 20% less than Unsloth's UD-Q3_K_XL, and from 46 to 58 GB the AD-Q2_0_* files drift 6 to 18% less than the Unsloth file of each size, and at 87 GB AD-Q4_K_S drifts 13% less than UD-Q4_K_XL.

What is Atomic Dynamic

Atomic Dynamic is how we decide which tensor gets which bits. The files use only standard llama.cpp quantization types: upstream llama.cpp with nemotron_h_moe support reads them, no fork and no special build.

• The experts take the compression, nothing else does. In this model about 93% of the weights are experts. The Mamba-2 projections, attention, the latent projections and the shared expert together are under 6% of the size, so keeping them at q8_0 or q6_k in every file costs little, and every bit saved goes to the experts instead.
• Bits follow the measured activity, not the position. The blocks that get extra precision are the ones the importance matrix ranks highest (here 76, 79, 81, 83, 85, 87), not a fixed fraction of the depth.
• Calibrated on the real chat format. The importance matrix is collected on the BF16 weights over 4.6M tokens rendered through this model's own chat template, with the special tokens read as tokens.
• A better q2_0. llama.cpp's own q2_0 quantizer ignores the importance matrix and scales each block by its largest weight, so it never uses the format's fourth level and rounds most weights to zero. Ours searches each block's scale over both signs, uses all four levels and weighs the error by the importance matrix: on this model the same 51 GB layout drifts 0.107 with it and 0.258 with the stock quantizer. The files are ordinary q2_0, so nothing special is needed to run them.
• No silent type swaps. llama.cpp quietly replaces types that do not fit a tensor's shape. We pick the replacement ourselves and check every tensor's type in the finished file.
• Measured against everyone, published only where we win. Each file is set against the other publishers' files of the same size or larger before it goes up.

Pick a file

Every number below is measured, not estimated. How we measured it is at the bottom, and the raw logs are in the metrics repo so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the original weights. Lower is better, and zero means identical. top-1 is how often it picks the same next word the original would have picked.

File	Size	KL divergence	top-1
AD-Q8_0	131.6 GB	0.00442	97.22%
AD-Q6_K	120.0 GB	0.00476	97.13%
AD-Q5_K_S	110.6 GB	0.00603	96.76%
AD-Q4_K_S	86.9 GB	0.00816	96.23%
AD-IQ4_NL	74.8 GB	0.01327	95.24%
AD-IQ3_M	70.0 GB	0.02366	93.75%
AD-IQ3_XXS	65.3 GB	0.03523	92.43%
AD-IQ2_M	59.3 GB	0.06429	89.75%
AD-Q2_0_XL	57.4 GB	0.06566	90.02%
AD-Q2_0_L	55.8 GB	0.07219	89.44%
AD-Q2_0_M	51.0 GB	0.10652	86.97%
AD-Q2_0_S	45.7 GB	0.14704	84.86%

The AD-Q2_0_* files put q2_0 on the expert down projections of the least active blocks, quantized with our own importance-weighted q2_0 quantizer (below).

AD- marks an Atomic Dynamic layout: the experts are compressed hardest, the attention, Mamba, shared-expert and embedding tensors stay at q8_0 or q6_k, and the blocks the importance matrix ranks highest get more bits. The rest of the name is the type of the expert up projections, the largest tensor group (the AD-Q2_0_* files are named after the q2_0 on their down projections instead); the exact type of every tensor is in ladder/ in the metrics repo. Compare files by size on disk, not by name.

Which one fits your card

The file has to fit, and so does the context. Only 8 of the 88 layers are attention; the Mamba layers keep a fixed-size state. The attention cache is about 8 KB per token, 0.1 GB at 8k context and 0.2 GB at 32k, plus about 1 GB of compute buffers.

Memory for the model (GPU or unified)	Largest file that fits
48 GiB	AD-Q2_0_S (45.7 GB on disk)
56 GiB	AD-Q2_0_M (51.0 GB on disk)
64 GiB	AD-IQ3_XXS (65.3 GB on disk)
80 GiB	AD-IQ4_NL (74.8 GB on disk)
96 GiB	AD-Q4_K_S (86.9 GB on disk)
128 GiB	AD-Q6_K (120.0 GB on disk)
192 GiB and up	AD-Q8_0 (131.6 GB on disk)

If you are choosing between two neighbouring files, take the larger one. The steps between them cost one or two gigabytes and buy noticeably fewer wrong words.

Running it

llama-server -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-AD-IQ3_M.gguf -ngl 99 -c 8192

The chat template is inside every file, llama.cpp applies it on its own.

The model's multi token prediction head is inside every file (q8_0 in AD-Q8_0 and AD-Q6_K, q5_k down to AD-IQ3_XXS, q4_k below). We have not tested speculative decoding with it yet; the KL divergence numbers do not involve it.

Speed

Images

This model reads images, not only text. That needs one extra file, the vision projector, which is separate from the quant and shared by all of them:

• mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-F16.gguf for most people
• mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf if your setup prefers bfloat16

Download it once alongside whichever quant you picked:

llama-mtmd-cli \
  -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-AD-IQ3_M.gguf \
  --mmproj mmproj-NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-F16.gguf \
  --image your-photo.jpg -ngl 99 -c 8192 \
  -p "What is in this image?"

We checked image input on AD-Q8_0 with a simple shapes-and-colours picture. Video input was not tested.

How these compare to other builds

We downloaded the other publishers' files and measured them with the same reference logits, held-out text, context, chunk count and llama.cpp commit as ours. Each of our files is set against the other file that is the same size or larger:

Ours	Size	KL divergence	top-1	Theirs	Size	KL divergence	top-1
AD-Q8_0	131.60 GB	0.00442	97.22%	unsloth Q8_0	131.61 GB	0.00443	97.23%
AD-Q6_K	120.00 GB	0.00476	97.13%	unsloth UD-Q6_K_XL	120.99 GB	0.00482	97.08%
AD-Q5_K_S	110.57 GB	0.00603	96.76%	unsloth UD-Q5_K_XL	110.70 GB	0.00628	96.66%
AD-Q4_K_S	86.90 GB	0.00816	96.23%	unsloth UD-Q4_K_XL	86.91 GB	0.00937	95.98%
AD-IQ4_NL	74.83 GB	0.01327	95.24%	lmstudio-community NVFP4	75.24 GB	0.02564	93.16%
AD-IQ4_NL	74.83 GB	0.01327	95.24%	unsloth NVFP4	77.69 GB	0.02530	93.17%
AD-IQ3_M	69.97 GB	0.02366	93.75%	lmstudio-community Q4_K_M	70.10 GB	0.04527	90.97%
AD-IQ3_XXS	65.34 GB	0.03523	92.43%	unsloth UD-Q3_K_XL	65.76 GB	0.04433	91.61%
AD-IQ2_M	59.32 GB	0.06429	89.75%	unsloth UD-IQ3_XXS	59.76 GB	0.06702	89.83%
AD-Q2_0_XL	57.43 GB	0.06566	90.02%	unsloth UD-Q2_K_XL	57.79 GB	0.07666	89.09%
AD-Q2_0_L	55.84 GB	0.07219	89.44%	unsloth UD-IQ2_M	55.88 GB	0.08837	88.27%
AD-Q2_0_M	51.02 GB	0.10652	86.97%	unsloth UD-IQ2_XXS	51.12 GB	0.11707	86.50%
AD-Q2_0_S	45.72 GB	0.14704	84.86%	unsloth UD-IQ1_M	46.14 GB	0.15571	84.39%

The Q8_0 pair is a tie: both are plain q8_0, as they should be. AD-IQ2_M drifts less than UD-IQ3_XXS on average but matches the original's top token slightly less often.

Our files were measured on 8x A100-SXM4-80GB, theirs on 8x A100-SXM4-40GB, same llama.cpp commit and CUDA build; the Q8_0 pair, which should be equal, comes out within 0.2% of each other across the two machines.

What we found while building these

• Every expert in this model is 2688 wide, which is not a multiple of 256. The expert down projections can therefore only take llama.cpp's 32-wide block types (q4_0, q5_0, q8_0, iq4_nl), or q2_0. Our files write those types directly instead of letting llama-quantize swap them in silently.
• The blocks that get extra precision (76, 79, 81, 83, 85, 87) were picked from the importance matrix statistics, not by position.
• Below 60 GB the down projections are the problem: iq4_nl (4.5 bits) is the lowest calibrated type they take, which leaves too few bytes for the up projections. Unsloth's files solve it with q2_0 on the down projections of some blocks. We do the same with our own q2_0 quantizer (above), which is where the gain at these sizes comes from.
• Which blocks take q2_0 matters as much as the quantizer. Our importance-matrix ranking grows with depth on this model and picked the wrong blocks: with the same quantizer it measured 8 to 18% worse than the block sets in Unsloth's files, which keep blocks 19 to 26 and 72 to 87 at iq4_nl. The AD-Q2_0_* files use those block sets; the quantizer is ours.
• On this model the expert up projections matter more than the down projections. Our first 87 GB file kept the down projections a step above the up ones and drifted 0.0104; AD-Q4_K_S puts q5_k on every up projection, takes the down projections' layout from UD-Q4_K_XL (iq4_nl on blocks 28 to 43, q5_0 elsewhere) and spends what is left on the Mamba, attention, latent and shared expert tensors at q8_0: 0.0082.

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe nemotron-omni-moe. It is 4,589,438 tokens across 2,820 documents:

Part	Share
agentic and tool use	24.0%
multilingual	19.6%
code	17.4%
reasoning	13.1%
long context	11.1%
vocabulary sweep	5.9%
structured data	4.4%
graphics	4.4%

Every conversation is rendered through this model's own chat template, so the chat markup appears as the single tokens the model actually sees.

If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out set is 6.3673 plus or minus 0.03710.

Held-out text: eval_neutral from our calibration dataset, never used for calibration. 95 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement.

Hardware: 8x A100-SXM4-80GB.

The raw logits of the reference are published in the metrics repo, split into parts because of the file size limit. With them you can measure your own build against exactly the same point we did:

cat base-neutral.kld.*.part > base-neutral.kld

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, split across 8 workers and merged. Splitting by chunk range gives the same result as one long run, because the statistic is a sum:

llama-imatrix -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf -f calib_train.txt -o shard-0.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special --from-chunk 0 --chunks 1121

llama-imatrix -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf --in-file shard-0.gguf,shard-1.gguf,... \
  -o imatrix.gguf

Then, for AD-IQ3_M:

llama-quantize \
  --imatrix imatrix.gguf \
  --tensor-type '^blk\.88\.ffn_down_exps\.weight$=q5_1' \
  --tensor-type '^blk\.88\.=q5_k' \
  --tensor-type '^token_embd\.weight$=q8_0' \
  --tensor-type '^output\.weight$=q8_0' \
  --tensor-type '^blk\.(76|79|81|83|85|87)\.ffn_down_exps\.weight$=q5_0' \
  --tensor-type '^blk\.(76|79|81|83|85|87)\.ffn_up_exps\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_down_exps\.weight$=iq4_nl' \
  --tensor-type '^blk\.\d+\.ffn_up_exps\.weight$=iq3_s' \
  --tensor-type '^blk\.\d+\.ffn_latent_(down|up)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ffn_(down|up)_shexp\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.attn_(q|k|v|qkv|output)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ssm_(in|out)\.weight$=q8_0' \
  --override-kv general.file_type=int:26 \
  NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16.gguf NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-AD-IQ3_M.gguf IQ3_M

The rules for every other file are in ladder/ in the metrics repo, next to its quantization log. The first rule that matches a tensor wins.

Model details

88 layers (40 Mamba-2, 40 mixture-of-experts with 512 experts, 8 attention), hidden size 4096, vocabulary 131,072. One multi token prediction head. NVIDIA's config sets a maximum context of 1,048,576 tokens; we measured at 4096 and did not test long context.

Needs a recent llama.cpp with nemotron_h_moe architecture support. Built and tested with llama.cpp b11668 (cd0a5b6a1).