majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit

KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

Nemotron-3-Nano-Omni-30B-A3B-Reasoning - TurboQuant MLX 8-bit

MLX 8-bit quantization of the text tower of Nemotron-3-Nano-Omni-30B-A3B-Reasoning (nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) with TurboQuant weight method. Apple Silicon native via mlx-lm.

This variant covers the LLM backbone only. Vision (CRADIO v4-H) + audio (Parakeet-TDT-0.6B-v2) encoders are NOT included — the MLX-VLM Nemotron-Omni model class is supported upstream as of mlx-vlm v0.4.4 (PR #1518, merged 2026-07-07; audio weight fixes in PR #1743, merged 2026-07-29). For multimodal inference, use mlx-vlm (v0.4.4+) or the GGUF variants with llama-mtmd-cli.

For the matched-KV stack — TurboQuant weights + TurboQuant KV-cache modifier — For the runtime KV-cache modifier itself, see majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant.

Quickstart

# Today (mlx-lm 0.31.x): the NemotronH_Nano_Omni_Reasoning_V3 model class
# is not yet registered in mlx-lm. The cell below is the API shape that WILL
# work once upstream lands the class (track ml-explore/mlx-lm#386).

from mlx_lm import load, generate

model, tokenizer = load("majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-TurboQuant-MLX-8bit")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Solve: 17 * 23"}],
    add_generation_prompt=True,
    enable_thinking=False,  # set True to enable extended reasoning (default)
)

response = generate(
    model, tokenizer,
    prompt=prompt,
    max_tokens=512,
    sampler=lambda x: x.argmax(axis=-1),  # or use mlx_lm.sample_utils.make_sampler(temp=0.6, top_p=0.95)
)
print(response)

⚠️ This variant covers the text tower only. For multimodal inference (vision + audio + video), use the GGUF variants with llama-mtmd-cli — see the GGUF cards in this family.

Modality matrix

Modality	Encoder	Quantization in this variant
Text	LLM backbone (Mamba-2 + Transformer hybrid Sparse MoE)	per the variant suffix
Image	CRADIO v4-H	BF16 (kept full-precision in every non-GGUF variant; GGUF uses mmproj-F16 split file)
Audio	Parakeet-TDT-0.6B-v2	BF16 (same rationale)
Video	Parakeet-TDT-0.6B-v2 + frame sampler	BF16 (≤ 2 min, 256 frames @ 2 FPS)

NVIDIA's official FP8 / NVFP4 recipe keeps both encoders + the cross-modal MLP projectors in BF16 to preserve multimodal accuracy. We follow that convention in every quantized variant we ship.

Runtime quirks

MLX-LM (text-only)

This variant covers the LLM backbone only. Vision + audio encoders are NOT included — the MLX-VLM Nemotron-Omni model class is supported upstream as of mlx-vlm v0.4.4 (PR #1518, merged 2026-07-07; audio weight fixes in PR #1743, merged 2026-07-29).

Use the mlx_lm.generate API; enable_thinking is a runtime flag (see below).

Reasoning mode

enable_thinking defaults to True. To disable extended reasoning (e.g., for latency-sensitive cases), pass enable_thinking=False to the chat template / generate call. No separate "no-think" variant card exists — this is a runtime flag, not a model variant.

Quant trade-off (MLX lane)

Bits	Approx size	Use case	Recommendation
2-bit	~8.1 GB	Aggressive quantization	Very low-RAM Macs
3-bit	~11 GB	Lossy but small	Low-RAM Macs
4-bit	~13 GB	Balanced default	Recommended for most Macs
5-bit	~16 GB	Higher fidelity	Quality-sensitive
6-bit	~19 GB	Approaching FP16 quality	High-fidelity
8-bit	~24 GB	Near-lossless reference	Fidelity-critical work

(Current variant — 8bit — is bolded.)

Variants in this family

(Showing 56 sibling variants under majentik/nemotron3-nano-omni-30b-*. The current variant — TurboQuant-MLX-8bit — is bolded.)

Variant	Runtime	Approx size	Use case
TurboQuant-MLX-8bit	mlx-lm	~35 GB	Apple Silicon reference

About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's release labels, not distinct quantization algorithms — for any given tier, both brand repos carry byte-identical weights produced with the standard MLX / llama.cpp quantizers. No brand-specific speedup is claimed or measured.