--- base_model: bottlecapai/ThinkingCap-Qwen3.6-27B base_model_relation: quantized new_version: bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4 library_name: transformers tags: - qwen3_6 - fp4 - nvfp4 - compressed-tensors - vllm - token-efficient - efficient-thinking license: apache-2.0 extra_gated_heading: "Request access to the BottleCap AI model" extra_gated_prompt: "Please fill out the form below to request access. Name, Company Name, and Company Email are optional — just enter NA if you'd prefer not to share. Your email may be used to send you information about BottleCap AI model updates and early access before public release." extra_gated_fields: Name: text Company Name: text Company Email: text extra_gated_button_content: "Agree and request access" --- ThinkingCap — BottleCap AI 👉 **Newer release: [ThinkingCap-Qwen3.8-27B-NVFP4](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4)** — NVFP4 weight-only build of [ThinkingCap-Qwen3.8-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B), the second model in the ThinkingCap series, finetuned from Qwen3.8-27B. Read [the blog post](https://bottlecapai.com/post/thinkingcap-qwen3-8-27b/). Other quantized builds: [FP8](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-FP8) · [NVFP4A4-AWQ](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ) · [GGUF](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF) · [MLX 4-bit DWQ](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-MLX-4bit-DWQ). 👈 # bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4 FP4 (NVFP4) quantization of [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B) — capability of Qwen3.6-27B with **50% less** thinking tokens on average, achieved by finetuning [Qwen3.6-27B (Qwen Team, 2026)](https://huggingface.co/Qwen/Qwen3.6-27B) while preserving the original answer quality and style. ➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B). ## About this quantization Weights are stored at **4 bits** in NVIDIA's [NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/) format (E2M1 values with per-group-of-16 FP8 scales) while activations stay in bf16 — weight-only `NVFP4A16` (W4A16), produced data-free with [llm-compressor](https://github.com/vllm-project/llm-compressor) in the [compressed-tensors](https://github.com/neuralmagic/compressed-tensors) `nvfp4-pack-quantized` format that vLLM loads natively. ≈19 GB instead of ≈55 GB bf16 — **~2.9× smaller**. Being weight-only (W4A16), it serves through vLLM's `MarlinNvFp4LinearKernel` on both Hopper (H100/H200) and Blackwell: the 4-bit weights unpack to bf16 for compute, so the decode win is **memory bandwidth** (≈2× bf16 tok/s), not FP4 tensor cores (native cutlass FP4 compute would require a W4A4 / FP4-activation checkpoint). The model's **MTP (multi-token-prediction) head is preserved**, so self-speculative decoding works (≈3.2 accepted tokens/step). Kept in bf16: `lm_head`, the **MTP head**, the **vision tower**, and the two small Gated-DeltaNet input gates `linear_attn.in_proj_a` / `in_proj_b` (48-wide output, not a multiple of the 4-bit kernel's tile size and fused to width 96 at serve time, so quantizing them breaks loading on non-Blackwell GPUs; keeping them bf16 costs ~60 MB). The remaining group-quantizable Linear layers — MLP, self-attention, and the other linear-attention projections (`in_proj_qkv` / `in_proj_z` / `out_proj`) — carry the 4-bit weights. ## Usage ```bash vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4 ``` ### Speculative decoding (MTP) The MTP (multi-token-prediction / NextN) head is kept in bf16, so vLLM can run self-speculative decoding for a decode speed-up — no separate draft model needed. Add `--speculative-config` when serving (requires vLLM ≥ 0.24.0): ```bash vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` Speculative decoding is lossless — the output is identical to standard decoding. It accepts ≈3.2 drafts per verify step here, for a further ≈1.5× on top of the finetune's token savings (see below). Served on **vLLM**; SGLang cannot currently load weight-only NVFP4 for this architecture. For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at [bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF); for near-lossless FP8 at half the memory (loads in vLLM **and** SGLang), see [bottlecapai/ThinkingCap-Qwen3.6-27B-FP8](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B-FP8). ## Expected performance Measured on our internal serving harness on **8× RTX PRO 6000 (Blackwell)** with **vLLM 0.25.0**, one configuration per GPU. Each cell is **N=200 questions/dataset × 3 generation seeds** over a fixed problem subset, batch size 16, sampled decoding (temperature 1.0, top_p 0.95, top_k 20); `acc` is the mean ± 95% CI across the seeds. Generation budgets are 24,576 tokens (MMLU-Pro) and 16,384 tokens (RealWorldQA). FP4 matches the bf16 finetune's accuracy (all 95% CIs overlap) while decoding **≈2.3× faster** (52.7 vs 22.6 tok/s, MMLU-Pro standard — the memory-bandwidth win of 4-bit weights via the Marlin kernel), and MTP self-speculative decoding (≈3.1–3.2 accepted tokens per verify step) adds a further ≈1.5× — stacking with the finetune's token savings to **≈4.7–8.4× faster per task** than the unquantized base (MMLU-Pro 11.6 s vs 97.3 s; RealWorldQA 7.3 s vs 34.2 s). All timings are measured per request during the eval. `median tokens` = median completion length (the finetune's lever); `tok/s` = per-request steady-state decode rate (prefill excluded via time-to-first-token); `task s` = measured end-to-end wall-clock per request under the batch-size-16 concurrency; `speedup` = task s of the base model in standard decoding ÷ task s of the row. [`unsloth NVFP4`](https://huggingface.co/unsloth/Qwen3.6-27B-NVFP4) and [`NVIDIA NVFP4`](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4) are community NVFP4 quants of the *base* model, shown for comparison; `ThinkingCap-FP8` is our FP8 sibling ([bottlecapai/ThinkingCap-Qwen3.6-27B-FP8](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B-FP8)). **NVFP4 kernel (important for reading `tok/s`).** On this hardware the vLLM NVFP4 kernel is chosen by the **checkpoint's FP4 scheme**, not a flag, so the NVFP4 rows are *not* on identical footing: **this model** and **NVIDIA NVFP4** are weight-only **W4A16** → `MarlinNvFp4LinearKernel` (4-bit weights unpacked for compute; a memory-bandwidth decode win), whereas **unsloth NVFP4** is **W4A4** (weights *and* activations FP4, `mixed-precision`) → `FlashInferCutlassNvFp4LinearKernel`, the cutlass/flashinfer FP4 GEMM. vLLM has no cutlass path for weight-only W4A16 on sm_120, so those fall back to Marlin — a checkpoint-format property, bit-exact-ish and not a source of accuracy difference. On Hopper (H100/H200) there is no native FP4 compute at all, so every NVFP4 serves via Marlin weight-unpack; the cutlass FP4 tensor-core advantage (W4A4) shows mostly on compute-bound prefill / high batch and on datacenter Blackwell. **MMLU-Pro (reasoning)** | config | acc | median tokens | tok/s | task s | speedup | accept_len | |---|---|---|---|---|---|---| | Qwen3.6-27B base bf16 · standard | 0.902 ± 0.019 | 2186 | 22.6 | 97.3 | 1.00× | — | | Qwen3.6-27B base bf16 · MTP | 0.892 ± 0.031 | 2117 | 47.7 | 45.6 | 2.13× | 3.23 | | unsloth NVFP4 (base) · standard | 0.878 ± 0.038 | 2186 | 44.5 | 49.1 | 1.98× | — | | unsloth NVFP4 (base) · MTP | 0.888 ± 0.047 | 2123 | 87.4 | 25.4 | 3.83× | 3.21 | | NVIDIA NVFP4 (base) · standard | 0.892 ± 0.026 | 2032 | 51.0 | 40.0 | 2.43× | — | | NVIDIA NVFP4 (base) · MTP | 0.907 ± 0.036 | 1996 | 93.7 | 21.6 | 4.50× | 3.23 | | ThinkingCap bf16 · standard | 0.890 ± 0.022 | 963 | 23.4 | 41.6 | 2.34× | — | | ThinkingCap bf16 · MTP | 0.895 ± 0.012 | 910 | 51.1 | 18.8 | 5.18× | 3.26 | | **ThinkingCap-NVFP4 · standard** | **0.882 ± 0.031** | **926** | **52.7** | **17.7** | **5.50×** | — | | **ThinkingCap-NVFP4 · MTP** | **0.885 ± 0.033** | **878** | **81.3** | **11.6** | **8.39×** | **3.24** | **RealWorldQA (vision)** | config | acc | median tokens | tok/s | task s | speedup | accept_len | |---|---|---|---|---|---|---| | Qwen3.6-27B base bf16 · standard | 0.802 ± 0.019 | 718 | 21.7 | 34.2 | 1.00× | — | | Qwen3.6-27B base bf16 · MTP | 0.798 ± 0.064 | 752 | 42.2 | 18.7 | 1.83× | 3.09 | | unsloth NVFP4 (base) · standard | 0.788 ± 0.019 | 777 | 43.8 | 18.7 | 1.83× | — | | unsloth NVFP4 (base) · MTP | 0.783 ± 0.040 | 731 | 79.2 | 10.9 | 3.14× | 3.08 | | NVIDIA NVFP4 (base) · standard | 0.785 ± 0.012 | 732 | 46.6 | 17.3 | 1.98× | — | | NVIDIA NVFP4 (base) · MTP | 0.790 ± 0.050 | 700 | 79.8 | 10.5 | 3.26× | 3.09 | | ThinkingCap bf16 · standard | 0.818 ± 0.047 | 344 | 21.5 | 17.4 | 1.97× | — | | ThinkingCap bf16 · MTP | 0.805 ± 0.045 | 372 | 42.5 | 9.9 | 3.45× | 3.09 | | **ThinkingCap-NVFP4 · standard** | **0.802 ± 0.007** | **317** | **40.9** | **9.2** | **3.72×** | — | | **ThinkingCap-NVFP4 · MTP** | **0.800 ± 0.012** | **320** | **55.5** | **7.3** | **4.68×** | **3.09** |