Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF

English | ็น้ซ”ไธญๆ–‡

26991


English

Unsloth-style UD-Q4_K_XL quantization of huihui-ai/Huihui-Qwen3.6-27B-abliterated for llama.cpp, with the built-in MTP (Multi-Token Prediction) head fully preserved at Q8_0 for native speculative decoding.

Model Details

Item Value
Architecture Dense (27B), 64 layers โ€” 16 ร— (3 ร— Gated DeltaNet โ†’ FFN + 1 ร— Gated Attention โ†’ FFN) + MTP
Base model Qwen/Qwen3.6-27B
Fine-tuned by huihui-ai (abliteration)
Quantized by YuYu1015
Model size 16.96 GB (vs ~55.6 GB BF16 original) โ€” 5.21 BPW average
Format GGUF (single file), UD-style mixed precision via imatrix
Context length Up to 262,144 tokens (tested at 65,536)
Thinking mode Supported (enable_thinking: true/false)
MTP Built-in MTP head preserved at Q8_0 (drafter for --spec-type draft-mtp)
Multimodal Vision tower dropped by convert_hf_to_gguf.py (GGUF is text-only, saves ~1 GB)
llama.cpp b9200+ required (older builds miss hybrid GDN + MTP loading)

Per-Tensor Precision (UD Mask)

Tensor Type Reason
FFN (gate/up/down) Q4_K_M Bulk weight, high quantization tolerance
Attention Q/K/V Q6_K Sensitive to noise, upranked from base
Output projection Q6_K UD convention
Token embedding Q6_K UD convention
MTP / NextN head Q8_0 Drafter โ€” Q8_0 keeps acceptance near-lossless

Notes on the Source Checkpoint

  • huihui's abliterated checkpoint already contains the MTP head. Common wisdom says abliteration drops MTP, but convert_hf_to_gguf.py picks up all 4 blk.64.nextn.* tensors directly โ€” no Qwen-official MTP graft needed.
  • llama.cpp drops the vision tower automatically. The base model is Qwen3_5ForConditionalGeneration (multimodal), but GGUF only packs the LLM portion. Text-only inference works out of the box.

Serving with llama.cpp (RTX 3090)

llama-server \
  -m Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP.gguf \
  -fit off \
  -c 65536 \
  -np 1 \
  -fa on \
  -ngl 99 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --host 0.0.0.0 --port 8080
Flag Why
-fit off Newer llama.cpp's auto-fit aborts on Qwen3.6 hybrid GDN+MTP layout
-np 1 MTP path is single-slot only
--spec-type draft-mtp New flag (older versions used --spec-type mtp)
--spec-draft-n-max 2 Empirically the sweet spot โ€” n=3 regresses ~22% due to hybrid bug
--cache-type-k/v q8_0 KV cache compression, no measurable quality loss on this model

Performance (RTX 3090 + Xeon E5-2696 v3)

Configuration Throughput
No speculative decoding 39.15 tok/s
MTP --spec-draft-n-max 1 55 tok/s
MTP --spec-draft-n-max 2 60 tok/s (best)
MTP --spec-draft-n-max 3 55 tok/s (-22% vs n=2 ideal due to hybrid rollback bug)

Note on the CPU: the test rig uses an old Intel Xeon E5-2696 v3 (2.30 GHz, 2015). Modern CPUs (Zen 4 / Raptor Lake or newer) consistently push 30โ€“50% higher tok/s on the same RTX 3090 because llama.cpp's hybrid GDN path still touches CPU per step. These numbers are a floor, not a ceiling.

Common Deployment Pitfalls

  1. Auto-fit abort on newer llama.cpp: failed to fit params to free device memory, n_gpu_layers already set by user โ†’ add -fit off.
  2. Old llama.cpp can't load this: missing tensor 'blk.64.ssm_conv1d.weight' โ†’ upgrade to b9200+.

Safety Warning

This model has safety filtering removed (abliterated) and may generate sensitive, controversial, or inappropriate content. Users are solely responsible for all consequences arising from its use. Please ensure usage complies with local laws and ethical standards. Not suitable for public-facing or production applications.

Credits


็น้ซ”ไธญๆ–‡

huihui-ai/Huihui-Qwen3.6-27B-abliterated ็š„ Unsloth ้ขจๆ ผ UD-Q4_K_XL ้‡ๅŒ–็‰ˆๆœฌ๏ผŒ้‡ๅฐ llama.cpp ้ƒจ็ฝฒ๏ผŒไธฆๅฎŒๆ•ดไฟ็•™ๅ…งๅปบ MTP๏ผˆMulti-Token Prediction๏ผ‰head๏ผˆQ8_0๏ผ‰๏ผŒๅŽŸ็”Ÿๆ”ฏๆดๆŠ•ๆฉŸ่งฃ็ขผใ€‚

ๆจกๅž‹่ณ‡่จŠ

้ …็›ฎ ๆ•ธๅ€ผ
ๆžถๆง‹ Dense๏ผˆ27B๏ผ‰๏ผŒ64 ๅฑค โ€” 16 ร— (3 ร— Gated DeltaNet โ†’ FFN + 1 ร— Gated Attention โ†’ FFN) + MTP
ๅŸบ็คŽๆจกๅž‹ Qwen/Qwen3.6-27B
ๅพฎ่ชฟ่€… huihui-ai๏ผˆabliteration๏ผ‰
้‡ๅŒ–่€… YuYu1015
ๆจกๅž‹ๅคงๅฐ 16.96 GB๏ผˆๅŽŸ็‰ˆ BF16 ็ด„ 55.6 GB๏ผ‰โ€” ๅนณๅ‡ 5.21 BPW
ๆ ผๅผ GGUF๏ผˆๅ–ฎๆช”๏ผ‰๏ผŒUD ้ขจๆ ผๆททๅˆ็ฒพๅบฆ๏ผŒimatrix ๆ กๆบ–
Context ้•ทๅบฆ ๆœ€้ซ˜ 262,144 tokens๏ผˆๅฏฆๆธฌ 65,536๏ผ‰
ๆ€่€ƒๆจกๅผ ๆ”ฏๆด๏ผˆenable_thinking: true/false๏ผ‰
MTP ๅ…งๅปบ MTP head ไปฅ Q8_0 ไฟ็•™๏ผˆไฝœ็‚บ --spec-type draft-mtp ็š„่‰็จฟๅ™จ๏ผ‰
ๅคšๆจกๆ…‹ convert_hf_to_gguf.py ่‡ชๅ‹•่ทณ้Ž่ฆ–่ฆบๅก”๏ผˆGGUF ็ด”ๆ–‡ๅญ—๏ผŒ็œ ~1 GB๏ผ‰
llama.cpp **ๅฟ…้ ˆ b9200+**๏ผˆ่ˆŠ็‰ˆไธๆ”ฏๆด hybrid GDN + MTP ่ผ‰ๅ…ฅ๏ผ‰

้€ Tensor ็ฒพๅบฆ๏ผˆUD ้ฎ็ฝฉ๏ผ‰

Tensor ้กžๅž‹ ็†็”ฑ
FFN๏ผˆgate/up/down๏ผ‰ Q4_K_M ๅคงๅฎ—ๆฌŠ้‡๏ผŒ้‡ๅŒ–ๅฎนๅฟๅบฆ้ซ˜
Attention Q/K/V Q6_K ๅฐ้›œ่จŠๆ•ๆ„Ÿ๏ผŒ็”ฑๅŸบ็คŽ้กžๅž‹ๅ‡็ดš
Output projection Q6_K UD ๆ…ฃไพ‹
Token embedding Q6_K UD ๆ…ฃไพ‹
MTP / NextN head Q8_0 ่‰็จฟๅ™จ โ€” Q8_0 ๅนพไนŽ็„กๆ็ถญๆŒ acceptance

ไพ†ๆบ Checkpoint ่จป่จ˜

  • huihui abliterated checkpoint ๅทฒๅ…งๅซ MTP headใ€‚ ไธปๆต่ชชๆณ•ๆ˜ฏใ€Œabliteration ๆœƒๅผ„ไธŸ MTPใ€๏ผŒไฝ†ๅฏฆ้š› convert_hf_to_gguf.py ๅฏ็›ดๆŽฅๆŠ“ๅˆฐ blk.64.nextn.* ๅ…ฑ 4 ๅ€‹ tensor โ€” ไธ้œ€ๅซๆŽฅ Qwen ๅฎ˜ๆ–น MTPใ€‚
  • llama.cpp ่‡ชๅ‹•่ทณ้Ž่ฆ–่ฆบๅก”ใ€‚ ๅŸบ็คŽๆจกๅž‹ๆ˜ฏ Qwen3_5ForConditionalGeneration๏ผˆๅคšๆจกๆ…‹๏ผ‰๏ผŒไฝ† GGUF ๅชๅŒ… LLM ้ƒจๅˆ†ใ€‚็ด”ๆ–‡ๅญ—ๆŽจ็†็›ดๆŽฅๅฏ็”จใ€‚

ไฝฟ็”จ llama.cpp ้ƒจ็ฝฒ๏ผˆRTX 3090๏ผ‰

llama-server \
  -m Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP.gguf \
  -fit off \
  -c 65536 \
  -np 1 \
  -fa on \
  -ngl 99 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --host 0.0.0.0 --port 8080
Flag ็”จ้€”
-fit off ๆ–ฐ็‰ˆ llama.cpp ็š„ auto-fit ๅฐ Qwen3.6 hybrid GDN+MTP ๆœƒ abort
-np 1 MTP ่ทฏๅพ‘ๅช่ƒฝๅ–ฎ slot
--spec-type draft-mtp ๆ–ฐ็‰ˆ flag๏ผˆ่ˆŠ็‰ˆๅซ --spec-type mtp๏ผ‰
--spec-draft-n-max 2 ๅฏฆๆธฌ็”œ่œœ้ปž โ€” n=3 ๅ›  hybrid bug ๅ›ž้€€็ด„ 22%
--cache-type-k/v q8_0 KV cache ๅฃ“็ธฎ๏ผŒๆญคๆจกๅž‹ไธŠ็„กๅฏ้‡ๆธฌๅ“่ณชๆๅคฑ

ๆ•ˆ่ƒฝ๏ผˆRTX 3090 + Xeon E5-2696 v3๏ผ‰

่จญๅฎš ้€Ÿๅบฆ
็„กๆŠ•ๆฉŸ่งฃ็ขผ 39.15 tok/s
MTP --spec-draft-n-max 1 55 tok/s
MTP --spec-draft-n-max 2 60 tok/s๏ผˆๆœ€ไฝณ๏ผ‰
MTP --spec-draft-n-max 3 55 tok/s๏ผˆhybrid rollback bug ๆ‹–็ดฏ๏ผŒ่ผƒ n=2 ็†่ซ–ๅ€ผ -22%๏ผ‰

CPU ่ชชๆ˜Ž๏ผš ๆธฌ่ฉฆๆฉŸไฝฟ็”จ 2015 ๅนด็š„่ˆŠ Intel Xeon E5-2696 v3๏ผˆ2.30 GHz๏ผ‰ใ€‚ๆ›ๆˆ็พไปฃ CPU๏ผˆZen 4 / Raptor Lake ไปฅไธŠ๏ผ‰ๅœจๅŒๆจฃ RTX 3090 ไธŠ้€šๅธธๅฏๅ†ๆๅ‡ 30โ€“50% tok/s๏ผŒๅ› ็‚บ llama.cpp ็š„ hybrid GDN ่ทฏๅพ‘ๆฏๆญฅไป้œ€ CPU ไป‹ๅ…ฅใ€‚้€™ๅ€‹ๆ•ธๅญ—ๆ˜ฏไธ‹้™๏ผŒไธๆ˜ฏไธŠ้™ใ€‚

้ƒจ็ฝฒๅธธ่ฆ‹ๅœฐ้›ท

  1. ๆ–ฐ็‰ˆ llama.cpp auto-fit abort๏ผš failed to fit params to free device memory, n_gpu_layers already set by user โ†’ ๅŠ  -fit offใ€‚
  2. ่ˆŠ็‰ˆ llama.cpp ่ผ‰ไธไบ†๏ผš missing tensor 'blk.64.ssm_conv1d.weight' โ†’ ๅ‡็ดšๅˆฐ **b9200+**ใ€‚

ๅฎ‰ๅ…จ่ญฆๅ‘Š

ๆญคๆจกๅž‹ๅทฒ็งป้™คๅฎ‰ๅ…จ้ŽๆฟพๆฉŸๅˆถ๏ผˆabliterated๏ผ‰๏ผŒๅฏ่ƒฝ็”ข็”Ÿๆ•ๆ„Ÿใ€็ˆญ่ญฐๆ€งๆˆ–ไธ็•ถๅ…งๅฎนใ€‚ไฝฟ็”จ่€…้ ˆ่‡ช่กŒๆ‰ฟๆ“”ๆ‰€ๆœ‰้ขจ้šช่ˆ‡ๆณ•ๅพ‹่ฒฌไปป๏ผŒไธฆ็ขบไฟไฝฟ็”จๆ–นๅผ็ฌฆๅˆ็•ถๅœฐๆณ•่ฆ่ˆ‡ๅ€ซ็†ๆจ™ๆบ–ใ€‚ไธ้ฉ็”จๆ–ผๅ…ฌ้–‹ๆˆ–็”Ÿ็”ข็’ฐๅขƒใ€‚

่‡ด่ฌ

Downloads last month
102
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF

Base model

Qwen/Qwen3.6-27B
Quantized
(28)
this model

Collection including YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF