Instructions to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: ./build/bin/llama cli -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Use Docker
docker model run hf.co/YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
- Llama new
- LM Studio
- Jan
- vLLM
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
- Ollama
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with Ollama:
ollama run hf.co/YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
- Unsloth Desktop
- Pi
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with Docker Model Runner:
docker model run hf.co/YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
- Lemonade
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Run and chat with the model
lemonade run user.Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF-UD-Q4_K_XL
List all available models
lemonade list
- Hermes Agent
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF:UD-Q4_K_XL" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF
English
Unsloth-style UD-Q4_K_XL quantization of huihui-ai/Huihui-Qwen3.6-27B-abliterated for llama.cpp, with the built-in MTP (Multi-Token Prediction) head fully preserved at Q8_0 for native speculative decoding.
Model Details
| Item | Value |
|---|---|
| Architecture | Dense (27B), 64 layers โ 16 ร (3 ร Gated DeltaNet โ FFN + 1 ร Gated Attention โ FFN) + MTP |
| Base model | Qwen/Qwen3.6-27B |
| Fine-tuned by | huihui-ai (abliteration) |
| Quantized by | YuYu1015 |
| Model size | 16.96 GB (vs ~55.6 GB BF16 original) โ 5.21 BPW average |
| Format | GGUF (single file), UD-style mixed precision via imatrix |
| Context length | Up to 262,144 tokens (tested at 65,536) |
| Thinking mode | Supported (enable_thinking: true/false) |
| MTP | Built-in MTP head preserved at Q8_0 (drafter for --spec-type draft-mtp) |
| Multimodal | Vision tower dropped by convert_hf_to_gguf.py (GGUF is text-only, saves ~1 GB) |
| llama.cpp | b9200+ required (older builds miss hybrid GDN + MTP loading) |
Per-Tensor Precision (UD Mask)
| Tensor | Type | Reason |
|---|---|---|
FFN (gate/up/down) |
Q4_K_M | Bulk weight, high quantization tolerance |
Attention Q/K/V |
Q6_K | Sensitive to noise, upranked from base |
| Output projection | Q6_K | UD convention |
| Token embedding | Q6_K | UD convention |
| MTP / NextN head | Q8_0 | Drafter โ Q8_0 keeps acceptance near-lossless |
Notes on the Source Checkpoint
- huihui's abliterated checkpoint already contains the MTP head. Common wisdom says abliteration drops MTP, but
convert_hf_to_gguf.pypicks up all 4blk.64.nextn.*tensors directly โ no Qwen-official MTP graft needed. - llama.cpp drops the vision tower automatically. The base model is
Qwen3_5ForConditionalGeneration(multimodal), but GGUF only packs the LLM portion. Text-only inference works out of the box.
Serving with llama.cpp (RTX 3090)
llama-server \
-m Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP.gguf \
-fit off \
-c 65536 \
-np 1 \
-fa on \
-ngl 99 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--host 0.0.0.0 --port 8080
| Flag | Why |
|---|---|
-fit off |
Newer llama.cpp's auto-fit aborts on Qwen3.6 hybrid GDN+MTP layout |
-np 1 |
MTP path is single-slot only |
--spec-type draft-mtp |
New flag (older versions used --spec-type mtp) |
--spec-draft-n-max 2 |
Empirically the sweet spot โ n=3 regresses ~22% due to hybrid bug |
--cache-type-k/v q8_0 |
KV cache compression, no measurable quality loss on this model |
Performance (RTX 3090 + Xeon E5-2696 v3)
| Configuration | Throughput |
|---|---|
| No speculative decoding | 39.15 tok/s |
MTP --spec-draft-n-max 1 |
55 tok/s |
MTP --spec-draft-n-max 2 |
60 tok/s (best) |
MTP --spec-draft-n-max 3 |
55 tok/s (-22% vs n=2 ideal due to hybrid rollback bug) |
Note on the CPU: the test rig uses an old Intel Xeon E5-2696 v3 (2.30 GHz, 2015). Modern CPUs (Zen 4 / Raptor Lake or newer) consistently push 30โ50% higher tok/s on the same RTX 3090 because llama.cpp's hybrid GDN path still touches CPU per step. These numbers are a floor, not a ceiling.
Common Deployment Pitfalls
- Auto-fit abort on newer llama.cpp:
failed to fit params to free device memory, n_gpu_layers already set by userโ add-fit off. - Old llama.cpp can't load this:
missing tensor 'blk.64.ssm_conv1d.weight'โ upgrade to b9200+.
Safety Warning
This model has safety filtering removed (abliterated) and may generate sensitive, controversial, or inappropriate content. Users are solely responsible for all consequences arising from its use. Please ensure usage complies with local laws and ethical standards. Not suitable for public-facing or production applications.
Credits
- Original Model: Qwen/Qwen3.6-27B by Alibaba Qwen Team
- Abliteration: huihui-ai
- GGUF UD Quantization: YuYu1015
- UD Recipe Inspiration: Unsloth Dynamic Quants
็น้ซไธญๆ
huihui-ai/Huihui-Qwen3.6-27B-abliterated ็ Unsloth ้ขจๆ ผ UD-Q4_K_XL ้ๅ็ๆฌ๏ผ้ๅฐ llama.cpp ้จ็ฝฒ๏ผไธฆๅฎๆดไฟ็ๅ งๅปบ MTP๏ผMulti-Token Prediction๏ผhead๏ผQ8_0๏ผ๏ผๅ็ๆฏๆดๆๆฉ่งฃ็ขผใ
ๆจกๅ่ณ่จ
| ้ ็ฎ | ๆธๅผ |
|---|---|
| ๆถๆง | Dense๏ผ27B๏ผ๏ผ64 ๅฑค โ 16 ร (3 ร Gated DeltaNet โ FFN + 1 ร Gated Attention โ FFN) + MTP |
| ๅบ็คๆจกๅ | Qwen/Qwen3.6-27B |
| ๅพฎ่ชฟ่ | huihui-ai๏ผabliteration๏ผ |
| ้ๅ่ | YuYu1015 |
| ๆจกๅๅคงๅฐ | 16.96 GB๏ผๅ็ BF16 ็ด 55.6 GB๏ผโ ๅนณๅ 5.21 BPW |
| ๆ ผๅผ | GGUF๏ผๅฎๆช๏ผ๏ผUD ้ขจๆ ผๆททๅ็ฒพๅบฆ๏ผimatrix ๆ กๆบ |
| Context ้ทๅบฆ | ๆ้ซ 262,144 tokens๏ผๅฏฆๆธฌ 65,536๏ผ |
| ๆ่ๆจกๅผ | ๆฏๆด๏ผenable_thinking: true/false๏ผ |
| MTP | ๅ
งๅปบ MTP head ไปฅ Q8_0 ไฟ็๏ผไฝ็บ --spec-type draft-mtp ็่็จฟๅจ๏ผ |
| ๅคๆจกๆ | convert_hf_to_gguf.py ่ชๅ่ทณ้่ฆ่ฆบๅก๏ผGGUF ็ดๆๅญ๏ผ็ ~1 GB๏ผ |
| llama.cpp | **ๅฟ ้ b9200+**๏ผ่็ไธๆฏๆด hybrid GDN + MTP ่ผๅ ฅ๏ผ |
้ Tensor ็ฒพๅบฆ๏ผUD ้ฎ็ฝฉ๏ผ
| Tensor | ้กๅ | ็็ฑ |
|---|---|---|
FFN๏ผgate/up/down๏ผ |
Q4_K_M | ๅคงๅฎๆฌ้๏ผ้ๅๅฎนๅฟๅบฆ้ซ |
Attention Q/K/V |
Q6_K | ๅฐ้่จๆๆ๏ผ็ฑๅบ็ค้กๅๅ็ด |
| Output projection | Q6_K | UD ๆ ฃไพ |
| Token embedding | Q6_K | UD ๆ ฃไพ |
| MTP / NextN head | Q8_0 | ่็จฟๅจ โ Q8_0 ๅนพไน็กๆ็ถญๆ acceptance |
ไพๆบ Checkpoint ่จป่จ
- huihui abliterated checkpoint ๅทฒๅ
งๅซ MTP headใ ไธปๆต่ชชๆณๆฏใabliteration ๆๅผไธ MTPใ๏ผไฝๅฏฆ้
convert_hf_to_gguf.pyๅฏ็ดๆฅๆๅฐblk.64.nextn.*ๅ ฑ 4 ๅ tensor โ ไธ้ๅซๆฅ Qwen ๅฎๆน MTPใ - llama.cpp ่ชๅ่ทณ้่ฆ่ฆบๅกใ ๅบ็คๆจกๅๆฏ
Qwen3_5ForConditionalGeneration๏ผๅคๆจกๆ ๏ผ๏ผไฝ GGUF ๅชๅ LLM ้จๅใ็ดๆๅญๆจ็็ดๆฅๅฏ็จใ
ไฝฟ็จ llama.cpp ้จ็ฝฒ๏ผRTX 3090๏ผ
llama-server \
-m Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP.gguf \
-fit off \
-c 65536 \
-np 1 \
-fa on \
-ngl 99 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--host 0.0.0.0 --port 8080
| Flag | ็จ้ |
|---|---|
-fit off |
ๆฐ็ llama.cpp ็ auto-fit ๅฐ Qwen3.6 hybrid GDN+MTP ๆ abort |
-np 1 |
MTP ่ทฏๅพๅช่ฝๅฎ slot |
--spec-type draft-mtp |
ๆฐ็ flag๏ผ่็ๅซ --spec-type mtp๏ผ |
--spec-draft-n-max 2 |
ๅฏฆๆธฌ็่้ป โ n=3 ๅ hybrid bug ๅ้็ด 22% |
--cache-type-k/v q8_0 |
KV cache ๅฃ็ธฎ๏ผๆญคๆจกๅไธ็กๅฏ้ๆธฌๅ่ณชๆๅคฑ |
ๆ่ฝ๏ผRTX 3090 + Xeon E5-2696 v3๏ผ
| ่จญๅฎ | ้ๅบฆ |
|---|---|
| ็กๆๆฉ่งฃ็ขผ | 39.15 tok/s |
MTP --spec-draft-n-max 1 |
55 tok/s |
MTP --spec-draft-n-max 2 |
60 tok/s๏ผๆไฝณ๏ผ |
MTP --spec-draft-n-max 3 |
55 tok/s๏ผhybrid rollback bug ๆ็ดฏ๏ผ่ผ n=2 ็่ซๅผ -22%๏ผ |
CPU ่ชชๆ๏ผ ๆธฌ่ฉฆๆฉไฝฟ็จ 2015 ๅนด็่ Intel Xeon E5-2696 v3๏ผ2.30 GHz๏ผใๆๆ็พไปฃ CPU๏ผZen 4 / Raptor Lake ไปฅไธ๏ผๅจๅๆจฃ RTX 3090 ไธ้ๅธธๅฏๅๆๅ 30โ50% tok/s๏ผๅ ็บ llama.cpp ็ hybrid GDN ่ทฏๅพๆฏๆญฅไป้ CPU ไปๅ ฅใ้ๅๆธๅญๆฏไธ้๏ผไธๆฏไธ้ใ
้จ็ฝฒๅธธ่ฆๅฐ้ท
- ๆฐ็ llama.cpp auto-fit abort๏ผ
failed to fit params to free device memory, n_gpu_layers already set by userโ ๅ-fit offใ - ่็ llama.cpp ่ผไธไบ๏ผ
missing tensor 'blk.64.ssm_conv1d.weight'โ ๅ็ดๅฐ **b9200+**ใ
ๅฎๅ จ่ญฆๅ
ๆญคๆจกๅๅทฒ็งป้คๅฎๅ จ้ๆฟพๆฉๅถ๏ผabliterated๏ผ๏ผๅฏ่ฝ็ข็ๆๆใ็ญ่ญฐๆงๆไธ็ถๅ งๅฎนใไฝฟ็จ่ ้ ่ช่กๆฟๆๆๆ้ขจ้ช่ๆณๅพ่ฒฌไปป๏ผไธฆ็ขบไฟไฝฟ็จๆนๅผ็ฌฆๅ็ถๅฐๆณ่ฆ่ๅซ็ๆจๆบใไธ้ฉ็จๆผๅ ฌ้ๆ็็ข็ฐๅขใ
่ด่ฌ
- ๅๅงๆจกๅ๏ผQwen/Qwen3.6-27B๏ผAlibaba Qwen ๅ้
- ๅปๅฏฉๆฅ๏ผhuihui-ai
- GGUF UD ้ๅ๏ผYuYu1015
- UD Recipe ้ๆ๏ผUnsloth Dynamic Quants
- Downloads last month
- 102
4-bit
Model tree for YuYu1015/Huihui-Qwen3.6-27B-abliterated-UD-Q4_K_XL-MTP-GGUF
Base model
Qwen/Qwen3.6-27B