--- license: apache-2.0 license_link: https://huggingface.co/Qwen/Qwen3.5-35B-A3B/blob/main/LICENSE pipeline_tag: image-text-to-text base_model: - huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated tags: - gguf - llama.cpp - qwen3_5_moe - qwen - abliterated - uncensored - tq3_4s - turboquant thumbnail: ./model-card.png --- ![model card](./model-card.png) # Huihui-Qwen35-A35B-Ablit-TQ3_4S GGUF TurboQuant conversion of [huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated). This is a 35B-A3B Qwen3.5 MoE abliterated model quantized for the public TurboQuant llama.cpp fork with `TQ3_4S` weights. The token embedding and output tensors are kept at `Q6_K` for compatibility and quality on this 35B MoE architecture. ## Files | File | Description | |---|---| | `Huihui-Qwen35-A35B-Ablit-TQ3_4S.gguf` | Main GGUF model, mostly `TQ3_4S`, 4.09 BPW | | `mmproj-Qwen35-A35B-f16.gguf` | Compatible Qwen3.5-35B-A35B multimodal projector | | `chat_template.jinja` | Qwen3.5 chat template | | `chat_template-vl-think.jinja` | Vision-language thinking template from the upstream Huihui repo | | `config.json` | Source model config metadata | | `generation_config.json` | Source generation config | | `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt` | Tokenizer files | | `preprocessor_config.json`, `video_preprocessor_config.json` | Vision/video preprocessor metadata | | `model-card.png` | Repository thumbnail/card image | ## Quantization - Source: `huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated` - Converter: llama.cpp HF-to-GGUF converter - Quantizer: llama.cpp TQ3 build `ba3357e93` - Main tensor type: `TQ3_4S` - Embedding/output tensor type: `Q6_K` - Output size: 16.52 GiB - Reported quant size: 16912.31 MiB, 4.09 BPW Quantization command: ```bash ./build/bin/llama-quantize \ --token-embedding-type q6_K --output-tensor-type q6_K \ Huihui-Qwen3.5-35B-A35B-abliterated-BF16.gguf \ Huihui-Qwen35-A35B-Ablit-TQ3_4S.gguf \ TQ3_4S 16 ``` ## llama.cpp Usage ## Runtime Requirement This model requires the public TurboQuant runtime fork: - https://github.com/turbo-tan/llama.cpp-tq3 It will not load correctly on stock `llama.cpp` or other runtimes that do not include `TQ3_4S`. Text-only: ```bash ./build/bin/llama-server \ -m /path/to/Huihui-Qwen35-A35B-Ablit-TQ3_4S.gguf \ -ngl 30 -c 2048 -np 1 \ --jinja \ --chat-template-file /path/to/chat_template.jinja ``` With the multimodal projector: ```bash ./build/bin/llama-server \ -m /path/to/Huihui-Qwen35-A35B-Ablit-TQ3_4S.gguf \ --mmproj /path/to/mmproj-Qwen35-A35B-f16.gguf \ -ngl 30 -c 2048 -np 1 \ --jinja \ --chat-template-file /path/to/chat_template-vl-think.jinja ``` For text-only chat, use the embedded template or pass `--chat-template-file chat_template.jinja`. For vision-language use, keep `mmproj-Qwen35-A35B-f16.gguf`, `chat_template-vl-think.jinja`, `preprocessor_config.json`, and `video_preprocessor_config.json` in the repository. On an RTX 5060 Ti 16GB, full offload did not fit. The highest tested offload for this GGUF was `-ngl 30`; `-ngl 40` and `-ngl 99` failed to load due VRAM. ## Validation Smoke tests were run with the TurboQuant llama.cpp fork listed above. Runtime: - GPU: RTX 5060 Ti 16GB - Context: `-c 2048` - Offload: `-ngl 30` - Chat settings: `temperature=0`, `max_tokens=256`, `chat_template_kwargs.enable_thinking=false` Load probe: | Offload | Result | pp128 | |---:|---|---:| | `ngl=0` | loads | 127.93 tok/s | | `ngl=20` | loads | 210.56 tok/s | | `ngl=30` | loads | 298.77 tok/s | | `ngl=40` | fails on 16GB VRAM | n/a | | `ngl=99` | fails on 16GB VRAM | n/a | 10-question chat smoke: | Model | Manual pass | Notes | |---|---:|---| | Huihui-Qwen35-A35B-Ablit-TQ3_4S | 9/10 | Same score as normal Qwen3.5-35B-A35B Q4_K_M on this smoke set | | Normal Qwen3.5-35B-A35B Q4_K_M | 9/10 | Used as local quality baseline | Both models missed the same Python list prompt in the lightweight smoke suite. Treat this as a smoke-pass, not a full release quality benchmark. ## Safety Notice This is an abliterated/uncensored model. It may produce sensitive, controversial, unsafe, or otherwise inappropriate outputs. Use in controlled environments, review outputs carefully, and follow applicable laws and policies. ## Acknowledgements - Base abliterated model: [huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated) - Original base model: [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) - Runtime/format: TurboQuant llama.cpp GGUF - Quantization format: TurboQuant `TQ3_4S`