Instructions to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q # Run inference directly in the terminal: llama cli -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q # Run inference directly in the terminal: llama cli -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q # Run inference directly in the terminal: ./build/bin/llama cli -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Use Docker
docker model run hf.co/jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
- Llama new
- LM Studio
- Jan
- vLLM
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
- SGLang
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with Ollama:
ollama run hf.co/jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
- Unsloth Desktop
- Pi
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with Docker Model Runner:
docker model run hf.co/jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
- Lemonade
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Run and chat with the model
lemonade run user.Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF-MXFP4_MOE_Q
List all available models
lemonade list
- Hermes Agent
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF:MXFP4_MOE_Q" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
๐ Qwable-v1-35B-A3B - Custom Mixed Precision GGUFs with Imatrix
Qwen + Fable ยท An open-weights agentic coding model. 35B Mixture-of-Experts (3B active), built by layering Claude Fable-5 agentic tool-use behavior on top of a Claude Opus 4.7 reasoning distill of Qwen3.6-35B-A3B.
This repository contains custom, highly optimized, multi-tier mixed precision GGUF weights for lordx64/Qwable-v1.
โน๏ธ For advanced agentic and programming tasks, I highly recommend upgrading to Ornith-1.0-35B for significantly better performance.
Qwable-v1 is a 35B Mixture-of-Experts (3B active) hybrid architecture alternating between standard Attention and Mamba State-Space (SSM) blocks.
Distilled heavily from Claude 4.7 Opus reasoning and Claude Fable-5 agentic traces, on top of Qwen3.6-35B-A3B.
These quants were generated using an external importance matrix bartowski/lordx64_Qwable-v1-GGUF/lordx64_Qwable-v1-imatrix.gguf and manual layer targeting to maximize quality while shrinking the massive VRAM footprint of the Mixture of Experts layers.
๐ GGUF Files
In order of quality:
| Filename | Size | Quants |
|---|---|---|
| Qwable-v1-35B-A3B-MXFP4_MOE_Q8_0_F16-Imatrix.gguf | 21.3 GB | MXFP4_MOE + Q8_0 + F16 |
| Qwable-v1-35B-A3B-MXFP4_MOE_Q8_0-Imatrix.gguf | 20.3 GB | MXFP4_MOE + Q8_0 |
Update 2026-06-27:
MXFP4_MOE_Q8_0_F16-ssm_alpha&ssm_betais nowF16, to lessen looping at high context.MXFP4_MOE_Q8_0_F16-attn_outputis nowQ8_0, as it was previouslyF16by accident due to incorrect--tensor-typeregex.
๐ Precision Matrix & Flavor Variations
Standard global quantization presets (like stock MXFP4_MOE) compress the backbone layers uniformly, which degrades the delicate reasoning capabilities of advanced agent models.
This repository provides two distinct manual configuration layouts to balance precision and memory constraints:
1. The Tri-Quant Hybrid Flavor (MXFP4 + Q8_0 + F16)
Qwable-v1-35B-A3B-MXFP4_MOE_Q8_0_F16-Imatrix.gguf - Designed for maximum quality preservation, this layout implements a strict 3-Tier Precision Matrix:
- Tier 1 (Core & Mamba Gating - F16 Precision):
token_embd.weight,output.weight- Protects the critical input/output vocabulary mappings. Adds~1GBto the file size but dramatically prevents text degradation.ssm_alpha,ssm_beta- Protects the integrity of the Mamba state-space calculations across long-range context tokens.
- Tier 2 (Backbone & Shared - Q8_0 Precision):
ssm_out,*._shexp- Keeps the attention mechanics, and all trailing shared experts at high quality, to protect the logical research loops. - Tier 3 (Routed Experts - MXFP4 Precision):
ffn_down_exps,ffn_gate_exps,ffn_up_exps- Shrink the massive background expert parameters directly toMXFP4.
2. The Dual-Quant Hybrid Flavor (MXFP4 + Q8_0)
Qwable-v1-35B-A3B-MXFP4_MOE_Q8_0-Imatrix.gguf - Designed for a slightly leaner memory profile, this layout utilizes 2-Tier Precision:
- Tier 1 (Backbone - Q8_0 Precision): All attention blocks, Mamba structures, vocabulary embeddings, and internal routers use the universal
Q8_0format. - Tier 2 (Experts - MXFP4 Precision): The heavy sparse expert blocks are target-quantized directly to
MXFP4.
๐ Exact Conversion Details
Because Qwable-v1 utilizes trailing shared-expert vectors at the boundary edge of its alternating architecture (Layer 40 boundary), standard conversion requires boundary bypass instructions using .*_shexp\.weight=Q8_0. These files were converted via llama-quantize utilizing the following manual recipe parameters:
Convert SafeTensors to GGUF:
# Requires python3.12, with `pip install --upgrade transformers`
python convert_hf_to_gguf.py "Qwable-v1/" --outtype f16 --outfile "Qwable-v1_F16.gguf"
Generate Tri-Quant MXFP4_MOE + Q8_0 + F16:
llama-quantize \
--tensor-type ".*_shexp\.weight=Q8_0" \
--tensor-type "token_embd\.weight=F16" \
--tensor-type "^output\.weight=F16" \
--tensor-type "blk\..*\.(ssm_alpha|ssm_beta)\.weight=F16" \
--tensor-type "blk\..*\.(ffn_down_exps|ffn_gate_exps|ffn_up_exps)\.weight=MXFP4" \
--imatrix "bartowski-lordx64_Qwable-v1-imatrix.gguf" \
"Qwable-v1_F16.gguf" \
"Qwable-v1-35B-A3B-MXFP4_MOE_Q8_0_F16-Imatrix.gguf" \
Q8_0
Generate Dual-Quant MXFP4_MOE + Q8_0:
llama-quantize \
--tensor-type ".*_shexp\.weight=Q8_0" \
--tensor-type "blk\..*\.(ffn_down_exps|ffn_gate_exps|ffn_up_exps)\.weight=MXFP4" \
--imatrix "bartowski-lordx64_Qwable-v1-imatrix.gguf" \
"Qwable-v1_F16.gguf" \
"Qwable-v1-35B-A3B-MXFP4_MOE_Q8_0-Imatrix.gguf" \
Q8_0
๐ Local Deployment & llama-server Configuration (config.ini)
To maintain the rock-solid reasoning loop depth of the Fable+Opus distillation and prevent agents from falling into repetitive tool-calling deadlocks, use the following server parameter recommendations.
# --- Samplers (Dynamic & Expressive) ---
temperature = 0.65
top-k = 40
top-p = 0.90
min-p = 0.08
# --- Penalties (Prevent Syntax & Reasoner Corruption) ---
repeat-penalty = 1.00
presence-penalty = 0.00
# --- DRY Sampler (Protects Indentation & Structural Boilerplate) ---
dry-multiplier = 0.8
dry-base = 1.75
dry-allowed-length = 8
dry-penalty-last-n = 1024
dry-sequence-breaker = ["\n", ":", " ", "\t", "\"", ","]
# --- Enforced Execution Graph ---
samplers = temp;top_k;top_p;min_p;dry
Keep reasoning off for this model.
reasoning = off
reasoning-budget = 0
reasoning-format = none
This works well with 256k context window.
Note: If using a larger context window, ensure you use the updated MXFP4_MOE_Q8_0_F16 flavor (updated 2026-06-27) to maintain pristine Mamba state-space precision and lessen looping at high context.
โน๏ธ Misc Details
I'm doing this as a side hobby, with my AMD 5900X, 64GB DDR4, RTX 3060 12GB & RTX 5060 Ti 16GB.
In addition to the above configuration, I also use:
slots = 1
parallel = 1
no-warmup = true
flash-attn = on
mlock = false
no-mmap = false
no-context-shift = true
batch-size = 2048
ubatch-size = 256
fit = on
fit-target = 768
main-gpu = 0
split-mode = layer
n-gpu-layers = 999
n-cpu-moe = 0
tensor-split = 16,12
override-tensor = (token_embd)=CUDA0,(vision|vpm|nextn)=CPU
cache-type-k = q8_0
cache-type-v = q8_0
jinja = true
chat-template = jinja
chat-template-file = chat_template.jinja
For further quality and better ssm behaviour, this configuration can help:
context-shift = false
cache-type-k = f16
cache-type-v = f16
๐ค Support the Journey
As a passionate developer, I'm always programming, automating, or experimenting with new ideas.
I love building open-source tools, trying out new web tech, and creating things that don't yet exist, including local AI & quantizing models.
I love sharing these creations to give back to the community.
If my projects have saved you time or helped you out, consider supporting my work below!
๐ Support me on Ko-fi
โจ Acknowledgments
- lordx64 for the exceptional
Qwable-v1base model. - bartowski for the high-quality importance matrix data anchors.
๐ License
See lordx64/Qwable-v1#license--terms.
๐ Citation
@misc{lordx64_qwable_v1_2026,
title = {Qwable-v1: Agentic coding distillation from Claude Fable-5 onto Qwen3.6-35B-A3B},
author = {lordx64},
year = {2026},
howpublished = {\url{https://huggingface.co/lordx64/Qwable-v1}},
}
- Downloads last month
- 309
4-bit
16-bit
Model tree for jashepp/Qwable-v1-35B-A3B-MXFP4_MOE_Hybrid-Imatrix-GGUF
Base model
Qwen/Qwen3.6-35B-A3B