Luni/Swift1.5-Qwen3.8-Flash-Next-EXL3-3.05bpw

Swift1.5-Qwen3.8-Flash-Next - EXL3 3.05bpw (h5_ng5)

This is an ExLlamaV3 (EXL3) 3.05 bpw quantization of ukisai/Swift1.5-Qwen3.8-Flash-Next, the community-consensus overall champion of the Qwen3.8-Flash-Next ecosystem.

Key Architectural & Quantization Specifications

• Format: Native ExLlamaV3 SafeTensors shards (7 shards + 5-bit N-gram table).
• Target Bitrate: 3.05 bpw (routed MoE experts).
• Effective Bitrate: 3.08 bpw (decoder overall), 6.00 bpw (LM head).
• Quantization Flags: -b 3.05 -hq -ngb 5 -mb 4 -vb 6
• Total Storage Size: 80.2 GB (49.6 GB model weights + 30.4 GB 5-bit N-gram table).
• Hardware Profile: Ideal for 24-32 GB GPUs (e.g., RTX 5090 / 4090 / 3090) paired with 64 GB DDR5 host RAM.

Quantization Quality & Noise Telemetry

Signal-to-Quantization-Noise Ratio (SQNR) and error metrics measured against unquantized BF16 reference states:

Network Region	Layers	Measured SQNR	Angular Drift (Cosine Distance)	Frobenius Error (RFN)
Early Trunk	Layers 0 - 8	38.80 dB - 44.50 dB	0.00015 - 0.00045	0.0165 - 0.0295
Core MoE Trunk	Layers 9 - 26	39.50 dB - 45.20 dB	0.00025 - 0.00062	0.0180 - 0.0270
Deep Trunk	Layers 27 - 37	42.10 dB - 48.90 dB	0.00028 - 0.00048	0.0210 - 0.0340
Final Decoder	Layers 38 - 47	37.60 dB - 42.80 dB	0.00012 - 0.00055	0.0065 - 0.0290
Output Head	lm_head	41.50 dB (6.00 bpw)	0.00005	0.0088

(Note: In signal analysis, +6 dB represents a 50% reduction in noise power. An SQNR exceeding 38-48 dB preserves reasoning coherence and structured tool calling without token degradation.)

Asymmetric Precision Allocations (-hq & -ngb 5)

• MoE Expert Matrices: 3.05 bpw across 512 experts (10 active per token).
• GDN Linear Attention: Locked at 5.00 bpw (in_proj_qkv, in_proj_z, out_proj).
• QSA Sparse Attention Indexers: Locked at 5.00 bpw (q_proj, k_proj, v_proj, o_proj, index_qk_proj).
• Shared Experts: Locked at 5.00 bpw (down_proj, gate_proj, up_proj).
• 51.2B N-gram Table: Locked at 5-bit precision (-ngb 5, 30.4 GB).
• MTP Speculative Draft Head: Retained at 4.00 bpw (-mb 4).
• Vision Tower: Retained at 6-bit precision (-vb 6).
• LM Head: Locked at 6.00 bpw.

Runtime Memory Requirements

• VRAM: ~24 to 26 GiB active allocation (fits RTX 5090 / 4090 / 3090).
• System RAM: ~47.8 GiB working set (fits 64 GB DDR5).
• Storage: ~82 GB NVMe storage (mmap supported for N-gram lookup).

Inference with TabbyAPI (RTX 5090 32GB + 64GB DDR5)

model_name: Swift1.5-Qwen3.8-Flash-Next-EXL3-3.05bpw
model_dir: /path/to/model
backend: exllamav3
gpu_split: [26]
ngram_ram: false
cpu_moe_split_experts: 300
cpu_moe_pinned_arena: true
cpu_moe_threads: 12
draft_model:
  draft_mode: mtp
  draft_num_tokens: 3
  dynamic_draft: true
  hybrid_draft_tokens: 5