---
license: apache-2.0
base_model: empero-ai/Qwythos-27B-v1
base_model_relation: quantized
language:
- en
library_name: gguf
pipeline_tag: image-text-to-text
tags:
- gguf
- llama.cpp
- quantized
- qwen3.5
- reasoning
- uncensored
- long-context
- 1M-context
- multimodal
- vision
- mtp
- agentic
---


<p align="center">
  <img src="qwythos.png" alt="Qwythos-27B-v1" width="640"/>
</p> 

# Qwythos-27B-v1-GGUF

**Developed by [Empero](https://empero.org)**

GGUF quantizations of **[empero-ai/Qwythos-27B-v1](https://huggingface.co/empero-ai/Qwythos-27B-v1)** for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes. Qwythos-27B-v1 is a dense, hybrid Qwen3.5-27B—the larger continuation of Qwythos-9B, trained on the same curriculum at a larger scale. Nothing was removed to make this conversion fit: its native MTP head, vision tower, and 1,048,576-token (1M) context capability are intact. This is the pre-RL v1 checkpoint, following SFT, DPO, and ESFT; an RL-trained v2 will follow, while v1 is a complete checkpoint in its own right.

For the full capability, training, and evaluation writeup, see the **[base model card](https://huggingface.co/empero-ai/Qwythos-27B-v1)**. This card is intentionally focused on choosing a file and running it.

---

## Files

The repository is named `Qwythos-27B-v1-GGUF`, but the files on disk deliberately use the shorter `Qwythos-27B-*` names below. Use those exact filenames when downloading or passing `-m`.

### Normal text weights — trunk only (64 blocks)

| File | Quant | Size | Notes |
|---|---|---:|---|
| `Qwythos-27B-Q4_K_M.gguf` | Q4_K_M | 15.78 GiB / 16.95 GB | **recommended default** |
| `Qwythos-27B-Q5_K_M.gguf` | Q5_K_M | 18.11 GiB / 19.44 GB | K-quant |
| `Qwythos-27B-Q6_K.gguf` | Q6_K | 20.91 GiB / 22.45 GB | K-quant |
| `Qwythos-27B-Q8_0.gguf` | Q8_0 | 26.63 GiB / 28.60 GB | 8-bit quant |
| `Qwythos-27B-BF16.gguf` | BF16 | 50.11 GiB / 53.81 GB | full-precision conversion base |

If you do not know which to pick, start with **Q4_K_M**.

### MTP-enabled text weights (`nextn_predict_layers = 1`)

These include the native MTP head inside the GGUF. Use them with a recent llama.cpp build that supports MTP draft speculation, for example `--spec-type draft-mtp`.

| File | Quant | Size | Notes |
|---|---|---:|---|
| `Qwythos-27B-MTP-Q4_K_M.gguf` | Q4_K_M + MTP | 16.05 GiB / 17.24 GB | **recommended MTP default** |
| `Qwythos-27B-MTP-Q5_K_M.gguf` | Q5_K_M + MTP | 18.41 GiB / 19.76 GB | MTP K-quant |
| `Qwythos-27B-MTP-Q6_K.gguf` | Q6_K + MTP | 21.25 GiB / 22.82 GB | MTP K-quant |
| `Qwythos-27B-MTP-Q8_0.gguf` | Q8_0 + MTP | 27.05 GiB / 29.05 GB | MTP 8-bit quant |
| `Qwythos-27B-MTP-BF16.gguf` | BF16 + MTP | 50.90 GiB / 54.66 GB | MTP, full-precision conversion base |

### Vision projector — for image input

| File | Size | Notes |
|---|---:|---|
| `mmproj-Qwythos-27B-F16.gguf` | 0.86 GiB / 0.93 GB | vision encoder + projector at F16; **required for images**, pairs with any normal or MTP text quant |

The vision tower is intact in this release. Download the `mmproj` alongside a text quant whenever you want image input.

### What fits on a GPU?

These are practical **weight-size-based estimates**, not measured performance or VRAM benchmarks. They assume a modest context and leave room for runtime overhead and the KV cache; image input adds the 0.86 GiB `mmproj`. Longer contexts—especially the full 1M window—make the KV cache the dominant memory cost and may require CPU/KV offload or multiple GPUs regardless of weight quant.

**Measured:** `Qwythos-27B-Q4_K_M.gguf` fully offloaded to a single **RTX 5090 (32 GB)** at 4k context runs at **~74 tok/s generation, ~622 tok/s prompt** (llama.cpp CUDA build, `-ngl 99`). One data point on one card, not a benchmark sweep — but it confirms a 27B at Q4_K_M is comfortably interactive on a single consumer GPU.

| Quant | Approximate hardware guidance at modest context |
|---|---|
| Q4_K_M / MTP-Q4_K_M | The practical 24 GB-card starting point. Q4_K_M has the most headroom of these files for runtime and KV cache. |
| Q5_K_M / MTP-Q5_K_M | Can fit tightly on 24 GB at a modest context; 48 GB is the more comfortable single-GPU target. |
| Q6_K / MTP-Q6_K | A 48 GB card is the practical single-GPU target; on 24 GB, expect weight or KV offload. |
| Q8_0 / MTP-Q8_0 | Plan on 48 GB for modest-context single-GPU use; 24 GB requires offload. |
| BF16 / MTP-BF16 | The weights alone exceed 48 GB. Use an 80 GB-class GPU or multi-GPU/offload, with additional memory for runtime and KV cache. |

---

## Quick start

### llama.cpp (`llama-cli`)

```bash
llama-cli \
  -m Qwythos-27B-Q4_K_M.gguf \
  -p "Explain the design tradeoffs in a Gated-DeltaNet hybrid model." \
  -n 16384 \
  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  -c 16384
```

### llama.cpp (`llama-server`)

```bash
llama-server \
  -m Qwythos-27B-Q4_K_M.gguf \
  -c 16384 --port 8080
```

### Ollama

```bash
ollama run hf.co/empero-ai/Qwythos-27B-v1-GGUF:Q4_K_M
```

### LM Studio / jan / KoboldCpp

Download any text `.gguf` from this repository and load it in your runtime. Preserve the embedded Qwythos chat template if the runtime asks you to select or configure one.

### llama.cpp with MTP draft speculation

```bash
llama-server \
  -m Qwythos-27B-MTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  -c 16384 --port 8080
```

MTP support requires a recent llama.cpp build. If your runtime does not support MTP draft speculation, choose a trunk-only file instead.

---

## Vision (image input)

Download a text quant and `mmproj-Qwythos-27B-F16.gguf`, then use llama.cpp's multimodal CLI or server.

### llama.cpp (`llama-mtmd-cli`)

```bash
llama-mtmd-cli \
  -m Qwythos-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwythos-27B-F16.gguf \
  --image ./photo.jpg \
  -p "Describe this image in detail." \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -c 16384
```

### llama.cpp server

```bash
llama-server \
  -m Qwythos-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwythos-27B-F16.gguf \
  -c 16384 --port 8080
```

---

## Hybrid-precision quantization (Gated-DeltaNet / SSM tensors)

Qwythos is a hybrid model: three Gated-DeltaNet linear-attention layers for every full-attention layer. Its Gated-DeltaNet state tensors are disproportionately sensitive to low-bit quantization, so these K-quants retain them at higher precision than the surrounding weights:

| Tensor | Q4_K_M | Q5_K_M | Q6_K |
|---|---|---|---|
| `ssm_alpha` | Q8_0 | Q8_0 | Q8_0 |
| `ssm_beta` | Q8_0 | Q8_0 | Q8_0 |
| `ssm_out` | Q8_0 | Q8_0 | Q8_0 |
| `ssm_a`, `ssm_conv1d`, `ssm_dt`, `ssm_norm` | F32 | F32 | F32 |

**The entire Gated-DeltaNet state path is held at Q8_0 or better in every K-quant** — no part of it is allowed to drop to Q4/Q5/Q6, regardless of the nominal quant level. This costs roughly 2–4% in file size over a flat K-quant and is the single most important difference between these files and a naive conversion of a hybrid model. Q8_0 and BF16 are uniform and need no overrides. MTP variants additionally pin `nextn.eh_proj` to Q8_0.

Built with llama.cpp pinned at commit `d0f9d2e5`, using architecture `qwen35`.

---

## Identity and chat template

The embedded chat template identifies the model as: “You are Qwythos, a model created by Empero AI. Only bring up your identity if the user asks.” Use the runtime's chat/completions mode rather than hand-rolling a different prompt format where possible.

## Function calling

The embedded chat template carries Qwen3.5's full tool-use spec, so tool calling works through `llama-server`'s OpenAI-compatible endpoint without a wrapper — pass `tools` in your `/v1/chat/completions` request and parse the emitted `<tool_call>` blocks:

```
<tool_call>
<function=NAME>
  <parameter=PARAM>value</parameter>
</function>
</tool_call>
```

Given the terminal/agentic training behind this checkpoint, tool-augmented use is the intended deployment path.

## Sampling recommendations

Qwythos is a reasoning model — responses open with a `<think>…</think>` block before the final answer. These match the shipped `generation_config.json` defaults:

| Parameter | Recommendation |
|---|---|
| `temperature` | 0.6 for agentic, harness, and tool-use work; 1.0 for open-ended reasoning or creative work |
| `top_p` | 0.95 |
| `top_k` | 20 |
| `repeat_penalty` | 1.05 |
| `max_new_tokens` | 16,384 or more; reserve a generous budget for long reasoning traces and the final answer |

As llama.cpp flags:

```bash
--temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 -n 16384
```

Use `repeat_penalty 1.05` rather than Qwen's default of 1.0 — it prevents rare non-terminating reasoning loops on long generations.

## Long context (1M tokens)

The model configuration supports a **1,048,576-token** window: a 4× YaRN extension over the 262,144-token native window. Set the runtime context to the amount you actually need—lower `-c` values substantially reduce KV-cache memory requirements.

The 1M setting is an inference-time capability, not a promise that a single GPU can hold a 1M KV cache. For roughly 512k context, a YaRN factor of 2.0 is preferable; if you will never exceed the native 262k window, remove scaling entirely. Static YaRN scaling can slightly degrade short-context quality. For the full 1M window, plan for multi-GPU or aggressive KV offload.

## Verification

`SHA256SUMS` is included in this repository. Verify a download before loading it:

```bash
sha256sum -c SHA256SUMS
```

---

## Limitations

- **Pre-RL v1.** This release has completed SFT, DPO, and ESFT, but the planned RL-trained v2 is still to come.
- **Context costs memory.** Weight size is only part of the hardware budget; long contexts need substantial KV-cache memory.
- **MTP is runtime-dependent.** Use the trunk-only files when your GGUF runtime does not yet support `draft-mtp` speculation.
- **No standardized academic benchmark suite has been run for this checkpoint.** Validate the model on the tasks and operating conditions that matter to your deployment.
- **Uncensored model.** Add the application-level review and safety controls appropriate to an end-user deployment.

## Stay in the loop

Sign up for the Empero newsletter at **[empero.org](https://empero.org)** for releases, evals, and research notes.

## Support / Donate

If this model helped you, consider supporting the project:

- **BTC**: `bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v`
- **LTC**: `ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x`
- **XMR**: `42Dbm5xg5Nq26fdyzfEU7KBnAJfhi7Cvz5J2ex5CzHXkfKuNEJzYCcmJ1GTbgjFZ5MBx72sdG1G9239Cd6rsZfv4QeDkYJY`

---

## Provenance & licensing

Qwythos-27B-v1 is derived from **Qwen3.5-27B**. The weights are released under **Apache-2.0**, inherited from the Qwen3.5-27B base, and are shared for research and experimentation as-is.

## Acknowledgements

- Developed and released by [Empero](https://empero.org)
- Base model: [Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) (Alibaba Qwen team)
- Training: [TRL](https://github.com/huggingface/trl) + [Transformers](https://github.com/huggingface/transformers)
- GGUF quantization: [llama.cpp](https://github.com/ggml-org/llama.cpp) (ggml-org)