---
license: openmdw-1.1
base_model: poolside/Laguna-S-2.1
tags:
  - laguna
  - moe
  - code
  - colibri
  - int4
---

# Laguna-S-2.1 — colibri int4

An int4 conversion of [poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1)
(118B total / 8B active MoE, coding) into the **colibri container format**, for the
`laguna.c` standalone CPU/CUDA inference engine (part of the
[sabrewing](https://github.com/Schneewolf-Labs/sabrewing) fork of
[colibrì](https://github.com/JustVugg/colibri)).

## What this is

- **Routed experts** (>95% of the weights) are stored as packed **int4** — per-row
  symmetric scales (`.qs`, f32), low-nibble/high-nibble packing, `value = (nibble − 8) · scale`.
  Gate and up projections are fused block-concatenated per expert
  (`experts.gate_up_proj` `[E·2I, D/2]`); down is `experts.down_proj` `[E·D, I/2]`.
- **Residents** (attention q/k/v/o + per-head softplus gate, QK-norms, dense layer-0
  MLP, shared experts, embeddings, lm_head) are **bf16**; norms, the router weight and
  the loss-free `e_score_correction_bias` stay **f32**.
- Total footprint ≈ **63 GB** (vs 235 GB bf16), so the whole model is RAM-resident on a
  128 GB+ machine — no expert-streaming disk wall.

## Faithfulness

Produced by `tools/convert_laguna_int4.py`. The fuse+pack is bit-exact against the int4
round-trip, and the C engine reads the container within quantization noise
(tiny-model teacher-forced perplexity within ~0.13% of the f32 forward, which is itself
token-exact vs the `transformers` reference).

## Architecture (for the curious)

GQA (8 KV heads, head_dim 128) with interleaved 512-token sliding / global attention
(1:3), per-head QK-RMSNorm, a per-head softplus attention output gate, RoPE (YaRN
partial-rotary on global layers, plain full-rotary on sliding), a dense layer 0 +
sigmoid loss-free-routed MoE (256 experts, top-10, + 1 shared expert, routed scaling
2.5). 48 layers, hidden 3072, vocab 100,352.

## Usage

```
git clone https://github.com/Schneewolf-Labs/sabrewing && cd sabrewing/c
make laguna
SNAP=/path/to/Laguna-S-2.1-colibri-int4 ./laguna   # Stage-A oracle / prompt mode
```

Original model © Poolside, licensed under OpenMDW-1.1 — this conversion inherits it.