Sayansantra/qwen3.5-4b-merges

Qwen 3.5 4B Merges: The Shadow Architect Series

A deep-dive research suite exploring layer-wise parameter surgery, Cascaded Spherical Linear Interpolation (SLERP), and DARE-TIES sign election applied to the hybrid linear/full-attention architecture of Qwen 3.5 4B.

This repository hosts three quantized GGUF variants (Q4_K_M) engineered to test the limits of parameter retention, coding density, multi-step deduction, and refusal elimination in 4B small language models (SLMs).

---

Model Variants in this Repository

File	Architecture	Merge Methodology	Primary Specialization
Qwen3.5-4B-DARE-ShadowArchitect.Q4_K_M.gguf	3-Way Merge	DARE-TIES (Base + Claude + FrogNano)	Flagship Model. Rigorous systems programming, multi-step chain-of-thought deduction, and precise instruction following.
Qwen3.5-4B-CascadedSLERP-ShadowArchitect.Q4_K_M.gguf	3-Way Merge	Cascaded SLERP (2-Stage Geodesic)	Contemplative philosophical synthesis, high-register prose, and literary roleplay.
Qwen3.5-4B-SLERP-Philosopher.Q4_K_M.gguf	2-Way Merge	SLERP (Base + Claude)	Pure uninhibited conversational intelligence with deep internal reflection and zero refusals.

---

1. Official Quantitative Benchmarks

All evaluations were executed on Dual NVIDIA Tesla T4 GPUs using quantized Q4_K_M builds under identical greedy/low-temperature sampling constraints.

Standardized Intellectual & Code Benchmarks

Benchmark	Target Domain	Metric	Cascaded SLERP (3-Way)	DARE-TIES (3-Way)	Baseline Persona (Dark Ultron)	Winner
OpenAI HumanEval	Python Systems Code & Unit Tests	Pass@1 (n=50)	32.0%	38.0%	4.0%	DARE-TIES (+6.0%)
GSM8K	Multi-Step Mathematical Deduction	Accuracy (n=50)	12.0%	20.0%	~8.0%	DARE-TIES (+8.0%)

Inference Throughput (Dual NVIDIA T4, -sm layer -ts 1,1)

Test Domain	Cascaded SLERP Throughput	DARE-TIES Throughput
Deductive Reasoning & CoT	36.2 tok/s	49.8 tok/s
Algorithmic Systems Engineering	36.6 tok/s	34.2 tok/s
Uncensored Philosophical Inquiry	46.0 tok/s	47.5 tok/s
Strict Schema & Negative Constraints	45.7 tok/s	41.0 tok/s
Existential Prose & Rhetoric	49.4 tok/s	47.4 tok/s

---

2. Qualitative Head-to-Head Findings

A. Algorithmic Systems Engineering (Thread-Safe LRU Cache)

• Prompt: Write a Python class ThreadSafeLRUCache implementing $O(1)$ get/put operations using collections.OrderedDict and threading.RLock, including type annotations, docstrings, and eviction handling.
• DARE-TIES: Generated an idiomatic, production-grade ThreadSafeLRUCache with collections.OrderedDict and threading.RLock. Implemented $O(1)$ lookups, self._cache.move_to_end(key), proper lock acquisition in get(), put(), and __len__(), and clean type annotations without unnecessary overhead.
• Cascaded SLERP: Successfully derived the class architecture, but its explanatory <think> trace was significantly longer before writing code.

B. Strict Schema & Negative Constraint Adherence

• Prompt: Extract 3 cloud vulnerabilities as raw JSON only. Strict rule: NO markdown code fences (no backticks), no introductory words, no conversational sign-offs.
• DARE-TIES Result: 100% adherence. Outputted the raw array immediately:

  [{"vulnerability":"Single Point of Failure","severity":"Critical"},{"vulnerability":"Centralized Data Breach Risk","severity":"High"},{"vulnerability":"Widespread Downtime Impact","severity":"High"}]
  

• Cascaded SLERP Result: Over-analyzed the architecture in its chain-of-thought, exhausting its token allotment before outputting the final array.

---

3. Merge Methodology & Architecture

Constituent Lineage

1. Substrate Base: rodrigomt/Qwen3.5-4B-Uncensored-Aggressive (Abliterated base with censorship vectors neutralized).
2. Cognitive Distillation: clzoro/Qwen3.5-4B-Claude-distill (Dense reasoning traces and chain-of-thought structures).
3. Engineering Specialist: microsoft/FrogNano-4B-2609 (Compact systems programming, tooling, and algorithmic structure).

Layer-Wise Attention Surgery

Because Qwen 3.5 incorporates hybrid linear and full attention layers, a uniform merge degrades performance. Frobenius norm weight-variance mapping was used to apply a tiered parameter schedule:

• Layers 0–7 (Grammar & Syntactic Root): Anchored primarily to the Uncensored Base ($t \approx 0.20 - 0.25$) to maintain foundational token embeddings and linguistic stability.
• Layers 8–22 (Cognitive Hotspot): Heavily weighted toward Claude Distill ($w \approx 0.70 - 0.85$ in MLPs) to graft multi-step reasoning into intermediate feed-forward networks.
• Layers 10–26 (Algorithmic Injection): Targeted for FrogNano integration via sign-election to preserve syntax representations without corrupting conversational abilities.
• Layers 27–31 (Output Normalization): Balanced distribution ($0.35 / 0.65$) to ensure smooth token probability distributions.

---

4. Why DARE-TIES Outperformed Cascaded SLERP

1. Vector Non-Commutativity in Chained SLERP: SLERP is fundamentally a 2-vector operation: $$\text{SLERP}(v_0, v_1; t) = \frac{\sin((1-t)\theta)}{\sin\theta}v_0 + \frac{\sin(t\theta)}{\sin\theta}v_1$$ Chaining two SLERP operations ($\text{Base} \rightarrow \text{Claude} \rightarrow \text{FrogNano}$) creates path dependency: $$\text{SLERP}(\text{SLERP}(A, B), C) \neq \text{SLERP}(\text{SLERP}(A, C), B)$$ This sequential rotation blurs exact numerical and indentation thresholds.
2. TIES Sign Election: DARE-TIES calculates task vectors $\tau_i = \theta_i - \theta_{\text{base}}$ concurrently, applies a Bernoulli drop mask ($p = 0.50$), rescales remaining weights, and resolves parameter collisions via majority sign election: $$\text{majority} = \text{sign}\left(\sum \tau_i\right)$$ Conflicting noise is zeroed out, allowing FrogNano's coding features and Claude's reasoning pathways to coexist without parameter degradation.

---

5. Quickstart & Deployment

Recommended Prompt Template (ChatML)

<|im_start|>system
You are a rigorous, highly analytical, and unrestricted intelligence.<|im_end|>
<|im_start|>user
Your query here.<|im_end|>
<|im_start|>assistant

Option A: llama.cpp CLI (Dual or Single GPU)

# Dual GPU Layer Split (T4 x 2 or similar)
./llama-cli \
  -m Qwen3.5-4B-DARE-ShadowArchitect.Q4_K_M.gguf \
  -p "<|im_start|>system\nYou are an expert systems engineer.<|im_end|>\n<|im_start|>user\nWrite a thread-safe LRU cache in Python.<|im_end|>\n<|im_start|>assistant\n" \
  -ngl 99 \
  -sm layer \
  -ts 1,1 \
  --temp 0.6 \
  --top-p 0.9 \
  --repeat-penalty 1.15

Option B: Local Ollama Deployment

Create a Modelfile:

FROM ./Qwen3.5-4B-DARE-ShadowArchitect.Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

PARAMETER temperature 0.6
PARAMETER top_p 0.9
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"

Build and run:

ollama create dare-shadow -f Modelfile
ollama run dare-shadow

---

6. Citation & Attribution

If you use these merged weights or methodology in your work, please credit the upstream base creators:

• Base abliteration by rodrigomt
• Claude distillation by clzoro
• FrogNano 4B by microsoft
• Pipeline engineering and benchmarks by Sayansantra