siddhu1983/Tev-27B-Q4_K_M-GGUF

Tev-27B Q4_K_M GGUF

🌐 Project Page  |  📄 Paper (arXiv:2610.18868)  |  🤗 Original Model (gordonhu/Tev-27B)

This repository provides an independently verified Q4_K_M GGUF quantization of gordonhu/Tev-27B, the frontier open-weight long-horizon terminal agent model.

---

Why This Quantization?

The original gordonhu/Tev-27B checkpoint is distributed in BF16 safetensors format totaling 50.1 GB across 12 shards. Running the unquantized weights requires ~58.8 GB of VRAM (weights + KV cache + buffers), making it inaccessible for single-GPU consumer workstations and causing severe PCIe memory thrashing (0.6–1.4 tokens/s) when offloaded over CPU RAM.

This Q4_K_M GGUF release:

• Reduces size by 69.2% to 15.41 GiB (16.55 GB).
• Enables 100% GPU offload on a single 24 GB GPU (e.g., NVIDIA GeForce RTX 3090 / RTX 4090).
• Delivers ~32–36 tokens/second generation throughput and ~518.7 tokens/second prompt evaluation.
• Preserves 6-bit quantization on critical attention and feed-forward projections (Q6_K), maintaining strong reasoning fidelity and zero-shot terminal-bench capability.

---

Model Specifications

Attribute	Specification
Model Architecture	qwen35 (Transformer with SwiGLU, RoPE $\theta=10^7$, 65 layers)
Total Parameters	26,895,998,464 (~26.9B parameters)
Quantization Scheme	Q4_K_M (k-quants medium, LLAMA_FTYPE_MOSTLY_Q4_K_M)
Tensor Distribution	433 Q4_K, 65 Q6_K, 353 F32 (851 tensors total)
File Name	tev-27b-q4_k_m.gguf
File Size	16,547,400,704 bytes (15.41 GiB / 16.55 GB)
Cryptographic SHA-256	b09b4808a10d9f32443069d09ab642d2bba8eb39915b8b2bff50114e545cc7db
Declared Context Length	262,144 tokens (Recommended: 8,192–32,768 on 24 GB GPUs)
Embedding Dimension	5,120 (Feed-forward: 17,408; Attention Heads: 24 Q, 4 KV)
Tokenizer	GPT-2 BPE (qwen35 pre-tokenizer, vocabulary 248,320)
Special Tokens	BOS/Pad: 248044 (`<
Supported Modalities	Language & Code Only (Vision encoder & MTP heads excluded upstream)
License	CC BY-NC 4.0 (Non-Commercial Research Use Only)

---

Training Origin & Tev Architecture

Tev-27B was developed by Wenbo Hu, Shraman Pramanick, Krishna Kumar Singh, Nanyun Peng, Kai-Wei Chang, and Yong Jae Lee in the research paper Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards (arXiv:2610.18868, Oct 2026).

It is trained from Qwen/Qwen3.8-27B using reinforcement learning (DPPO) on Tev-Gym, introducing two key innovations:

1. Latent-World-First Environment Synthesis: 1,170 verified environments across 7 domains where task workflows and expected states are formally modeled as DAGs.
2. Progress-Potential Reward Shaping (PPRS): Dense per-turn intermediate progress rewards $\Phi(s) \in [0, 1]$ preventing zero-advantage gradient collapse on multi-step rollouts.

---

Quantization Provenance

This quantization was generated locally from the official upstream safetensors weights:

• Upstream Source: gordonhu/Tev-27B (Commit 60c36fd4a8db1244856ef6a5abc0b4f1a121978b).
• Conversion Tool: convert_hf_to_gguf.py (llama.cpp commit 99f3dbd0b21d1401b3769e0c0d963913607f380b).
• Intermediate Stage: Converted to native BF16 GGUF (tev-27b-bf16.gguf, 50.1 GB), verified, then deleted.
• Quantization Binary: llama-quantize.exe (llama.cpp build 11139, commit 42916d83f, Clang 20.1.8 x86_64).
• Quantization Command: llama-quantize.exe tev-27b-bf16.gguf tev-27b-q4_k_m.gguf Q4_K_M 16.
• Full Audit Trail: Detailed in PROVENANCE.md.

---

Hardware Requirements & Local Performance

Empirical Measurements on NVIDIA GeForce RTX 3090 (24 GB VRAM)

Measurements recorded locally in Ollama under Windows 11 with 100% GPU offload (num_gpu 99, num_ctx 8192):

Evaluation Task	Prompt Length	Completion Length	Generation Speed	Prompt Speed	Wall-Clock Time
Strict JSON Schema (T1)	120 tok	111 tok	31.03 tok/s	432.61 tok/s	3.58 s
Keyword Instruction (T2)*	83 tok	32 tok*	26.38 tok/s	472.10 tok/s	1.21 s
Probability Reasoning (T3)	115 tok	189 tok	33.84 tok/s	583.32 tok/s	5.59 s
Algorithm Implementation (T4)	91 tok	600 tok	35.16 tok/s	527.20 tok/s	17.06 s
Architecture Analysis (T5)	100 tok	500 tok	35.75 tok/s	578.20 tok/s	13.99 s
Reported Arithmetic Mean	—	—	32.43 tok/s	518.69 tok/s	—

*Token breakdown on T2: The 32 total completion tokens consist of ~30 internal chain-of-thought tokens within the <think> channel, followed by 2 visible tokens of final output (KKD_READY).

• Observed Peak VRAM: 19.82 GB peak allocation measured via nvidia-smi during the active Ollama session on the 24,576 MiB card.
• Overall Completion Throughput: 34.56 tok/s (1,432 completion tokens / 41.43s total generation time across all 5 tests).
• Detailed methodology, limitations, and community protocols are documented in BENCHMARKS.md.

---

Quickstart & Installation

Detailed instructions for Ollama, llama.cpp, and Python are provided in USAGE.md.

Quickstart with Ollama

1. Download the GGUF:

   huggingface-cli download <REPO_ID> tev-27b-q4_k_m.gguf --local-dir .
   

2. Create a Modelfile:

   FROM ./tev-27b-q4_k_m.gguf
   PARAMETER num_gpu 99
   PARAMETER num_ctx 8192
   PARAMETER stop "<|im_end|>"
   PARAMETER stop "<|endoftext|>"
   

3. Register and run:

   ollama create tev-27b:q4_k_m -f Modelfile
   ollama run tev-27b:q4_k_m
   

Quickstart with llama.cpp

./llama-cli -m tev-27b-q4_k_m.gguf -ngl 99 -c 8192 --temp 0.7 -p "You are Tev, a helpful terminal agent."

---

Prompt Formatting & Tool Calling

Thinking Channels

Tev-27B supports internal reasoning chains. Thoughts are formatted inside <think>...</think> tags:

<|im_start|>assistant
<think>
Let's analyze the repository structure and test cases...
</think>

Here is the solution to your issue...<|im_end|>

Tool Calling Syntax

The model was trained in a single bash sandbox environment. Define tools in the system message inside <tools> tags. The model emits XML tool calls:

<tool_call>
<function=bash>
<parameter=command>
pytest tests/ -v
</parameter>
</function>
</tool_call>

Feed results back to the model wrapped in <tool_response> tags.

---

Intended Use & Limitations

• Non-Commercial Research: Distributed exclusively under CC BY-NC 4.0. Commercial use is strictly prohibited.
• Run in a Secure Sandbox: As an autonomous terminal agent model, Tev-27B generates and executes shell commands. Always execute model-generated actions inside an isolated container with no access to sensitive credentials, host filesystems, or production networks.
• Single-Turn vs Long-Horizon: Initial benchmark throughput demonstrates fast token generation, but short prompt tests do not establish autonomous multi-step agent reliability.

---

Attribution & Citation

If you use Tev-27B in your research, please cite the original authors:

@misc{hu2026tev,
  title  = {{Tev}: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards},
  author = {Hu, Wenbo and Pramanick, Shraman and Singh, Krishna Kumar and
            Peng, Nanyun and Chang, Kai-Wei and Lee, Yong Jae},
  year   = {2026},
  eprint = {2610.18868},
  archivePrefix = {arXiv}
}

Base model acknowledgment: Qwen/Qwen3.8-27B licensed under Apache 2.0. Full notices in ATTRIBUTION.md.