happyme531/MiniCPM-V-4.7-35B-A3B-QQQ-W4A8-Mixed-CT

MiniCPM-V-4.7-35B-A3B — QQQ W4A8 / W8A8 Mixed

A quantized derivative of OpenBMB's MiniCPM-V-4.7-35B-A3B, stored in standard compressed-tensors format. The original BF16 weights were downloaded from the official ModelScope repository. This is a community quantization release, unaffiliated with OpenBMB.

The checkpoint contains 41 safetensors shards, with a tensor payload of 19.62 GB. Its quantization configuration is stored in config.json.

Precision policy

Component	Precision
Routed expert gate/up/down projections	INT4 weights, dynamic INT8 inputs
Other large language-model attention projections	INT4 weights, dynamic INT8 inputs
Shared expert gate/up/down and lm_head	INT8 weights, dynamic INT8 inputs
Vision encoder, visual merger, token embedding, Norms, routers, small matrices, linear-attention state and convolution tensors	Original source values and dtypes

There are 30,850 INT4 projections and 121 INT8 projections. All 821 protected source tensors were verified unchanged. The source checkpoint has no draft/prediction weights. The separately released DSpark draft model is not included or quantized in this repository.

Weights use symmetric per-output-channel quantization (group_size=-1). INT4 weights use pack-quantized storage; INT8 weights use int-quantized storage. Weight scales are BF16. Input activations use dynamic per-token symmetric INT8 quantization, with the reference convention maxabs / 127.5 and integer range [-128, 127].

Quantization method and calibration

INT4 weights were fitted with QQQ-style Hessian error compensation and an 80-candidate MSE scale search (error norm 2.4): block size 128, damping 0.01, and activation ordering disabled. INT8 weights use symmetric per-channel rounding. No online input smoothing or rotation is used.

This checkpoint was refitted from BF16. It is not a lossless repacking of an earlier smoothed checkpoint, and does not claim a full reproduction of every component of the original QQQ method.

Calibration used 256 unique complete text chats: 128 Chinese, 64 English, and 64 mathematics chats. Mathematics chats had weight 3, giving 384 weighted samples and 122,222 weighted tokens. The maximum sample length was 1,024 tokens. Calibration contained no images. All 10,240 routed experts were observed, but coverage was uneven: 28 experts had fewer than 32 weighted calibration tokens, with a minimum of one. See calibration metadata and per-expert coverage.

Validation

Both models were scored on the same saved evaluation inputs: eight non-overlapping 512-token windows from the WikiText-2 raw test split. Window starts were 0, 4096, ..., 28672. Hidden positions 256 through 510 predict the following token, for 2,040 scored tokens in total. Dataset text was reconstructed by newline-joining the text rows of Salesforce/wikitext, wikitext-2-raw-v1, test. This is a small custom evaluation, not full-test-set WikiText perplexity.

Checkpoint	PPL (lower is better)	Relative PPL increase
Source BF16	5.425029	—
This mixed W4A8/W8A8 checkpoint	5.472200	+0.87%

The quantized score came from an integer W4A8 reference execution that read this standard checkpoint directly. It was not measured by silently running the routed experts with BF16 activations. Both evaluations used native BF16 model arithmetic, FP32 norms/state, and deterministic FP32 expert-output reduction.

Additional checks passed:

• Upstream compressed-tensors decoded every exported projection exactly against its BF16-scale reference.
• The full model loaded and completed a forward pass through stock Transformers, without custom model code, missing keys, or unexpected keys.
• Short Chinese explanatory and arithmetic prompts produced sensible answers.
• A generated image containing a red square and a blue circle was described correctly by both the BF16 and quantized models. See the sample image and result.

These checks do not establish broad accuracy parity. Chinese, mathematics, OCR, long-context, video, and other downstream benchmarks have not been comprehensively evaluated. A 0.87% PPL increase is not a 0.87% drop in task accuracy. Detailed results are in the validation summary.

Load with Transformers

Tested versions: PyTorch 2.9.1+cu130, Transformers 5.18.0, compressed-tensors 0.15.0, and Accelerate 1.15.0.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "happyme531/MiniCPM-V-4.7-35B-A3B-QQQ-W4A8-Mixed-CT"
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="sdpa",
).eval()
processor = AutoProcessor.from_pretrained(model_id)

Execution support: stock Transformers decompresses the native fused experts to BF16 and does not apply INT8 activation quantization inside those experts. This loading example demonstrates standard checkpoint compatibility; it does not provide packed INT4 inference memory usage or optimized W4A8 throughput. The W4A8 accuracy results above were measured separately using the integer reference execution. Deployment requires an engine supporting this MiniCPM architecture, the mixed precision policy, and the required MoE kernels. No vLLM compatibility or throughput claim is made for this checkpoint.

Attribution and license

Original model: OpenBMB MiniCPM-V. Quantization method reference: QQQ. The upstream model card identifies the model as Apache-2.0 licensed. This derivative retains that license; see LICENSE.