keystone-systems/Qwen2.5-1.5B-Instruct-onnx-dml-int4

Qwen2.5-1.5B-Instruct, ONNX Runtime GenAI, DirectML, int4

An int4 ONNX Runtime GenAI build of Qwen/Qwen2.5-1.5B-Instruct for the DirectML execution provider. Keystone Systems uses it in Picket, a parental-control product, as an on-device page-review model. It is published here so that the build can be pinned to a commit.

Modifications

This repository holds a modified version of Qwen2.5-1.5B-Instruct (Apache-2.0, section 4(b)): the original weights were converted to ONNX and quantized to int4. Nothing else was changed.

• Base model: Qwen/Qwen2.5-1.5B-Instruct at commit 989aa7980e4cf806f80c7fef2b1adb7bc71aa306.

• Builder: ONNX Runtime GenAI 0.18.0 model builder (CPU), with torch 2.14.1+cpu, transformers 5.20.2, onnx 1.23.2 and onnxruntime 1.31.0.

• Command:

  python -m onnxruntime_genai.models.builder -i <Qwen2.5-1.5B-Instruct@989aa79> -o out -p int4 -e dml -c cache \
      --extra_options shared_embeddings=false
  

  shared_embeddings=false keeps the embeddings out of GatherBlockQuantized, which DirectML can't run; without it the model fails to load with graph capture on.

Files

File	Bytes	sha256
genai_config.json	1,867	e496d8c3778c3595a440fa06d2c16237522549d3d4f98eb6f4c7cb7a2d755c21
model.onnx	310,715	09d14950e1e362ab5b3e783f1711af9c81793acb2646f30f63effdaae77a11bc
model.onnx.data	1,343,750,144	6f13389c8f11c0deb0ba855aa5e6450326312576b8bf10f60913f94451f0dfa3
tokenizer.json	11,421,892	3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
tokenizer_config.json	721	415b7f0fb037ee2a98f8ab6e7e2ba9e3d43ac270a992bdd149b008d25f69918f
chat_template.jinja	2,561	8aa40ce145adb73cb3a75194dc0224702a95850ec5275cabb728496bbd749fc6

Use

Tested with Microsoft.ML.OnnxRuntimeGenAI.WinML 0.18.0 on Windows ML 2.4.89 (DirectML): about 0.86 s p50 for a 10-token JSON answer to a ~1,070-token prompt on an Intel Arc 140V iGPU.

Known issue: with graph capture on (the default in genai_config.json), private memory grows by about 56 MB per generation on that runtime. Foundry Local's generic-gpu build of the same model behaves the same way. Turning graph capture off stops the growth but makes generation about 4× slower.

Licence

Apache License 2.0, as for the base model (see LICENSE, copied from the base model's repository). Qwen2.5 is by the Qwen team, Alibaba Cloud.