Instructions to use aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8")
model = AutoModelForMultimodalLM.from_pretrained("aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8

SGLang

How to use aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8 with Docker Model Runner:
```
docker model run hf.co/aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8
```

JEV-27B-VL — AWQ + GPTQ INT4, W4A8 (int4 weights, int8 activations)

4-bit quantization of autotrust/JEV-27B-VL with its LoRA decision head kept as a LoRA (rank 32) exactly as the base checkpoint ships it. Built with llm-compressor, served by vLLM on a single 24 GB GPU. Weights are INT4 (group 128, symmetric); activations are quantized to INT8 per token, so prefill runs on int8 tensor cores via Marlin.

What is quantized

part	precision
every text `Linear`, including the GatedDeltaNet projections	int4, group 128, symmetric + int8 per-token dynamic activations (Marlin W4A8 / `CompressedTensorsWNA8Int`)
`lm_head` (decision head)	bf16
vision tower	bf16 (untouched; spliced back after quantization)

Recipe: AWQ smoothing (hybrid-attention mappings, duo scaling) followed by GPTQ, calibrated on 1024 kind-balanced serving-format prompts (45% choice) from the provider corpus (SargeDev/jev-distill-corpus-v3, calibration split), max_seq_length 2048. Built with llm-compressor 0.14 / compressed-tensors 0.19 / transformers 5.17 / torch 2.14.

Validation

Served with the same stack as the deployed W4A16 build (--kv-cache-dtype fp8, --max-num-seqs 8, one RTX 4090) and scored on the same 500 held-out corpus rows (seeded shuffle), through the same POST /v1/decide shim:

slice	W4A8 + AWQ (this build)	deployed W4A16 g64 (v2)
KL mean, all rows	0.0147	0.0143
noul	0.0022	0.0021
score	0.0186	0.0180
choice	0.0385	0.0374
top-1 noul	0.732	0.774
top-1 choice	0.825	0.851
card example (`billing`)	p = 0.9963	p = 0.9976

Image decisions verified (red circle -> circle, p = 0.99999), so the bf16 vision tower is intact. Cold prefill on a 5,098-token decision: 1.76 s (0.345 ms/token) — the deployed W4A16 build measures 0.350 ms/token on the same harness, i.e. parity: at these batch sizes this workload is bandwidth-bound, not GEMM-bound, so 8-bit activations do not buy prefill throughput here.

Serve

hf download aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8 --local-dir JEV-27B-VL-AWQ-INT4-W4A8

serve.sh / serve_decide.py are the provider's own server; start vLLM with --enable-lora --max-lora-rank 32 --lora-modules jev-decision=<dir>/adapter_vllm (see the base model card) plus --kv-cache-dtype fp8 --gpu-memory-utilization 0.95 --max-num-seqs 8 on a 24 GB card. decision_head/ carries the head bias + per-kind temperatures.

License

Apache-2.0, same as autotrust/JEV-27B-VL.

Downloads last month: 28

Safetensors

Model size

27B params

Tensor type

I32

BF16

Model tree for aioutfitters/JEV-27B-VL-AWQ-INT4-W4A8

Base model

Qwen/Qwen3.8-27B

Adapter

autotrust/JEV-27B-VL

Quantized

(13)

this model

aioutfitters
/

JEV-27B-VL-AWQ-INT4-W4A8