latentdivergence/realvisxl-inpaint-npu-8elite

RealVisXL Inpaint (V5.0 Lightning) - Hexagon NPU (Snapdragon 8 Elite, HTP v79)

A photoreal SDXL inpainting UNet for on-device editing (Erase refine, and Replace/Uncrop fallback) in Latent Studio, precompiled for the Qualcomm Hexagon NPU with Qualcomm AI Hub (QAIRT 2.45) as QNN EPContext ONNX graphs. All graphs take and return fp32 tensors (driven through ONNX Runtime's QNN execution provider) at 1024x1024.

How the UNet was made

An add-difference merge, no training:

unet = inpainting_0.1 + (RealVisXL_V5.0_Lightning - sdxl_base_1.0)

diffusers/stable-diffusion-xl-1.0-inpainting-0.1 is SDXL base 1.0 fine-tuned for inpainting (9 input channels: latent 4 + mask 1 + masked-image latent 4). Adding the difference between RealVisXL V5.0 Lightning and SDXL base carries RealVisXL's photoreal fine-tune and its Lightning distillation over (on the four latent channels of conv_in; the five mask/image channels stay the inpainting model's). The text encoders and VAE are RealVisXL's own, unchanged.

Sampling: Euler with trailing timestep spacing, 8 steps, guidance 1 (CFG on this distilled UNet tints the fill pink). Repaints start from the init image latent under full noise ("strength 0.99"; a pure-noise start leaves a hazy fill). Erase refines a LaMa fill at strength 0.75 (0.99 for holes of 12% of the frame or more). time_ids (1024, 1024, 0, 0, 1024, 1024). Masked pixels are mid-grey before the VAE encode; latents scale by 0.13025.

Files

Folder	Graph	Precision	AI Hub ms/run	cosine vs fp32 (worst, real device)
unet/	inpainting UNet, batch 1: sample[1,9,128,128], timestep[1], encoder_hidden_states[1,77,2048], text_embeds[1,1280], time_ids[1,6] -> out_sample[1,4,128,128]	w8a16 (int8 weights, int16 activations)	739.8	0.999765
vae_encoder_npu/	SDXL VAE encoder (fp16-fix), image[1,3,1024,1024] in [-1,1] -> posterior mean[1,4,128,128] (unscaled)	fp16	818.8	0.999757

The VAE decoder is the SDXL (fp16-fix) decoder already hosted for this chip in latentdivergence/lcm-ssd1b-npu-8elite (vae_decoder/). The CLIP-L / OpenCLIP-bigG text encoders (int8, CPU) are RealVisXL Lightning's, from latentdivergence/realvisxl-lightning-int8.

w8a16 calibration (32 samples): forward-noised inputs on the exact 8-step schedule (x0 + sigma_i * noise) for real photo crops with Replace brush masks, Erase blobs over LaMa fills, Uncrop borders and full masks, with the real mask channel, masked-image latent, prompt embeddings and time_ids. Cosine is measured on held-out samples against the fp32 PyTorch UNet, on AI Hub's physical devices.

License

CreativeML Open RAIL++-M (LICENSE.md), inherited from all three source models (RealVisXL V5.0 Lightning by SG161222, SDXL inpainting 0.1 by the Diffusers team, SDXL base 1.0 by Stability AI). This is a derivative work (merged, quantised and compiled; no retraining). The use-based restrictions in Attachment A of the license apply to anyone who uses or redistributes these files, and must be passed on.