Smaller quants
Hi Harry,
Would it be possible to get some smaller quants of this model to save some VRAM, like for instance Q4_K_S, Q4_K_M, i1-Q4_K_S or i1-Q4_K_M?
Alternatively could you provide a link to the original SafeTensors model somewhere?
Hi! It was an experiment with Unsloth, so no original Safetensors remain. In general - don't worry, it was a failure anyway, the model simply returns an erroneous output "I cannot understand" to everything.
But I have a dataset I used I think. I pulled it from Skyrim by a tool
mradermacher just posted a gguf version that is imatrix and normal static quants that likely goes down to Q1 or Q2.
Though if HarryDn thinks it's a failure then probably pass on it and have it retrain the finetune.
Btw @RogerS-01 , in my experience MOE's like 26B A4B's have a much higher output speed even if it doesn't fit all on your card vs tenser 12B's. Due to that i have a preference for the 26B (G4) and 35B (Qwen) models.