jiteshkprasad
6
followers ·
8 following Building udgamlabs.com
replied to their post about 11 hours ago Last week we released Gargi, a 384M English language model trained from scratch on about 17B tokens, roughly 1,000× less training data than Qwen2.5-0.5B (~18T tokens, as published by its authors).
What we measured (same harness for every model: lm-evaluation-harness, 0-shot, greedy, each model's start token):
- Base model vs Qwen2.5-0.5B: ARC-Easy 62.8 vs 64.7, ARC-Challenge 32.3 vs 32.4, WinoGrande 54.6 vs 56.1.
- Chat model: thinking on or off; GSM8K 36.0 with thinking on.
- Long inputs: needle-in-a-haystack recall 100% at 8K, 93% at 16K and 90% at 20K tokens, so use it up to about 16K.
Where it falls short: knowledge and everyday common sense still scale with data. MMLU is 25.5 vs 47.3, and it trails on HellaSwag and PIQA. It is not a maths specialist, and it is not safety-tuned: put a moderation layer in front of any public use.
It runs on a laptop: GGUF (Q8_0 at 410 MB, Q4_K_M, f16) and MLX builds.
`ollama run hf.co/udgamlabs/Gargi-GGUF:Q8_0`
Apache-2.0 ·
weights: https://huggingface.co/udgamlabs/Gargi
try it: https://udgamlabs.com/models/gargi
code: https://github.com/udgamlabs/gargi
Named after Gargi Vachaknavi, the Vedic philosopher remembered for asking the hardest questions. replied to their post about 13 hours ago Last week we released Gargi, a 384M English language model trained from scratch on about 17B tokens, roughly 1,000× less training data than Qwen2.5-0.5B (~18T tokens, as published by its authors).
What we measured (same harness for every model: lm-evaluation-harness, 0-shot, greedy, each model's start token):
- Base model vs Qwen2.5-0.5B: ARC-Easy 62.8 vs 64.7, ARC-Challenge 32.3 vs 32.4, WinoGrande 54.6 vs 56.1.
- Chat model: thinking on or off; GSM8K 36.0 with thinking on.
- Long inputs: needle-in-a-haystack recall 100% at 8K, 93% at 16K and 90% at 20K tokens, so use it up to about 16K.
Where it falls short: knowledge and everyday common sense still scale with data. MMLU is 25.5 vs 47.3, and it trails on HellaSwag and PIQA. It is not a maths specialist, and it is not safety-tuned: put a moderation layer in front of any public use.
It runs on a laptop: GGUF (Q8_0 at 410 MB, Q4_K_M, f16) and MLX builds.
`ollama run hf.co/udgamlabs/Gargi-GGUF:Q8_0`
Apache-2.0 ·
weights: https://huggingface.co/udgamlabs/Gargi
try it: https://udgamlabs.com/models/gargi
code: https://github.com/udgamlabs/gargi
Named after Gargi Vachaknavi, the Vedic philosopher remembered for asking the hardest questions. reacted to their post with 👀 about 15 hours ago We tested 27 small open language models, all under 400M parameters, in the same way. Same harness, same settings, and the model versions pinned. I think small models are really hard to compare from their own cards, so we put the whole thing out.
Board: https://huggingface.co/spaces/udgamlabs/small-model-board
Dataset: https://huggingface.co/datasets/udgamlabs/small-model-board-scores
Write-up on the Udgam Labs blog: https://udgamlabs.com/blog/small-model-board
The most surprising part for me was MMLU. In the usual A, B, C, D format, just answering "D" every time gets 26.9%, and 23 of these 27 models score at or below that. So at this size the letter format basically tests if a model can pick a letter, not what it knows. When we asked the same questions in cloze format, where the answer text itself is scored, all 23 did better.
Our own models are in there like everyone else. Laghu-Base is 5th of 19 models up to 200M, and Gargi-Base is 7.3 points behind SmolLM2-360M. So we are not on top, and that is fine. What I wanted was a fair comparison.
Do you compare small models on letter or cloze MMLU? View all activity view post We tested 27 small open language models, all under 400M parameters, in the same way. Same harness, same settings, and the model versions pinned. I think small models are really hard to compare from their own cards, so we put the whole thing out. Board: udgamlabs/small-model-board Dataset: udgamlabs/small-model-board-scores Write-up on the Udgam Labs blog: https://udgamlabs.com/blog/small-model-board The most surprising part for me was MMLU. In the usual A, B, C, D format, just answering "D" every time gets 26.9%, and 23 of these 27 models score at or below that. So at this size the letter format basically tests if a model can pick a letter, not what it knows. When we asked the same questions in cloze format, where the answer text itself is scored, all 23 did better. Our own models are in there like everyone else. Laghu-Base is 5th of 19 models up to 200M, and Gargi-Base is 7.3 points behind SmolLM2-360M. So we are not on top, and that is fine. What I wanted was a fair comparison. Do you compare small models on letter or cloze MMLU? See translation
Laghu: a 152M chat model trained from scratch for $146. Everything is open.
