A leaderboard for sub-400M parameter language models, evaluated using LM-eval harness or a custom benchmark script available here ArithMark-3.0.
To support our work and help us keep this leaderboard up to date, please consider giving the space a like and follow!
Ranked by Intelligence Index
Short notes on new models and leaderboard shifts.
The leaderboard now welcomes models below 400M parameters, expanding the previous 150M limit. Use the new <250M filter to compare models within the wider range, or select All to explore the full leaderboard.
GPT-X3-150M has taken the #1 spot from cagliostro, becoming the first ever model to surpass 30 on the Intelligence Index
Bench Labs' cagliostro-v3.5 has taken the #1 spot, becoming the first model to defeat Huggingface's SmolLM2.
Bench Labs' cagliostro-v3 hits Intelligence Index 26.13, a 146M-param model trained on 75b tokens, from scratch on a single consumer GPU. It places 2nd on the leaderboard, making it the most capable SLM we've seen trained on consumer hardware.
The leaderboard UI now brings filter leaders, animated benchmark expansion, Intelligence Index fit scores, and paged release notes into one clearer workspace.
ArithMark-3 is now the primary math benchmark for ranking, weighting subject to change. ArithMark-2 remains available in the expanded columns.
UCR's 2.74M SLM uses custom tokenization and digit features to reach 69.4% ArithMark accuracy, setting the leaderboard's #1 score
Built on Axiomic Labs' T-X4 refresh-gated XSA stack, GPT-S2-5M pushes into the sub-10M bracket and lands at the top, dethroning SLM-10M.
Veyra AI's 49M Apricot checkpoint posts a 37.63 average, setting the strongest score in the sub-50M bracket.
Zero-shot evaluation. Higher is better for all columns. Gold names mark size category leaders. Click any header to sort.
| Compare | # ▼ | Model | Params | Int Index | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3 | Avg | Fit Std Devs | ArithMark-2 |
|---|
Models you add from the leaderboard line up side by side here. None selected
Top scores for the active size and benchmark filters.
Score vs parameter count (log scale). Shaded zone = above regression line.
Indie model leadership across the <150M, <100M, <50M, <10M and <3M categories.
Each step marks a new category leader, using addition dates and current II. Colours identify the lab. Hover a dot for model details; select a category to show or hide its line.
Size categories overlap. A lab leading two categories for one day earns two class-days.
Total #1 class-days per model, summed across the same five size categories.
Each currently listed, uncommented model joins the timeline on its stored addition date. Rankings throughout the timeline use the model's current Intelligence Index, parameter count and organization. For each category, the highest-scoring eligible model receives each calendar day from its lead's start through the day before the next leader arrives. Exact ties follow the leaderboard's model order. Hugging Face, Meta, OpenAI, Google and Microsoft entries are excluded from these indie lab charts.
The timeline is reconstructed from addition dates and current scores, rather than past score revisions. Totals include today in Australia/Sydney and are recalculated when the page loads.
Eligibility, evaluation, and submission requirements for the Open SLM Leaderboard.
The leaderboard is for language models with fewer than 400 million parameters.
The submitted model must be pretrained from scratch by you or your organization.
Model weights must be openly available and point to a genuine training checkpoint, not a merged model.
Provide zero-shot results for the listed benchmarks using LM-eval harness or the linked custom ArithMark script.
Submitted results are independently checked by the leaderboard team before they are merged.
Open a PR or discussion with the benchmark results in the leaderboard Space.
Open a PR or discussion on this Space with your model's results for the given benchmarks. They will be independently verified by our team and then your PR will be merged. Your model must be pretrained by you from scratch, open weights and be a training checkpoint (not merged) to qualify Open a PR →
Each benchmark is first adjusted for its random-chance floor, so chance performance maps to 0 and perfect performance maps to 100.
Combined ARC is the mean of ARC-Easy and ARC-Challenge before normalization. When a component is unavailable, its weight is removed from both the numerator and denominator; at least two components are required. Scores below chance may be negative. ArithMark-2 is displayed separately and is not included in the index.