Mostly Olmo-3 architecture 10M model with 2:1 SWA:GQA and HoPE embeddings. Trained on like 20B tokens of Fineweb-Edu (filtered to the splits before ChatGPT's release) and UltraData-Math.
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.1809 | ± | 0.0112 |
| none | 0 | acc_norm | ↑ | 0.2193 | ± | 0.0121 | ||
| arc_easy | 1 | none | 0 | acc | ↑ | 0.3809 | ± | 0.0100 |
| none | 0 | acc_norm | ↑ | 0.3544 | ± | 0.0098 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2722 | ± | 0.0044 |
| none | 0 | acc_norm | ↑ | 0.2751 | ± | 0.0045 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.5724 | ± | 0.0115 |
| none | 0 | acc_norm | ↑ | 0.5615 | ± | 0.0116 |
elementary_school_math_continuation::patterns::grades_3_4::medium 53 30.19% 30.19%
elementary_school_math_continuation::subtraction::grades_1_2::easy 117 17.95% 18.80%
elementary_school_math_continuation::time::grades_2_3::easy 55 100.00% 100.00%
elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium 46 28.26% 28.26%
elementary_school_math_continuation::two_step_addition::grades_2_3::medium 19 15.79% 15.79%
elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium 32 28.12% 28.12%
====================================================================
../step42526/ (9,835,712 params) RESULTS
====================================================================
Raw continuation accuracy 31.00%
Length-normalized accuracy 31.30%
Primary (acc_norm) 31.30%
====================================================================
Results saved to benchmark_cache/.._step42526__arithmark-3_results.json
- Downloads last month
- 45
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support