---
thumbnail: >-
  https://cdn-uploads.huggingface.co/production/uploads/634262af8d8089ebaefd410e/sJACk9FXM9BSk6x3B62UZ.png
license: cc-by-nc-sa-4.0
datasets:
  - HuggingFaceFW/fineweb-edu
  - openbmb/UltraData-Math
---
![4d696582-ab0b-44b9-b827-01ea7d71d55b](https://cdn-uploads.huggingface.co/production/uploads/634262af8d8089ebaefd410e/sJACk9FXM9BSk6x3B62UZ.png)


Mostly Olmo-3 architecture 10M model with 2:1 SWA:GQA and HoPE embeddings. Trained on like 20B tokens of Fineweb-Edu (filtered to the splits before ChatGPT's release) and UltraData-Math.

|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |     0|acc     |↑  |0.1809|±  |0.0112|
|             |       |none  |     0|acc_norm|↑  |0.2193|±  |0.0121|
|arc_easy     |      1|none  |     0|acc     |↑  |0.3809|±  |0.0100|
|             |       |none  |     0|acc_norm|↑  |0.3544|±  |0.0098|
|hellaswag    |      1|none  |     0|acc     |↑  |0.2722|±  |0.0044|
|             |       |none  |     0|acc_norm|↑  |0.2751|±  |0.0045|
|piqa         |      1|none  |     0|acc     |↑  |0.5724|±  |0.0115|
|             |       |none  |     0|acc_norm|↑  |0.5615|±  |0.0116|

```
  elementary_school_math_continuation::patterns::grades_3_4::medium    53    30.19%       30.19%
  elementary_school_math_continuation::subtraction::grades_1_2::easy   117    17.95%       18.80%
  elementary_school_math_continuation::time::grades_2_3::easy         55   100.00%      100.00%
  elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium    46    28.26%       28.26%
  elementary_school_math_continuation::two_step_addition::grades_2_3::medium    19    15.79%       15.79%
  elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium    32    28.12%       28.12%

====================================================================
  ../step42526/ (9,835,712 params) RESULTS
====================================================================
  Raw continuation accuracy        31.00%
  Length-normalized accuracy       31.30%
  Primary (acc_norm)           31.30%
====================================================================
Results saved to benchmark_cache/.._step42526__arithmark-3_results.json
```