glossia/Rambley-V2-350M-Preview

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_challenge	1	none	0	acc	↑	0.2295	±	0.0123
		none	0	acc_norm	↑	0.2637	±	0.0129
arc_easy	1	none	0	acc	↑	0.5400	±	0.0102
		none	0	acc_norm	↑	0.4722	±	0.0102
hellaswag	1	none	0	acc	↑	0.3140	±	0.0046
		none	0	acc_norm	↑	0.3595	±	0.0048
piqa	1	none	0	acc	↑	0.6556	±	0.0111
		none	0	acc_norm	↑	0.6442	±	0.0112

Comparison

Metric	Rambley-V2-350M-Preview	GPT-2 Medium	OPT-350m	Pythia 410M	GPT-3 Medium (closed source, reported)
Hellaswag	35.95	39.38	36.82	40.63	43.6
Arc-E	47.22	43.60	40.19	45.92	46.5
Arc-C	26.37	25.00	23.72	24.49	29.5
PIQA	64.42	66.38	64.36	67.30	70.2
Arithmark 3.0	37.80%	33.80%	38.30%	36.90%	N/A
Training Tokens	4B	Unk. (~20B)	180B	~300B	~300B

Bold indicates the best open source model, and italics indicate the best overall model when it is closed source. Bolded italics indicate the best overall model when it is open source.

Architecture Details

• 2.5x intermediate SwiGLU MLPs
• Headwise gated attention (headwise chosen instead of elementwise to save params for the smaller model families)
• Learned attention sinks on every attention head
• SWA:SWA:GQA attention pattern, with the final layer forced to GQA
• 512 sliding window size
• 2048 native context length
• HoPE positional embeddings (on both SWA and GQA layers)
• One MTP head for auxiliary CE loss at 0.3x multiplier (currently unused at inference)
• Optimizers:
  • Muon at 0.02 LR for matrices with Hyperball weight norm constraints
  • AdamW at 0.002 LR for embeddings and vectors, norms, etc. with no weight decay or Hyperball

Training Data

Full mix available here, but it's around 17B tokens of mostly fineweb-edu and dclm-edu with some ultradata-math mixed in

Code

Pretraining codebase: https://code.allura.moe/fizz/OLMo-core

Other Benchmarks

====================================================================                                                                                                       
  Loading ../exported_hf...                                                                                                                                                
====================================================================                                                                                                       
Loading weights:
  368,741,044 parameters (torch.bfloat16)                                                                                                                                  
  arithmark-3: raw 37.50% (375/1000)  normalized 37.80% (378/1000)                                                                                                         
  speed: 648.3 examples/s  (tokenize 0.24s, evaluate 1.54s)
  Category                                                             N       Raw   Normalized
  ----------------------------------------------------------------------------------------------
  elementary_school_math_continuation::addition::grades_1_2::easy    128    29.69%       29.69%
  elementary_school_math_continuation::comparison::grades_2_3::medium    44    20.45%       20.45%
  elementary_school_math_continuation::comparison_difference::grades_2_3::medium    48    31.25%       31.25%
  elementary_school_math_continuation::data::grades_2_3::easy         43    27.91%       27.91%
  elementary_school_math_continuation::division::grades_3_4::medium    54    42.59%       42.59%
  elementary_school_math_continuation::fractions_counting::grades_3_4::medium    50    30.00%       32.00%
  elementary_school_math_continuation::geometry_area::grades_4_5::medium    52    63.46%       65.38%
  elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium    45    53.33%       53.33%
  elementary_school_math_continuation::measurement::grades_2_3::easy    76    38.16%       38.16%
  elementary_school_math_continuation::money::grades_3_4::medium      64    31.25%       31.25%
  elementary_school_math_continuation::multiplication::grades_3_4::medium    74    54.05%       55.41%
  elementary_school_math_continuation::patterns::grades_3_4::medium    53    32.08%       32.08%
  elementary_school_math_continuation::subtraction::grades_1_2::easy   117    27.35%       27.35%
  elementary_school_math_continuation::time::grades_2_3::easy         55   100.00%      100.00%
  elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium    46    10.87%       10.87%
  elementary_school_math_continuation::two_step_addition::grades_2_3::medium    19    21.05%       21.05%
  elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium    32    12.50%       12.50%
====================================================================
  ../exported_hf (368,741,044 params) RESULTS
====================================================================
  Raw continuation accuracy        37.50%
  Length-normalized accuracy       37.80%
  Primary (acc_norm)           37.80%
====================================================================