Add code? (+suggestions)
I love the expansion to <400M. It really broadens the possibilities of what these models can do. I just think having some code marks or evaluation could be useful, because it would be nice to have small powerful LMs that can do some basic coding (like HTML template generation)
I also think temper evals (training on eval data -- 'benchmaxxing') is important. This could be possible by sampling the model and seeing whether the model is fine tuned / outputs benchmark data instead of a more even distribution.
Since models are getting larger now, I think it would be useful to show the max context length (toks) on the leaderboard, but it's not mandatory
MMLU-Pro and GPQA-D could be useful as well. The issue is this would mean that the whole Intelligence Index is calculated differently. Perhaps a Intelligence Index-L or v2?
Maybe instruction-following benchmarks would be handy. After all, these are all causal language models.
I feel like adding larger reference points > 400M like Qwen, MiniCPM, etc in a separate discrete section would be useful because we can see how far we can close the gap.
Hi @marissatech thanks for the excellent suggestions, this is just phase 1 of a planned leaderboard upgrade, we plan to add significantly more including a separate board for instruct models.
@Datdanboi25 think about adding https://huggingface.co/datasets/BananaMind/BananaMind-Instruct-Bench-1.1 for that
Will add it to the list of benchmarks to test, thanks banaxi!
if you want can make a 1.2 with some changes its kinda old now
if you want can make a 1.2 with some changes its kinda old now
Oh man that's so sad...
BUT DO NOT WORRY BECAUSE YOU CAN GO TO https://huggingface.co/datasets/ArrowLabs/ArrowBench-Instruct FOR A NEW, BETTER, LARGER, HARDER VERSION!