metadata
license: cc-by-sa-4.0
datasets:
- mlfoundations/dclm-baseline-1.0-parquet
- HuggingFaceFW/finepdfs-edu
- openbmb/UltraData-Code
- openbmb/UltraData-Math
8000 vocab size tokenizer padded to 8192, trained on the following data:
DATASETS = [
("mlfoundations/dclm-baseline-1.0-parquet", None, "train", "text", 75000),
("HuggingFaceFW/finepdfs-edu", "eng_Latn", "train", "text", 7500),
("openbmb/UltraData-Code", "UltraData-Code-L2", "cpp", "content", 5000),
("openbmb/UltraData-Code", "UltraData-Code-L2", "js", "content", 20000),
("openbmb/UltraData-Code", "UltraData-Code-L2", "py", "content", 20000),
("openbmb/UltraData-Code", "UltraData-Code-L2", "rust", "content", 5000),
("openbmb/UltraData-Math", "UltraData-Math-L2-preview", "train", "content", 15000),
]
Code is in notebook.py