---
license: cc-by-sa-4.0
datasets:
- mlfoundations/dclm-baseline-1.0-parquet
- HuggingFaceFW/finepdfs-edu
- openbmb/UltraData-Code
- openbmb/UltraData-Math
---

8000 vocab size tokenizer padded to 8192, trained on the following data:

```python
DATASETS = [
    ("mlfoundations/dclm-baseline-1.0-parquet", None, "train", "text", 75000),
    ("HuggingFaceFW/finepdfs-edu", "eng_Latn", "train", "text", 7500),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "cpp", "content", 5000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "js", "content", 20000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "py", "content", 20000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "rust", "content", 5000),
    ("openbmb/UltraData-Math", "UltraData-Math-L2-preview", "train", "content", 15000),
]
```

Code is in `notebook.py`