| --- |
| license: apache-2.0 |
| language: |
| - en |
| - fr |
| tags: |
| - text-generation |
| - causal-lm |
| - transformer |
| - jax |
| - flax |
| pipeline_tag: text-generation |
| datasets: |
| - wikipedia |
| - oscar-corpus/OSCAR-2301 |
| - bigcode/the-stack-dedup |
| library_name: transformers |
| --- |
| |
| # Julian 500M (10B Tokens) |
|
|
| Decoder-only transformer trained from scratch on 10 billion tokens of English and French text. |
|
|
| This model is part of the **Julian model family**, a personal research project exploring LLM training from scratch. The goal is educational: understanding every step of the process, from data preparation to distributed training on TPUs. |
|
|
| ## Architecture |
|
|
| - **Parameters**: 508M |
| - **Hidden size**: 1280 |
| - **Layers**: 18 |
| - **Attention heads**: 20 (head dim 64) |
| - **FFN**: 5120 (SwiGLU, 3 projections) |
| - **Context**: 2048 tokens |
| - **Vocab**: 50,000 (SentencePiece) |
|
|
| The model uses RoPE positional embeddings, RMSNorm, and has no bias terms in linear layers. |
|
|
| ## Training |
|
|
| Trained on TPU v5litepod-32 for approximately 8 hours. |
|
|
| **Data mix**: |
| - Wikipedia (EN/FR): 50% |
| - OSCAR web corpus: 30% |
| - The Stack (code): 15% |
| - Gutenberg books: 5% |
|
|
| Language ratio is roughly 70% English, 30% French. |
|
|
| **Hyperparameters**: |
| - Batch size: 131k tokens |
| - Learning rate: 6e-4 with cosine decay |
| - Warmup: 3000 steps |
| - AdamW (β2=0.95), weight decay 0.1 |
|
|
| Final validation perplexity: **12.1** |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("juliankerignard/julian-500m-10b") |
| tokenizer = AutoTokenizer.from_pretrained("juliankerignard/julian-500m-10b") |
| |
| inputs = tokenizer("La France est", return_tensors="pt") |
| outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.8) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Limitations |
|
|
| This is a base model trained on a modest amount of data. It will: |
| - Generate repetitive or incoherent text on longer sequences |
| - Hallucinate facts |
| - Not follow instructions (no fine-tuning) |
|
|
| Not recommended for production use. Intended for research and experimentation. |
|
|
| ## Julian Model Family |
|
|
| | Model | Parameters | Training Tokens | Status | |
| |-------|------------|-----------------|--------| |
| | julian-500m-10b | 508M | 10B | Released | |
| | julian-500m-39b | 508M | 39B | Training | |
|
|
| Future models (1B+) are planned as the project evolves. |
|
|
| ## Acknowledgments |
|
|
| This project would not have been possible without the **Google TPU Research Cloud (TRC) program**, which provides free access to Cloud TPU resources for open research. |
|
|
| A sincere thank you to the TRC team for supporting independent researchers and making this kind of work accessible. If you're working on ML research, I highly encourage applying to the program. |
|
|
| More info: [TPU Research Cloud](https://sites.research.google/trc/about/) |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{julian2026, |
| author = {Kerignard, Julian}, |
| title = {Julian 500M}, |
| year = {2026}, |
| url = {https://huggingface.co/juliankerignard/julian-500m-10b} |
| } |
| ``` |
|
|
