---
license: apache-2.0
language:
  - en
  - fr
tags:
  - text-generation
  - causal-lm
  - transformer
  - jax
  - flax
pipeline_tag: text-generation
datasets:
  - wikipedia
  - oscar-corpus/OSCAR-2301
  - bigcode/the-stack-dedup
library_name: transformers
---

# Julian 500M (10B Tokens)

Decoder-only transformer trained from scratch on 10 billion tokens of English and French text.

This model is part of the **Julian model family**, a personal research project exploring LLM training from scratch. The goal is educational: understanding every step of the process, from data preparation to distributed training on TPUs.

## Architecture

- **Parameters**: 508M
- **Hidden size**: 1280
- **Layers**: 18
- **Attention heads**: 20 (head dim 64)
- **FFN**: 5120 (SwiGLU, 3 projections)
- **Context**: 2048 tokens
- **Vocab**: 50,000 (SentencePiece)

The model uses RoPE positional embeddings, RMSNorm, and has no bias terms in linear layers.

## Training

Trained on TPU v5litepod-32 for approximately 8 hours.

**Data mix**:
- Wikipedia (EN/FR): 50%
- OSCAR web corpus: 30%
- The Stack (code): 15%
- Gutenberg books: 5%

Language ratio is roughly 70% English, 30% French.

**Hyperparameters**:
- Batch size: 131k tokens
- Learning rate: 6e-4 with cosine decay
- Warmup: 3000 steps
- AdamW (β2=0.95), weight decay 0.1

Final validation perplexity: **12.1**

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("juliankerignard/julian-500m-10b")
tokenizer = AutoTokenizer.from_pretrained("juliankerignard/julian-500m-10b")

inputs = tokenizer("La France est", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Limitations

This is a base model trained on a modest amount of data. It will:
- Generate repetitive or incoherent text on longer sequences
- Hallucinate facts
- Not follow instructions (no fine-tuning)

Not recommended for production use. Intended for research and experimentation.

## Julian Model Family

| Model | Parameters | Training Tokens | Status |
|-------|------------|-----------------|--------|
| julian-500m-10b | 508M | 10B | Released |
| julian-500m-39b | 508M | 39B | Training |

Future models (1B+) are planned as the project evolves.

## Acknowledgments

This project would not have been possible without the **Google TPU Research Cloud (TRC) program**, which provides free access to Cloud TPU resources for open research.

A sincere thank you to the TRC team for supporting independent researchers and making this kind of work accessible. If you're working on ML research, I highly encourage applying to the program.

More info: [TPU Research Cloud](https://sites.research.google/trc/about/)

## License

Apache 2.0

## Citation

```bibtex
@misc{julian2026,
  author = {Kerignard, Julian},
  title = {Julian 500M},
  year = {2026},
  url = {https://huggingface.co/juliankerignard/julian-500m-10b}
}
```