Upload folder using huggingface_hub

README.md

@@ -0,0 +1,110 @@
+---
+license: apache-2.0
+language:
+  - en
+  - fr
+tags:
+  - text-generation
+  - causal-lm
+  - transformer
+  - jax
+  - flax
+pipeline_tag: text-generation
+datasets:
+  - wikipedia
+  - oscar-corpus/OSCAR-2301
+  - bigcode/the-stack-dedup
+library_name: transformers
+---
+
+# Julian 500M (10B Tokens)
+
+Decoder-only transformer trained from scratch on 10 billion tokens of English and French text.
+
+This model is part of the **Julian model family**, a personal research project exploring LLM training from scratch. The goal is educational: understanding every step of the process, from data preparation to distributed training on TPUs.
+
+## Architecture
+
+- **Parameters**: 508M
+- **Hidden size**: 1280
+- **Layers**: 18
+- **Attention heads**: 20 (head dim 64)
+- **FFN**: 5120 (SwiGLU, 3 projections)
+- **Context**: 2048 tokens
+- **Vocab**: 50,000 (SentencePiece)
+
+The model uses RoPE positional embeddings, RMSNorm, and has no bias terms in linear layers.
+
+## Training
+
+Trained on TPU v5litepod-32 for approximately 8 hours.
+
+**Data mix**:
+- Wikipedia (EN/FR): 50%
+- OSCAR web corpus: 30%
+- The Stack (code): 15%
+- Gutenberg books: 5%
+
+Language ratio is roughly 70% English, 30% French.
+
+**Hyperparameters**:
+- Batch size: 131k tokens
+- Learning rate: 6e-4 with cosine decay
+- Warmup: 3000 steps
+- AdamW (β2=0.95), weight decay 0.1
+
+Final validation perplexity: **12.1**
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("juliankerignard/julian-500m-10b")
+tokenizer = AutoTokenizer.from_pretrained("juliankerignard/julian-500m-10b")
+
+inputs = tokenizer("La France est", return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.8)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Limitations
+
+This is a base model trained on a modest amount of data. It will:
+- Generate repetitive or incoherent text on longer sequences
+- Hallucinate facts
+- Not follow instructions (no fine-tuning)
+
+Not recommended for production use. Intended for research and experimentation.
+
+## Julian Model Family
+
+| Model | Parameters | Training Tokens | Status |
+|-------|------------|-----------------|--------|
+| julian-500m-10b | 508M | 10B | Released |
+| julian-500m-39b | 508M | 39B | Training |
+
+Future models (1B+) are planned as the project evolves.
+
+## Acknowledgments
+
+This project would not have been possible without the **Google TPU Research Cloud (TRC) program**, which provides free access to Cloud TPU resources for open research.
+
+A sincere thank you to the TRC team for supporting independent researchers and making this kind of work accessible. If you're working on ML research, I highly encourage applying to the program.
+
+More info: [TPU Research Cloud](https://sites.research.google/trc/about/)
+
+## License
+
+Apache 2.0
+
+## Citation
+
+```bibtex
+@misc{julian2026,
+  author = {Kerignard, Julian},
+  title = {Julian 500M},
+  year = {2026},
+  url = {https://huggingface.co/juliankerignard/julian-500m-10b}
+}
+```

config.json

@@ -0,0 +1,21 @@
+{
+  "architectures": ["JulianForCausalLM"],
+  "model_type": "julian",
+  "vocab_size": 50000,
+  "hidden_size": 1280,
+  "intermediate_size": 5120,
+  "num_hidden_layers": 18,
+  "num_attention_heads": 20,
+  "num_key_value_heads": 20,
+  "hidden_act": "swiglu",
+  "max_position_embeddings": 2048,
+  "rms_norm_eps": 1e-5,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "use_cache": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.40.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28547c8f2e66ed86fb6dbc58cfbcd37393771e1218f3d6473e21d0dc576aabb0
+size 1199831920

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e6ce770d5698bf300081156406831abea73ec8ae5aa85c16234a27bdce7f94b
+size 974175

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "tokenizer_class": "LlamaTokenizer",
+  "model_max_length": 2048,
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "unk_token": "<unk>",
+  "pad_token": "<pad>"
+}