HelaAI/SinhalaLM-V.0.0.1-it

🇱🇰 SinhalaLM — Instruct

A fully fine-tuned, instruction-following Sinhala question-answering model

SinhalaLM-Instruct · by HelaAI

---

📑 Table of contents

• Overview
• Relationship to the base model
• Prompt template
• Architecture
• Fine-tuning
• How to use
• Intended use
• Limitations and biases
• Files in this repo
• License
• Citation
• Contact

---

🔎 Overview

SinhalaLM — Instruct is an instruction-tuned, question-answering variant of SinhalaLM, a GPT-style decoder-only transformer built from scratch for the Sinhala language. It combines the same core architectural ingredients as the base model — Rotary Position Embeddings (RoPE), SwiGLU gated feed-forward blocks, and a custom Sinhala byte-pair encoding (BPE) tokenizer — with full fine-tuning on a Sinhala question-answer dataset, so that every parameter in the model (not just a low-rank adapter subset) adapts to the instruction-following QA task.

Given a Sinhala question wrapped in a fixed instruction template, the model generates a Sinhala answer directly, without requiring any few-shot examples at inference time.

This is a custom architecture — it is not a standard transformers (AutoModel) class. The full model definition, tokenizer, and prompt-building utilities ship in this repository as modeling_sinhalalm.py and must be imported directly (see How to use below).

Detailed internal architecture of a single SinhalaLM decoder block — showing the QKV split, per-head RoPE application, the fused Flash-Attention kernel, both residual connections, and the SwiGLU feed-forward path with weight tying between the token embedding and the LM head. The same block design underlies both the base and instruct-tuned checkpoints.

⚠️ Private repository. Only Hugging Face accounts explicitly granted access can pull this repo. Do not share the access token.

---

🔗 Relationship to the base model

SinhalaLM — Instruct is obtained by fully fine-tuning the private base checkpoint ChamaraVishwajithRajapaksha/SinhalaLM-V.0.0.1 — a GPT-style Sinhala language model pretrained from scratch — on Sinhala question-answer data. "Full" fine-tuning means every weight in the network is updated during this stage, rather than freezing the backbone and training a small set of adapter parameters (e.g. LoRA). This is practical here specifically because the base model is small (tens of millions, not billions, of parameters), which keeps full fine-tuning affordable on a single GPU while still letting every layer specialize to the QA task.

Base model	ChamaraVishwajithRajapaksha/SinhalaLM-V.0.0.1
Adaptation method	Full fine-tuning (all parameters updated)
Task	Sinhala instruction-following question answering
Fine-tuning data	ChamaraVishwajithRajapaksha/sinhala-question-answer-pairs, ihalage/sinhala-finetune-qa-eli5

---

📝 Prompt template (Alpaca-style, Sinhala)

The model was fine-tuned on prompts built like this — use the exact same format at inference time, and stop generating at <EOS>:

පහත ප්‍රශ්නයට සිංහලෙන් පිළිතුරු දෙන්න.

### ප්‍රශ්නය:
<your question here>

### පිළිතුර:
<model completes from here>

Loss was masked during training so that only the answer tokens (after ### පිළිතුර:) contributed to the loss — the model was never trained to reproduce the question or the system instruction header, only to continue them with an appropriate answer.

---

🏗️ Architecture

SinhalaLM — Instruct inherits its decoder-only transformer architecture directly from the SinhalaLM base model.

Component	Design choice
Parameters	~44.7M
Embedding dim	512
Attention heads	8
Transformer layers	8
Context length	512 tokens
Vocabulary size	15,000
Position encoding	Rotary (RoPE)
Feed-forward	SwiGLU (gated activation)
Normalization	RMSNorm (pre-norm)
Weight tying	Input token embedding tied to the output LM head

Note: the instruct checkpoint's parameter count (44.7M) is larger than the base model's (32.9M); the additional parameters come from task-specific components added during fine-tuning on top of the shared transformer backbone.

---

🎯 Fine-tuning

Dataset	ChamaraVishwajithRajapaksha/sinhala-question-answer-pairs / ihalage/sinhala-finetune-qa-eli5
Question column	input
Answer column	target
Adaptation method	Full fine-tuning (all parameters trainable)
Prompt template	Alpaca-style, Sinhala system header
Loss masking	Yes — answer tokens only
Training steps	447
Final train loss	2.254163
Final validation loss	2.381392
Best validation loss	2.381392
Best perplexity	10.82

Full fine-tuning curves and a CSV/JSON breakdown are provided in analytics/. Example generations from the fine-tuning run are in sample_generations.txt (if included in this repo).

---

🚀 How to use

Install dependencies:

pip install torch huggingface_hub safetensors

Download, load, and ask a question:

from huggingface_hub import snapshot_download
import torch, json, sys

# 1. Download the repo (private — requires a token with read access)
local_dir = snapshot_download(repo_id="ChamaraVishwajithRajapaksha/SinhalaLM-V.0.0.1-it", token="YOUR_HF_TOKEN")

# 2. Import the model + tokenizer classes shipped in this repo
sys.path.insert(0, local_dir)
from modeling_sinhalalm import SinhalaGPT, FastBPETokenizer, build_prompt, get_prompt_prefix

# 3. Load config
config = json.load(open(f"{local_dir}/config.json"))

# 4. Load tokenizer
tokenizer = FastBPETokenizer.load(f"{local_dir}/bpe_vocabulary.json")

# 5. Build model + load fine-tuned weights
from safetensors.torch import load_file
state_dict = load_file(f"{local_dir}/model.safetensors")

model = SinhalaGPT(
    vocab_size=config["vocab_size"],
    embed_dim=config["n_embed"],
    n_heads=config["n_heads"],
    n_layers=config["n_layers"],
    block_size=config["block_size"],
    dropout=0.0,  # no dropout at inference
)
model.load_state_dict(state_dict)
model.eval()

# 6. Ask a question using the exact fine-tuning prompt template
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

question = "ශ්‍රී ලංකාවේ අගනුවර කුමක්ද?"
prefix = get_prompt_prefix(question)
ids = [tokenizer.stoi.get("<BOS>", 2)] + tokenizer.encode(prefix)
ctx = torch.tensor([ids], dtype=torch.long, device=device)

out = model.generate(ctx, max_new_tokens=200, temperature=0.7, top_k=50)
answer_ids = out[0, len(ids):].tolist()
print(tokenizer.decode(answer_ids))

Generation parameters

Parameter	Effect
temperature	Lower (e.g. 0.5–0.7) → more focused, deterministic answers. Higher (e.g. 1.0+) → more varied/creative phrasing.
top_k	Restricts sampling to the k most likely tokens at each step. Lower = safer, less diverse.
max_new_tokens	Number of answer tokens to generate. Total context (prompt + answer) is capped at 512 tokens.

⚠️ Always build the prompt with get_prompt_prefix() (or the exact template shown above) — deviating from the fine-tuning prompt format will noticeably degrade answer quality, since the model was only trained to continue that specific structure.

---

🎯 Intended use

SinhalaLM — Instruct is intended for Sinhala question answering in research and prototyping contexts — e.g. exploring compact, from-scratch, instruction-tuned models for a low-resource language, or as a lightweight QA component in a larger Sinhala NLP pipeline.

Not intended for:

• ❌ Production deployment without further evaluation, safety review, and human oversight.
• ❌ High-stakes factual use (medical, legal, financial, or safety-critical questions) — answers are not verified or fact-checked.
• ❌ Multi-turn dialogue or open-ended chat — the model is fine-tuned specifically for single-turn question → answer completion in the prompt format above.
• ❌ Any use requiring inputs/outputs longer than 512 tokens total.

---

⚠️ Limitations and biases

• Fine-tuned on a relatively small QA dataset (447 training steps) — coverage of topics outside the fine-tuning distribution may be shallow or inaccurate.
• As with any fine-tuned language model, answers may be fluent but factually incorrect ("hallucinated") — always verify important claims independently.
• Inherits any biases present in both the base pretraining corpus and the QA fine-tuning data.
• Context window is limited to 512 tokens (prompt + answer combined).
• Strict reliance on the exact prompt template — inputs that deviate from the trained format may produce degraded or off-format answers.
• No safety filtering or content moderation has been applied to outputs.
• Monolingual (Sinhala) by design — not intended for multilingual or code-switched input.

---

📁 Files in this repo

File	Description
model.safetensors	Fine-tuned model weights
config.json	Architecture + fine-tuning config
bpe_vocabulary.json	Full tokenizer (vocab + merges + priorities) — load with FastBPETokenizer.load(...)
vocab.json, merges.txt, tokenizer.json, tokenizer_config.json, special_tokens_map.json	HF-style tokenizer files (informational; the authoritative loader is FastBPETokenizer)
modeling_sinhalalm.py	Full source of the model, tokenizer, and prompt-template functions (SinhalaGPT, FastBPETokenizer, build_prompt, get_prompt_prefix)
images/sinhalalm_detailed_architecture.svg	Detailed architecture diagram of a single SinhalaLM decoder block
sample_generations.txt	Example Q&A generations from the fine-tuning run (if present)
analytics/	Fine-tuning loss / validation / learning-rate / perplexity graphs, CSVs, and a text + JSON summary

---

📜 License

This model is released under a custom/other license. Update this section with your actual terms (e.g. permitted uses, redistribution, commercial use) before granting access to others.

---

📚 Citation

If you use this model, please cite:

@misc{sinhalalminstruct,
  title  = {SinhalaLM -- Instruct},
  author = {Chamara Vishwajith and Dr. Kushan Sudheera and Dineth Jayakody},
  year   = {2026},
  note   = {Instruction-tuned, fully fine-tuned Sinhala question-answering model, built on the SinhalaLM base checkpoint}
}

---

📬 Contact

Developed by HelaAI. For questions or access requests, contact one of the authors of the base SinhalaLM model, or the repository owner on Hugging Face.

—— SinhalaLM · Instruct · HelaAI ——