script-langid/xlm-roberta-base-language-detection-sinhala-pali-sanskrit

XLM-R language detection: Sinhala, Pali and Sanskrit

A LoRA adapter for papluca/xlm-roberta-base-language-detection. It adds three labels for languages written in Sinhala script: Sinhala (sin_Sinh), Pali (pli_Sinh) and Sanskrit (san_Sinh). It also adds Devanagari Sanskrit (san_Deva), Tamil (tam_Taml) and Bengali (ben_Beng). The base model's 20 languages are kept.

The base model has no label for any of these three languages, and as released it scores 0 F1 on them. This adapter was trained with rehearsal: the target-language data was mixed with eight replay languages (san_Deva, eng_Latn, tam_Taml, hin_Deva, ben_Beng, arb_Arab, fra_Latn, deu_Latn), so the model still recognises them. A variant trained on the target languages alone scores about the same on Sinhala, Pali and Sanskrit, but its macro-F1 on public benchmarks drops to 0.30–0.44 (see below). For that reason this rehearsal checkpoint is the one released.

Labels (26)

ja nl ar pl de it pt tr es hi el ur bg en fr zh ru th sw vi (from the base model), then sin_Sinh pli_Sinh san_Sinh san_Deva tam_Taml ben_Beng (added). The order is in lidpipe_labels.json.

Usage

The adapter replaces the whole classification head, so widen the base model's head to 26 outputs before loading it:

import json, torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "script-langid/xlm-roberta-base-language-detection-sinhala-pali-sanskrit"
meta = json.load(open(hf_hub_download(repo, "lidpipe_labels.json"), encoding="utf-8"))
labels = meta["labels"]

model = AutoModelForSequenceClassification.from_pretrained(meta["base"], revision=meta["base_revision"])
old = model.classifier.out_proj
model.classifier.out_proj = torch.nn.Linear(old.in_features, len(labels))
model.config.num_labels = model.num_labels = len(labels)
model.config.id2label = dict(enumerate(labels))
model.config.label2id = {l: i for i, l in enumerate(labels)}
model = PeftModel.from_pretrained(model, repo).merge_and_unload().eval()
tokenizer = AutoTokenizer.from_pretrained(repo)

texts = ["ධම්මපදය බෞද්ධ සාහිත්‍යයේ වඩාත් ප්‍රසිද්ධ ග්‍රන්ථයකි.",
         "මනොපුබ්බඞ්ගමා ධම්මා මනොසෙට්ඨා මනොමයා."]
with torch.inference_mode():
    enc = tokenizer(texts, truncation=True, max_length=128, padding=True, return_tensors="pt")
    probs = model(**enc).logits.softmax(-1)
for text, p in zip(texts, probs):
    print(labels[p.argmax()], round(p.max().item(), 4), text)

Evaluation

Seed 42. Macro-F1 is computed over the labels each set contains. Target is macro-F1 over sin_Sinh/pli_Sinh/san_Sinh, and Replay is macro-F1 over the eight replay languages. The public benchmarks are the 11-language subsets (targets + replay) used in our evaluation. In those subsets, the target rows come from our held-out target test split.

Eval set	n	Accuracy	Macro-F1	Target	Replay
Target test	18,982	0.9995	0.9995	0.9995	–
FLORES+	27,078	0.9995	0.9995	0.9995	0.9994
WiLI-2018	22,469	0.9957	0.9829	0.9995	0.9758
CommonLID	89,874	0.9817	0.9723	0.9981	0.9626

Per-language F1 on the target test set: Sinhala 0.9996, Pali 0.9993, Sanskrit 0.9996.

Macro-F1 for the same base model trained three ways:

Eval set	Zero-shot (base)	Target-only fine-tune	This model (rehearsal)
Target test	0.0000	0.9996	0.9995
FLORES+	0.4256	0.4043	0.9995
WiLI-2018	0.3590	0.4419	0.9829
CommonLID	0.4314	0.3043	0.9723

Training

• Method: LoRA (r=16, alpha=32, dropout=0.1) on query and value. The classification head was trained in full after being widened with zero-initialised rows for the six new labels.
• Data: 231,979 training rows and 28,999 validation rows, made up of the Sinhala/Pali/Sanskrit corpus plus replay data for the eight languages above.
• Hyperparameters: AdamW, batch size 32, max length 128 tokens, up to 3 epochs. The learning rate was chosen from {2e-5, 5e-5, 1e-4} by validation macro-F1. The best run was lr=1e-4 at epoch 3 (validation macro-F1 0.9918, accuracy 0.9951). The full grid is in selection_log.csv and the chosen run in chosen.json.
• Input normalisation: text is NFC-normalised before training and evaluation. Apply the same normalisation for best results.

Limitations

• The model separates Sinhala, Pali and Sanskrit written in Sinhala script. Pali or Sanskrit in other scripts was not part of training, apart from Devanagari Sanskrit (san_Deva).
• The base model's 20 languages were not in the replay set, apart from English, Hindi, Arabic, French and German. Its accuracy on the other 15 was not re-measured after fine-tuning.
• Very short inputs (one to three words) are harder to classify, because the three languages share much of their vocabulary.