Waqf-AI/arabic-berthash-v2

ArabicBertHash v2

ArabicBertHash v2 is an Arabic masked-language encoder with 49.4M parameters. It combines compact learned token representations with a six-layer transformer for masked-word prediction and text-feature extraction.

Each token uses an 8-dimensional learned code projected into a 512-dimensional hidden representation. This reduces the input embedding module to 516K parameters, while retaining a full-vocabulary MLM output head.

Model overview	Value
Language	Arabic
Primary task	Masked-language modeling
Parameters	49.41M total · 16.25M encoder
Layers / hidden size / attention heads	6 / 512 / 8
Vocabulary	64,001 tokens
Recommended input length	1,024 tokens
Weight format	FP32 safetensors, approximately 198 MB
License	Apache-2.0

Quick start

Tested with Python 3.13, PyTorch 2.13.0 and Transformers 5.14.1. Install the versions in requirements.txt with a PyTorch build suitable for your CPU or GPU. Authenticate with hf auth login to access the private repository.

The model uses custom Transformers code, so loading requires trust_remote_code=True. Review the included code and pin a repository revision for reproducible use.

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "Abdelkareem/arabic-berthash-v2"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()

text = "اللغة العربية هي لغة [MASK] الكريم"
batch = tokenizer(text, return_tensors="pt", truncation=True, max_length=1024)
mask = batch["input_ids"].eq(tokenizer.mask_token_id)
assert mask.sum().item() == 1

with torch.inference_mode():
    logits = model(**batch).logits[mask][0]
    probabilities, token_ids = logits.softmax(-1).topk(5)

for token_id, probability in zip(token_ids.tolist(), probabilities.tolist()):
    filled = batch["input_ids"][0].clone()
    filled[mask[0]] = token_id
    print(tokenizer.decode(filled, skip_special_tokens=True), probability)

Decode the completed sentence to inspect Arabic suffixes and spacing. Softmax probabilities describe token predictions; they are not calibrated factual-confidence scores. For GPU inference, move both the model and input tensors to the same device. A standalone example is available in example.py.

Extract text features

The encoder returns a 512-dimensional vector per token. This example produces a normalized sentence vector by mean-pooling non-special tokens:

import torch.nn.functional as F

batch = tokenizer(
    ["هذا مثال عربي.", "نص آخر للمقارنة."],
    padding=True, truncation=True, max_length=1024, return_tensors="pt",
)
with torch.inference_mode():
    hidden = model.bert(**batch).last_hidden_state
    valid = batch["attention_mask"].bool()
    special_ids = torch.tensor(tokenizer.all_special_ids, device=hidden.device)
    valid &= ~torch.isin(batch["input_ids"], special_ids)
    pooled = (hidden * valid.unsqueeze(-1)).sum(1)
    pooled /= valid.sum(1).clamp_min(1).unsqueeze(-1)
    vectors = F.normalize(pooled.float(), dim=-1)

For classification or retrieval, train and validate a task head or projection on your data. This checkpoint provides an MLM head; the pooling example does not include the learned projection used in the retrieval evaluation below.

Architecture

Component	Configuration
Transformer	6 encoder layers, hidden size 512
Attention	8 heads × 64 dimensions, bidirectional self-attention
Token representation	Learned 64,001 × 8 code table, RMSNorm, shared bias-free 8→512 projection
Token-type embeddings	2 × 512
Position encoding	RoPE, theta 10,000
Normalization	RMSNorm, including Q/K normalization
Feed-forward	SwiGLU, intermediate size 1,024
Training dropout	0.1 hidden and attention dropout
MLM head	BERT-style transform, normalization and untied vocabulary decoder

Each vocabulary token has its own learned code. The shared projection expands that code into the encoder's hidden dimension.

Parameter breakdown

Component	Parameters
Token-code table	512,008
Token-code module, including projection and normalization	516,112
Complete encoder	16,253,712
Complete MLM head	33,160,194
Complete model	49,413,906

The input embedding module is approximately 63.5× smaller than a dense 64,001×512 embedding table. This is a component-level reduction: the untied output head accounts for most of the complete model's parameters.

Pretraining used 1,024-token windows. The configuration allows 2,048 positions, but longer-context quality has not been evaluated.

Tokenizer

The included tokenizer has a 64K base vocabulary plus [MASK]. Use it directly on ordinary Arabic text; no external Arabic segmentation step is required.

Token	ID
<unk>	0
<s>	1
</s>	2
<pad>	3
[MASK]	64,000

The serialized tokenizer defines normalization and tokenization. Keep it paired with these weights when training or evaluating the model.

Training

Data

The training corpus combined Arabic Wikipedia and FineWeb-2, prepared as 51,492,343 tokenized examples. The recorded source-building configuration references:

• HuggingFaceFW/fineweb-2, Arabic-script subsets arb_Arab, ary_Arab, arz_Arab, apc_Arab and acm_Arab.
• wikimedia/wikipedia, Arabic 20231101.ar.

Exact source revisions, final mixture proportions and unique-document counts are not fully recorded. Training/evaluation overlap has not been exhaustively ruled out.

Recipe

Setting	Value
Objective	Masked-language modeling
Mask selection	15% of eligible tokens
Corruption	80% mask, 10% random token, 10% unchanged
Context length	1,024
Effective batch size	64: batch 16 × accumulation 4
Optimizer	AdamW, betas (0.9, 0.98)
Peak learning rate	1e-5
Schedule / warmup	Cosine / 10,000 updates
Weight decay / gradient clipping	0.01 / 0.5
Training precision	BF16
Completed / planned updates	1,420,000 / 1,609,135
Completed epochs	1.7649
Hardware	One RTX 5090

Training covered approximately 90.88M example passes. The batch/context calculation gives 93.06B nominal token positions, including repetition and padding where applicable; this is not a count of unique or attended content tokens.

Saved trainer-state fields and the training recipe are available in training_summary.json.

Evaluation

Luna-reviewed comparison

The comparison uses the same 100 examples per task and typed DSPy gpt-6-luna judgments. All scores are percentages.

Model	MLM top-1	MLM top-5	NLI judge agreement	Retrieval R@1	Retrieval R@5
ArabicBertHash v2	65	93	37	31	39
Arabic ModernBERT Unified	70	100	68	48	61
Arabic UltraBERT Small	61	93	65	44	61
Arabic UltraBERT Large	66	96	74	44	57
AraBERTv2	82	98	49	55	64
AraModernBERT	84	97	34	47	62
Arabic MiniBERT	31	79	35	46	63

How the tasks were measured

• MLM: one shared-tokenizer-eligible word is masked per question. Top-1 accepts the exact reference or a completion judged contextually acceptable; top-5 accepts at least one such completion. Uncertain judgments are excluded. Candidates are reviewed in their completed sentences with model identity and rank hidden. For this model, judge-acceptable-only scores were 62%/87%; adding exact-reference matches gives the reported 65%/93%.
• Classification: three-way natural language inference, using frozen CLS features and a linear probe trained for 400 updates on 4,096 pairs, with 512 development and 1,024 test pairs. The reviewed score measures agreement with Luna labels on 100 cases. It is not general classification accuracy.
• Retrieval: frozen mean-pooled features with a projection trained for 300 updates on filtered NLI-derived triplets. Evaluation uses 512 development queries, 512 test queries and a 1,024-sentence gallery; Luna reviews the retrieved top-five passages for 100 test queries. The released MLM weights do not include this evaluation projection.

Exact versus reviewed scores

On the paired 100-case subsets:

Task	Dataset-reference score	Luna-reviewed score
MLM top-1 / top-5	12% / 22%	65% / 93%
NLI	33% accuracy	37% judge agreement
Retrieval R@1 / R@5	33% / 45%	31% / 39%

Reviewed MLM includes plausible alternatives, while reviewed NLI and retrieval use the judge's labels/relevance judgments. These columns therefore measure different targets. In NLI, one judge label was uncertain and 24 confident labels disagreed with the dataset labels; automated review is not adjudicated ground truth.

On the full exact-reference diagnostic sets:

Task	Test size	Score
Shared single-token whole-word MLM	201 questions	Top-1 11.44%, top-5 23.88%
Frozen-feature NLI probe	1,024 pairs	Accuracy 33.20%
Frozen-feature retrieval probe	512 queries	R@1 27.15%, R@5 44.34%

Full-set scores and 100-case subset scores have different denominators. Machine-readable results are available in evaluation/summary.json, with protocol details in evaluation/protocol.md.

Evaluation limitations

These are small, previously exposed research diagnostics with automated judgments. The judge was checked on small synthetic Arabic fixtures, not independently validated by human annotators. Most training comparisons use one seed, so small differences should not be treated as reliable general rankings.

Models share evaluation examples and probe-training budgets, but differ in pretraining data, compute and task exposure. Some comparison models already received NLI/retrieval training. Retrieval uses NLI-derived sentences rather than a broad Arabic passage-search benchmark. MLM acceptance measures contextual plausibility and may allow a different meaning from the reference.

Intended use and limitations

The model supports research on compact Arabic encoders, masked-word completion and task-specific adaptation.

• Train and evaluate task heads for classification or retrieval before deployment.
• Test on representative Arabic domains and dialects; their coverage is uneven and not fully quantified.
• Independently verify factual and numerical outputs.
• Web/Wikipedia training data can contain errors, personal information, stereotypes and offensive material. Dedicated privacy, toxicity and fairness evaluations have not been completed.
• Speed, energy use and deployment efficiency have not been benchmarked. The smaller input embedding module does not establish whole-model efficiency over MiniBERT or other encoders.

Files and reproducibility

File	Contents
model.safetensors	FP32 model weights
config.json	Architecture settings and Transformers AutoClass mappings
modeling_berthash_v2.py, configuration_berthash_v2.py	Model and configuration implementation
tokenizer.json, tokenizer_config.json	Tokenizer and special-token settings
example.py, requirements.txt	Usage example and tested dependencies
training_summary.json	Training settings and saved state summary
evaluation/	Scores, protocol, case identifiers and judgment labels
release_manifest.json, verification.json, SHA256SUMS	Checksums and loading-verification results

Pin the repository revision when reporting experiments. The included evaluation artifacts contain scores, hashes and case identifiers; raw evaluation texts and complete probe-training data are not distributed here.

License and citation

Code and model weights are licensed under Apache-2.0. See LICENSE and NOTICE. Source texts and third-party components retain their respective licenses.

Developed by Karem Elkhateb, Hugging Face account Abdelkareem.

@misc{arabicberthashv2,
  author = {Elkhateb, Karem},
  title = {ArabicBertHash v2},
  year = {2026},
  howpublished = {Hugging Face model repository},
  url = {https://huggingface.co/Abdelkareem/arabic-berthash-v2},
  note = {Checkpoint 1420000; cite the repository revision used}
}