Sonam5/cow-behavior-dinov3-dora-r64

DINOv3 ViT-7B/16 + DoRA (all-linear, r = 64) for dairy cattle behavior classification

DoRA adapters and a classification head that adapt the 4-bit-quantized DINOv3 ViT-7B/16 backbone (6.7 billion parameters) to nine dairy cattle behaviors from 2,160 expert-verified images, trained on a single 16 GB GPU. From the paper Parameter-efficient fine-tuning of a billion-parameter vision model improves dairy cattle behavior classification under a 1:98 train-to-test ratio (arXiv:2603.17782); code, data splits and predictions: PEFT-Fine-tuning-cows; all models: PEFT4COWS collection.

Files

File	Content
best_lora_adapters/	DoRA adapters on all seven linear layers of the 40 transformer blocks (q, k, v, o, gate, up, down projections), rank 64, alpha 128 (179,732,480 parameters), saved at the best validation epoch
best_classifier.pth	state dict of the classification head: Linear 4096→1024, ReLU, Dropout 0.5, Linear 1024→512, ReLU, Dropout 0.25, Linear 512→9 (4,724,745 parameters)
labels.json	class index → behavior
inference.py	complete loading and prediction example

How to use

pip install torch transformers==4.57.1 peft==0.13.2 bitsandbytes==0.48.0 accelerate opencv-python-headless huggingface_hub
huggingface-cli login   # the DINOv3 backbone is gated: accept its licence first
python inference.py --repo Sonam5/cow-behavior-dinov3-dora-r64 crop1.jpg crop2.jpg

Input scaling matters. The adapters and head were trained and validated on crops of single animals resized to 224 × 224 pixels with RGB values in [0, 1] (the ImageNet normalization applied by the data pipeline was inverted before the backbone). Do not apply the DINOv3 image processor's ImageNet normalization: the model would then receive inputs on a different scale from the one it was trained on. inference.py reproduces the training preprocessing exactly (OpenCV bilinear resize, values divided by 255, fp16 autocast, 4-bit NF4 base with double quantization).

import torch, torch.nn as nn
from huggingface_hub import snapshot_download
from transformers import AutoModel, BitsAndBytesConfig
from peft import PeftModel, prepare_model_for_kbit_training

local = snapshot_download("Sonam5/cow-behavior-dinov3-dora-r64")
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16,
                         bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True)
base = AutoModel.from_pretrained("facebook/dinov3-vit7b16-pretrain-lvd1689m", quantization_config=bnb, device_map="auto")
base = prepare_model_for_kbit_training(base, use_gradient_checkpointing=False)
model = PeftModel.from_pretrained(base, f"{local}/best_lora_adapters").eval()
head = nn.Sequential(nn.Linear(4096, 1024), nn.ReLU(), nn.Dropout(0.5), nn.Linear(1024, 512), nn.ReLU(),
                     nn.Dropout(0.25), nn.Linear(512, 9))
head.load_state_dict(torch.load(f"{local}/best_classifier.pth", map_location="cpu"))
head = head.to(model.device).eval()
# x: float tensor (batch, 3, 224, 224) with values in [0, 1], on the GPU
with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
    logits = head(model(pixel_values=x).pooler_output.float())

Training data

Nine behaviors from two sources: six behaviors of lactating Holstein cows from MmCows (one free-stall pen, four synchronized cameras, per-cow labels each second) and three play behaviors of group-housed calves from PlayBehavior (four Dutch dairy farms; video, frames, annotations and calf tracks of the two farms that supplied 2,528 of the 2,819 calf crops are public under CC0: Calf-Play-Behavior-Dataset). For each behavior, 300 crops of single animals were verified by two annotators with veterinary training and split 240/60 into training and validation (2,160 / 540 crops). All other labeled crops of the two sources, 211,800 in total (98 times the training set), form the test set; their labels are the source annotations and were not visually verified. The split files are in data/splits.

Index	Behavior	Label in source dataset	Source
0	Drinking	Drinking	MmCows (lactating cows)
1	Eating head down	Feeding head down	MmCows (lactating cows)
2	Eating head up	Feeding head up	MmCows (lactating cows)
3	Lying	Lying	MmCows (lactating cows)
4	Standing	Standing	MmCows (lactating cows)
5	Walking	Walking	MmCows (lactating cows)
6	Frontal pushing	frontal_pushing	PlayBehavior (calves)
7	Gallop	gallop	PlayBehavior (calves)
8	Leap	leap	PlayBehavior (calves)

Training procedure

4-bit NormalFloat quantization of the backbone with double quantization and fp16 computation (bitsandbytes), DoRA adapters inserted with PEFT, prepare_model_for_kbit_training; AdamW (peak learning rate 1e-4, 3% linear warm-up, cosine decay to 10%, weight decay 0.01), label smoothing 0.1, batch size 4 with 8 gradient-accumulation steps, automatic mixed precision, gradient checkpointing, adapter dropout 0.05; on-the-fly augmentation (flips, rotation, shift-scale-rotate, perspective, colour, noise and blur, coarse dropout; 3 views per crop per epoch). Early stopping on validation accuracy (patience 5) ended training after 9 epochs; the checkpoint of epoch 4 is released. One NVIDIA Tesla V100-PCIE-16GB, end-to-end training time 10 h 59 min.

Evaluation

Best validation accuracy: 92.04% (540 verified crops, epoch 4). Test set: 211,800 crops (209,881 MmCows, 1,919 PlayBehavior); results of the evaluation reported in the paper:

Metric	Value
Accuracy	83.14% (Wilson 95% CI 82.98–83.30%)
Weighted F1	0.834
Macro-F1	0.670
Balanced accuracy	0.704
Accuracy on MmCows / PlayBehavior crops	83.27% / 68.99%

Behavior	Test crops	Precision	Recall	F1
Drinking	3,011	0.434	0.768	0.554
Eating head down	30,952	0.874	0.558	0.681
Eating head up	18,783	0.768	0.561	0.648
Lying	83,509	0.984	0.993	0.989
Standing	69,807	0.771	0.856	0.811
Walking	3,819	0.210	0.513	0.298
Frontal pushing	600	0.625	0.858	0.723
Gallop	575	0.776	0.614	0.685
Leap	744	0.674	0.613	0.642

Note on the test preprocessing. In the evaluation reported in the paper, the test crops were passed to the backbone ImageNet-normalized and without fp16 autocast, whereas the model was trained and validated (and inference.py runs it) on crops rescaled to [0, 1] under fp16 autocast. This model was not re-evaluated. For two other configurations, re-evaluation of a random sample of 2,000 test crops with the training-time preprocessing changed 2.8–4.0% of the predictions and accuracy by +0.35 (QLoRA, all-linear, r = 64) and +0.10 percentage points (DoRA, q_proj, r = 8). Details: preprocessing check.

The confidence interval assumes independent crops; consecutive frames and simultaneous camera views of the same animal are correlated, so it understates the uncertainty. The model was trained once (seed 42).

Limitations

Trained and tested on crops from the same pens, cameras and recording periods, so the test results measure in-domain performance rather than transfer to new farms. Each behavior comes from a single source (cow behaviors from MmCows, calf behaviors from PlayBehavior). Walking is hard to separate from standing in a single frame, and some labels cannot be verified from a single crop. Inference with the 6.7-billion-parameter backbone runs at about 3–7 crops per second on a V100 GPU, suited to batch analysis rather than real-time use.

Licence and terms of use

The weights in this repository are released under the MIT licence. Using them requires the DINOv3 backbone, which is distributed by Meta under the DINOv3 License (accept it on the backbone's model page), and the models were trained partly on MmCows, which is released for non-commercial use (CC BY-NC-SA 4.0). Check both terms before any commercial use.

Citation

@article{yang2026peftcows,
  title   = {Parameter-efficient fine-tuning of a billion-parameter vision model improves dairy cattle behavior classification under a 1:98 train-to-test ratio},
  author  = {Yang, Haiyu and Sharma, Sumit and Liu, Enhong and Hostens, Miel},
  journal = {arXiv preprint arXiv:2603.17782},
  year    = {2026}
}