--- library_name: timm tags: - image-classification - food-classification - binary-classification - food-or-drink - nutrify - knowledge-distillation license: apache-2.0 datasets: - mrdbourke/DataComp-1B-food-and-drink-3M pipeline_tag: image-classification --- # Food / Not Food Classifier — nextvit_small_384 (v1) CoreML deployable — CNN+ViT hybrid, best for on-device inference (iPhone/iPad) A binary image classifier for detecting food and drink in images, trained via knowledge distillation from SigLIP2-so400m zero-shot labels on 3.1M images from [DataComp-1B-food-and-drink-3M](https://huggingface.co/datasets/mrdbourke/DataComp-1B-food-and-drink-3M). Part of a 3-model portfolio for the [Nutrify](https://nutrify.app) food tracking pipeline. ## Model Details | Field | Value | |-------|-------| | Architecture | `nextvit_small.bd_ssld_6m_in1k_384` | | Parameters | 30.7M | | Input size | 384x384px | | Labels | `food_or_drink` (0), `not_food_or_drink` (1) | | FoodVision accuracy | 0.9220 | | FoodVision F1 | 0.9472 | | Training val accuracy | 0.9300 (epoch 5/5) | | Throughput | 1156.0 img/s (batch 64, RTX 4090) | ## Quick Start ```python import timm import torch from PIL import Image from huggingface_hub import hf_hub_download from safetensors.torch import load_file # Load model model = timm.create_model("nextvit_small.bd_ssld_6m_in1k_384", pretrained=False, num_classes=2) weights = hf_hub_download("mrdbourke/food-not-food-classifier-nextvit-v1", "model.safetensors") model.load_state_dict(load_file(weights)) model.eval() # Prepare transform (use timm's built-in config) from timm.data import resolve_data_config, create_transform data_config = resolve_data_config(model.pretrained_cfg) data_config["input_size"] = (3, 384, 384) transform = create_transform(**data_config, is_training=False) # Classify image = Image.open("photo.jpg").convert("RGB") input_tensor = transform(image).unsqueeze(0) with torch.inference_mode(): logits = model(input_tensor) probs = torch.softmax(logits, dim=1) pred = logits.argmax(dim=1).item() labels = ["food_or_drink", "not_food_or_drink"] print(f"{labels[pred]}: {probs[0][pred]:.1%}") ``` ## All 3 Models — Comparison These models were trained together as part of the Nutrify food/not_food classifier portfolio. Pick the right one for your use case: | Model | Role | FV Accuracy | FV F1 | Params | Throughput | Repo | |-------|------|-------------|-------|--------|------------|------| | siglip2_base_256 | Highest accuracy | 91.3% | 94.1% | 92.9M | 2099.9 img/s | [link](https://huggingface.co/mrdbourke/food-not-food-classifier-siglip2-v1) | | csatv2_11m | Fastest throughput | 92.2% | 94.7% | 10.7M | 4639.1 img/s | [link](https://huggingface.co/mrdbourke/food-not-food-classifier-csatv2-v1) | | nextvit_small_384 | CoreML deployable | 92.2% | 94.7% | 30.7M | 1156.0 img/s | [link](https://huggingface.co/mrdbourke/food-not-food-classifier-nextvit-v1) | ## Evaluation — FoodVision Test Set Evaluated on 153,911 human-labeled images from the Nutrify FoodVision dataset (118K food + 35K not_food). This is an out-of-distribution test — the model was trained on DataComp-1B web images, not FoodVision images. | Metric | Value | |--------|-------| | Accuracy | 0.9220 | | F1 | 0.9472 | | Precision | 0.9903 | | Recall | 0.9078 | | Total samples | 153,911 | | Correct | 141,899 | | Wrong | 12,012 | ## Training ### Data - **Source:** [mrdbourke/DataComp-1B-food-and-drink-3M](https://huggingface.co/datasets/mrdbourke/DataComp-1B-food-and-drink-3M) — 3.1M images from Recap-DataComp-1B - **Training set:** 2,952,644 images (all quality tiers) - **Validation set:** 155,403 images - **Labels:** Binary (food_or_drink vs not_food_or_drink) ### Distillation - **Teacher:** [google/siglip2-so400m-patch16-512](https://huggingface.co/google/siglip2-so400m-patch16-512) (878M params, zero-shot) - **Loss:** Hybrid KL divergence: alpha * soft_KL + (1-alpha) * hard_CE - **Alpha:** 0.7 | **Temperature:** 3.0 - **Backbone LR:** 0.0001 * 0.1 (differential learning rate after unfreeze) - **Epochs:** 5 | **Best epoch:** 5 ### Augmentations (torchvision.transforms.v2) - RandomResizedCrop (scale 0.6-1.0) — food can be a small part of a scene - RandomHorizontalFlip + RandomVerticalFlip — orientation robustness - RandomRotation (15 deg) — tilted phone shots - RandomPerspective (0.2, p=0.3) — angled views - ColorJitter (B=0.4, C=0.4, S=0.3, H=0.05) — restaurant lighting variation - GaussianBlur (p=0.2) — camera shake - RandomGrayscale (p=0.02) — B&W web images - RandomErasing (p=0.1) — partial occlusion ### Pipeline context This model is part of the Nutrify VLM pipeline — a cascading filter system for building a food/drink image dataset from billion-scale web crawls: 1. **Text classification:** 1B captions → 106M food/drink rows (ettin-150m) 2. **Structured extraction:** FoodExtract-v2 on 106M rows (Gemma 3 270M) 3. **Image download:** 3.1M images from filtered URLs 4. **SigLIP2 zero-shot:** 92-prompt classification + embeddings (teacher labels) 5. **This model:** Fast binary classifier for scale-up to 50M+ images ## Throughput Benchmarks (RTX 4090) | Batch Size | img/s | VRAM (MB) | |-----------|-------|----------| | 64 | 1156.0 | 10912.5 | | 128 | 1110.6 | 2451.3 | | 256 | 1096.7 | 4376.5 | | 512 | 1092.4 | 8226.9 | Peak: **1156.0 img/s** at batch 64 ## Intended Use - **Primary:** Fast filtering of food/drink images from large web-crawled datasets - **Secondary:** Binary food detection in apps (food tracking, dietary logging) - **Not for:** Fine-grained food classification (use a multi-class model), nutrition estimation ## Limitations - Binary only — does not distinguish food types, cuisines, or specific items - Trained on web images — may underperform on unusual angles, lighting, or cultural foods underrepresented in DataComp-1B - Confidence scores are compressed due to distillation temperature (T=3) — use relative ranking, not absolute thresholds - v1 model — trained on DataComp only, v2 will include human-verified FoodVision training data ## Related Resources - **Dataset:** [mrdbourke/DataComp-1B-food-and-drink-3M](https://huggingface.co/datasets/mrdbourke/DataComp-1B-food-and-drink-3M) - **Text classifier:** [mrdbourke/ettin-150m-food-or-drink-classifier](https://huggingface.co/mrdbourke/ettin-150m-food-or-drink-classifier) - **Food extraction:** [mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2](https://huggingface.co/mrdbourke/FoodExtract-gemma-3-270m-fine-tune-v2) - **Highest accuracy:** [mrdbourke/food-not-food-classifier-siglip2-v1](https://huggingface.co/mrdbourke/food-not-food-classifier-siglip2-v1) - **Fastest throughput:** [mrdbourke/food-not-food-classifier-csatv2-v1](https://huggingface.co/mrdbourke/food-not-food-classifier-csatv2-v1) - **App:** [nutrify.app](https://nutrify.app) ## Citation ```bibtex @misc{food-not-food-nextvit-small-384-v1, author = {Daniel Bourke}, title = {Food/Not Food Classifier — nextvit_small_384 v1}, year = {2026}, publisher = {HuggingFace}, url = {https://huggingface.co/mrdbourke/food-not-food-classifier-nextvit-v1} } ```