T.I.T.S. — 16M param text-to-image model, 3 epochs on COCO at 128px
Browse files- .gitattributes +5 -0
- README.md +135 -0
- dataset.py +64 -0
- diffusion.py +120 -0
- model.py +222 -0
- prepare_data.py +74 -0
- requirements.txt +5 -0
- run_training.sh +11 -0
- sample.py +73 -0
- samples/bird_attempt.png +3 -0
- samples/preview_epoch1.png +3 -0
- samples/preview_epoch2.png +3 -0
- samples/preview_epoch3.png +3 -0
- samples/preview_step2000.png +3 -0
- text_encoder.py +33 -0
- tits_epoch3.pt +3 -0
- train.py +172 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
samples/bird_attempt.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
samples/preview_epoch1.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
samples/preview_epoch2.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
samples/preview_epoch3.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
samples/preview_step2000.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,135 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
datasets:
|
| 4 |
+
- jxie/coco_captions
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
tags:
|
| 8 |
+
- text-to-image
|
| 9 |
+
- diffusion
|
| 10 |
+
- ddpm
|
| 11 |
+
- unet
|
| 12 |
+
- pytorch
|
| 13 |
+
- trained-from-scratch
|
| 14 |
+
- smol
|
| 15 |
+
pipeline_tag: text-to-image
|
| 16 |
+
library_name: pytorch
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# T.I.T.S. — Terrible Imagegen TranSformer
|
| 20 |
+
|
| 21 |
+
A 16M-parameter text-to-image diffusion model that draws **excellent buses**, **passable pizza**,
|
| 22 |
+
and **no birds whatsoever**. Trained from scratch in 7.5 hours on one RTX 4060 Ti, for roughly
|
| 23 |
+
the electricity cost of a load of laundry.
|
| 24 |
+
|
| 25 |
+
It is bad on purpose, honestly. The math is textbook DDPM, nothing was sabotaged, and no corners
|
| 26 |
+
were cut except the one that matters: it has 16 million parameters, which in 2026 is a rounding
|
| 27 |
+
error. Stable Diffusion has 860M. Flux has 12B. T.I.T.S. has slightly fewer parameters than the
|
| 28 |
+
number of pixels in the images it is trying to make.
|
| 29 |
+
|
| 30 |
+
Also, despite the name, **it is not a transformer**. It's a UNet with a single attention layer
|
| 31 |
+
bolted to the bottom. The acronym came first. The architecture was never consulted and has no
|
| 32 |
+
comment.
|
| 33 |
+
|
| 34 |
+

|
| 35 |
+
|
| 36 |
+
*Final samples. Rows: "a motorcycle parked on the street", "a dog on a couch", "a plate of pizza",
|
| 37 |
+
"a red bus on a city street". Four samples each, guidance 3.0. Note that row 4 is doing great and
|
| 38 |
+
row 2 is a war crime.*
|
| 39 |
+
|
| 40 |
+
## What it can and cannot do
|
| 41 |
+
|
| 42 |
+
| Prompt | Result | Grade |
|
| 43 |
+
|---|---|---|
|
| 44 |
+
| Buses, streets, vehicles | Windows. Wheels. A red front end. An actual bus. | A- |
|
| 45 |
+
| Pizza, plates of food | Melted cheese, red toppings, crust. You would eat it. | B |
|
| 46 |
+
| Motorcycles | Two wheels, handlebars, roughly the correct number of each | C+ |
|
| 47 |
+
| Dogs, cats, people | Fleshy blobs arranged on furniture | F |
|
| 48 |
+
| Birds | Immaculate foliage. Zero birds. | F |
|
| 49 |
+
| Dragons, castles, anime | Abstract expressionism (unintentional) | N/A |
|
| 50 |
+
|
| 51 |
+
The pattern is **boxes yes, animals no**. A bus is a rectangle with windows in it. A dog is a
|
| 52 |
+
deformable mammal photographed from four hundred angles in four hundred lighting conditions.
|
| 53 |
+
Sixteen million parameters can afford exactly one of those.
|
| 54 |
+
|
| 55 |
+

|
| 56 |
+
|
| 57 |
+
*"a blue tit on a branch" at guidance 6. It rendered the branch, the leaves, the dappled light,
|
| 58 |
+
and the general feeling of being in a garden. Then it clocked off for the day.*
|
| 59 |
+
|
| 60 |
+
## Specs
|
| 61 |
+
|
| 62 |
+
| | |
|
| 63 |
+
|---|---|
|
| 64 |
+
| Parameters | 16.10M (image model) + frozen CLIP text encoder |
|
| 65 |
+
| Architecture | TinyUNet — `base_ch=64`, mults `(1,1,2,4)`, 2 res blocks/stage, one lonely self-attention layer at 16×16 |
|
| 66 |
+
| Resolution | 128×128, pixel space (no VAE, no latents, no shortcuts) |
|
| 67 |
+
| Text conditioning | Frozen CLIP ViT-B/32 pooled embedding → FiLM in every ResBlock |
|
| 68 |
+
| Diffusion | Linear-beta DDPM, 1000 steps, ε-prediction, ancestral sampling with x₀ clipping |
|
| 69 |
+
| Guidance | Classifier-free, 10% caption dropout during training. Default scale 3.0 |
|
| 70 |
+
| Training data | COCO Captions — 113,287 images, 566,747 captions, all of it from 2014 |
|
| 71 |
+
| Training | 3 epochs / 53,130 steps / batch 32 / AdamW @ 2e-4 / bf16 / EMA 0.9995 |
|
| 72 |
+
| Hardware | 1× RTX 4060 Ti (8GB), 7.5 hours, ~117W, shared with a browser |
|
| 73 |
+
| Final loss | 0.0209 |
|
| 74 |
+
| Inference | ~13s per image, which is 13s longer than you'll want to wait |
|
| 75 |
+
|
| 76 |
+
## Usage
|
| 77 |
+
|
| 78 |
+
```bash
|
| 79 |
+
pip install -r requirements.txt
|
| 80 |
+
python sample.py --checkpoint tits_epoch3.pt --prompt "a red bus on a city street" \
|
| 81 |
+
--num_images 4 --guidance 3.0 --upscale 2 --out bus.png
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
Ask it for a bus. It's really good at buses. Let it have this.
|
| 85 |
+
|
| 86 |
+
`--guidance` is the classifier-free guidance scale: 1 = off, 3 = good, 6 = saturated,
|
| 87 |
+
10 = radioactive. Raising it does not summon a bird. This was tested thoroughly and at some
|
| 88 |
+
personal cost.
|
| 89 |
+
|
| 90 |
+
The checkpoint ships `ema_state_dict` (used by default, much cleaner) and `model_state_dict`.
|
| 91 |
+
Optimizer state was stripped, so this is for inference, not for resuming. You do not want to
|
| 92 |
+
resume this.
|
| 93 |
+
|
| 94 |
+
## Training your own
|
| 95 |
+
|
| 96 |
+
```bash
|
| 97 |
+
python prepare_data.py --out_dir data --image_size 128 # streams COCO, saves at 128px
|
| 98 |
+
./run_training.sh # 3 epochs, ~7.5h on a 4060 Ti
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
`prepare_data.py` streams from HF and only ever writes the downscaled images, so the 19GB source
|
| 102 |
+
never touches your disk. Resumable, because it will be interrupted.
|
| 103 |
+
`run_training.sh --resume checkpoints/latest.pt` picks up from the last save (every 2000 steps).
|
| 104 |
+
|
| 105 |
+
## Things learned the hard way
|
| 106 |
+
|
| 107 |
+
- **Web-scraped alt-text data comes with watermarks.** Attempt one used CC3M. Roughly 1 in 10
|
| 108 |
+
images had an Alamy or Shutterstock watermark visible even at 64×64, and the model was well on
|
| 109 |
+
its way to concluding that a professional photograph has a grey bar across the middle. Switched
|
| 110 |
+
to COCO. COCO is clean, if permanently stuck in 2014.
|
| 111 |
+
- **Vanilla DDPM ancestral sampling drifts.** Without clipping predicted x₀ each step, a small
|
| 112 |
+
model's errors compound over 1000 steps until every image is the same shade of burnt orange.
|
| 113 |
+
One `.clamp(-1, 1)` fixes it. Attempt one looked like a house fire.
|
| 114 |
+
- **Classifier-free guidance is not optional.** Without caption dropout the prompt gently suggests
|
| 115 |
+
rather than instructs, and every output converges on the platonic average COCO photograph, which
|
| 116 |
+
is a beige room with something in it.
|
| 117 |
+
- **EMA goes through a blue phase.** Early in training the averaged weights still contain a few
|
| 118 |
+
percent of the random initialization, and everything comes out smurf-colored. It passes.
|
| 119 |
+
- **Raising the resolution made it dangerously competent.** The README originally warned that
|
| 120 |
+
bumping past 64×64 would quietly turn this into a real model. At 128×128 it started producing
|
| 121 |
+
buses you could identify. Consider this a cautionary tale.
|
| 122 |
+
|
| 123 |
+

|
| 124 |
+
|
| 125 |
+
*Step 2,000 of 53,130, back when it only made weather.*
|
| 126 |
+
|
| 127 |
+
## Limitations and intended use
|
| 128 |
+
|
| 129 |
+
For entertainment, education, and finding out exactly where 16M parameters runs out of road.
|
| 130 |
+
Not for production. Inherits every bias in COCO, and its worldview ends in 2014 — it has never
|
| 131 |
+
seen a modern phone, an e-scooter, or a Tesla, and it is happier that way.
|
| 132 |
+
|
| 133 |
+
## Citation
|
| 134 |
+
|
| 135 |
+
Please don't.
|
dataset.py
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Expects a directory of images plus a captions file mapping filename -> prompt.
|
| 3 |
+
|
| 4 |
+
captions format (JSONL, one object per line):
|
| 5 |
+
{"file": "000001.jpg", "caption": "a photo of a dog on a beach"}
|
| 6 |
+
|
| 7 |
+
Point --data_dir at a folder like:
|
| 8 |
+
data/
|
| 9 |
+
images/
|
| 10 |
+
000001.jpg
|
| 11 |
+
000002.jpg
|
| 12 |
+
captions.jsonl
|
| 13 |
+
|
| 14 |
+
A LAION-Aesthetics subset or Conceptual Captions slice both work fine — just
|
| 15 |
+
get everything into this filename/caption shape first.
|
| 16 |
+
"""
|
| 17 |
+
import json
|
| 18 |
+
import os
|
| 19 |
+
from typing import Optional
|
| 20 |
+
|
| 21 |
+
from PIL import Image
|
| 22 |
+
from torch.utils.data import Dataset
|
| 23 |
+
from torchvision import transforms
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
class CaptionedImageDataset(Dataset):
|
| 27 |
+
def __init__(self, data_dir: str, image_size: int = 64, captions_file: Optional[str] = None):
|
| 28 |
+
self.images_dir = os.path.join(data_dir, "images")
|
| 29 |
+
captions_file = captions_file or os.path.join(data_dir, "captions.jsonl")
|
| 30 |
+
|
| 31 |
+
self.entries = []
|
| 32 |
+
with open(captions_file, "r") as f:
|
| 33 |
+
for line in f:
|
| 34 |
+
line = line.strip()
|
| 35 |
+
if not line:
|
| 36 |
+
continue
|
| 37 |
+
obj = json.loads(line)
|
| 38 |
+
self.entries.append((obj["file"], obj["caption"]))
|
| 39 |
+
|
| 40 |
+
self.transform = transforms.Compose(
|
| 41 |
+
[
|
| 42 |
+
transforms.Resize(image_size),
|
| 43 |
+
transforms.CenterCrop(image_size),
|
| 44 |
+
transforms.ToTensor(),
|
| 45 |
+
transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]), # -> [-1, 1]
|
| 46 |
+
]
|
| 47 |
+
)
|
| 48 |
+
|
| 49 |
+
def __len__(self):
|
| 50 |
+
return len(self.entries)
|
| 51 |
+
|
| 52 |
+
def __getitem__(self, idx):
|
| 53 |
+
filename, caption = self.entries[idx]
|
| 54 |
+
img = Image.open(os.path.join(self.images_dir, filename)).convert("RGB")
|
| 55 |
+
img = self.transform(img)
|
| 56 |
+
return img, caption
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def collate_fn(batch):
|
| 60 |
+
imgs = [b[0] for b in batch]
|
| 61 |
+
captions = [b[1] for b in batch]
|
| 62 |
+
import torch
|
| 63 |
+
|
| 64 |
+
return torch.stack(imgs, dim=0), captions
|
diffusion.py
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
DDPM math: noise schedule, forward (q) process for training, reverse (p) sampling.
|
| 3 |
+
Standard linear-beta DDPM — nothing exotic, the "terrible" comes from the tiny
|
| 4 |
+
UNet and low training budget, not from a broken sampler.
|
| 5 |
+
"""
|
| 6 |
+
import torch
|
| 7 |
+
import torch.nn.functional as F
|
| 8 |
+
from tqdm import tqdm
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class GaussianDiffusion:
|
| 12 |
+
def __init__(
|
| 13 |
+
self,
|
| 14 |
+
timesteps: int = 1000,
|
| 15 |
+
beta_start: float = 1e-4,
|
| 16 |
+
beta_end: float = 0.02,
|
| 17 |
+
device: str = "cuda",
|
| 18 |
+
):
|
| 19 |
+
self.timesteps = timesteps
|
| 20 |
+
self.device = device
|
| 21 |
+
|
| 22 |
+
betas = torch.linspace(beta_start, beta_end, timesteps, device=device)
|
| 23 |
+
alphas = 1.0 - betas
|
| 24 |
+
alphas_cumprod = torch.cumprod(alphas, dim=0)
|
| 25 |
+
alphas_cumprod_prev = F.pad(alphas_cumprod[:-1], (1, 0), value=1.0)
|
| 26 |
+
|
| 27 |
+
self.betas = betas
|
| 28 |
+
self.alphas = alphas
|
| 29 |
+
self.alphas_cumprod = alphas_cumprod
|
| 30 |
+
self.alphas_cumprod_prev = alphas_cumprod_prev
|
| 31 |
+
self.sqrt_alphas_cumprod = torch.sqrt(alphas_cumprod)
|
| 32 |
+
self.sqrt_one_minus_alphas_cumprod = torch.sqrt(1.0 - alphas_cumprod)
|
| 33 |
+
self.posterior_variance = (
|
| 34 |
+
betas * (1.0 - alphas_cumprod_prev) / (1.0 - alphas_cumprod)
|
| 35 |
+
)
|
| 36 |
+
|
| 37 |
+
def _extract(self, a: torch.Tensor, t: torch.Tensor, shape) -> torch.Tensor:
|
| 38 |
+
out = a.gather(0, t)
|
| 39 |
+
return out.reshape(t.shape[0], *((1,) * (len(shape) - 1)))
|
| 40 |
+
|
| 41 |
+
def q_sample(self, x0: torch.Tensor, t: torch.Tensor, noise: torch.Tensor):
|
| 42 |
+
"""Forward diffusion: add noise to a clean image at timestep t."""
|
| 43 |
+
sqrt_ac = self._extract(self.sqrt_alphas_cumprod, t, x0.shape)
|
| 44 |
+
sqrt_1m_ac = self._extract(self.sqrt_one_minus_alphas_cumprod, t, x0.shape)
|
| 45 |
+
return sqrt_ac * x0 + sqrt_1m_ac * noise
|
| 46 |
+
|
| 47 |
+
def training_loss(self, model, x0: torch.Tensor, text_embed: torch.Tensor):
|
| 48 |
+
"""Standard epsilon-prediction MSE loss, uniform-random t per sample."""
|
| 49 |
+
b = x0.shape[0]
|
| 50 |
+
t = torch.randint(0, self.timesteps, (b,), device=x0.device).long()
|
| 51 |
+
noise = torch.randn_like(x0)
|
| 52 |
+
x_t = self.q_sample(x0, t, noise)
|
| 53 |
+
pred_noise = model(x_t, t, text_embed)
|
| 54 |
+
return F.mse_loss(pred_noise, noise)
|
| 55 |
+
|
| 56 |
+
@torch.no_grad()
|
| 57 |
+
def p_sample(
|
| 58 |
+
self,
|
| 59 |
+
model,
|
| 60 |
+
x_t: torch.Tensor,
|
| 61 |
+
t: int,
|
| 62 |
+
text_embed: torch.Tensor,
|
| 63 |
+
null_embed: torch.Tensor = None,
|
| 64 |
+
guidance_scale: float = 1.0,
|
| 65 |
+
):
|
| 66 |
+
"""Single reverse step: x_t -> x_{t-1}.
|
| 67 |
+
|
| 68 |
+
With null_embed and guidance_scale > 1, uses classifier-free guidance: run the
|
| 69 |
+
model with and without the prompt and push the prediction away from the
|
| 70 |
+
unconditional one. Needs a model trained with caption dropout.
|
| 71 |
+
"""
|
| 72 |
+
b = x_t.shape[0]
|
| 73 |
+
t_batch = torch.full((b,), t, device=x_t.device, dtype=torch.long)
|
| 74 |
+
if null_embed is not None and guidance_scale != 1.0:
|
| 75 |
+
null = null_embed.expand(b, -1)
|
| 76 |
+
eps = model(torch.cat([x_t, x_t]), torch.cat([t_batch, t_batch]), torch.cat([text_embed, null]))
|
| 77 |
+
eps_cond, eps_uncond = eps.chunk(2)
|
| 78 |
+
pred_noise = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
|
| 79 |
+
else:
|
| 80 |
+
pred_noise = model(x_t, t_batch, text_embed)
|
| 81 |
+
|
| 82 |
+
alpha_cumprod = self.alphas_cumprod[t]
|
| 83 |
+
alpha_cumprod_prev = self.alphas_cumprod_prev[t]
|
| 84 |
+
beta = self.betas[t]
|
| 85 |
+
|
| 86 |
+
# Predict x0 and clip it to the valid pixel range before computing the
|
| 87 |
+
# posterior mean. Without this, small errors from a tiny model compound over
|
| 88 |
+
# 1000 steps and images drift to oversaturated orange / pure black / pure white.
|
| 89 |
+
pred_x0 = (x_t - torch.sqrt(1 - alpha_cumprod) * pred_noise) / torch.sqrt(alpha_cumprod)
|
| 90 |
+
pred_x0 = pred_x0.clamp(-1, 1)
|
| 91 |
+
mean = (
|
| 92 |
+
torch.sqrt(alpha_cumprod_prev) * beta / (1 - alpha_cumprod) * pred_x0
|
| 93 |
+
+ torch.sqrt(1 - beta) * (1 - alpha_cumprod_prev) / (1 - alpha_cumprod) * x_t
|
| 94 |
+
)
|
| 95 |
+
if t == 0:
|
| 96 |
+
return mean
|
| 97 |
+
noise = torch.randn_like(x_t)
|
| 98 |
+
var = self.posterior_variance[t]
|
| 99 |
+
return mean + torch.sqrt(var) * noise
|
| 100 |
+
|
| 101 |
+
@torch.no_grad()
|
| 102 |
+
def sample(
|
| 103 |
+
self,
|
| 104 |
+
model,
|
| 105 |
+
text_embed: torch.Tensor,
|
| 106 |
+
image_size: int = 64,
|
| 107 |
+
channels: int = 3,
|
| 108 |
+
show_progress: bool = True,
|
| 109 |
+
null_embed: torch.Tensor = None,
|
| 110 |
+
guidance_scale: float = 1.0,
|
| 111 |
+
) -> torch.Tensor:
|
| 112 |
+
"""Full reverse process from pure noise -> image, DDPM ancestral sampling."""
|
| 113 |
+
b = text_embed.shape[0]
|
| 114 |
+
x = torch.randn(b, channels, image_size, image_size, device=self.device)
|
| 115 |
+
steps = range(self.timesteps - 1, -1, -1)
|
| 116 |
+
if show_progress:
|
| 117 |
+
steps = tqdm(steps, desc="sampling")
|
| 118 |
+
for t in steps:
|
| 119 |
+
x = self.p_sample(model, x, t, text_embed, null_embed, guidance_scale)
|
| 120 |
+
return x.clamp(-1, 1)
|
model.py
ADDED
|
@@ -0,0 +1,222 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
T.I.T.S. — Terrible Imagegen TranSformer
|
| 3 |
+
Tiny text-conditioned pixel-space UNet for DDPM. Deliberately small (~15M params)
|
| 4 |
+
so it trains fast and produces low-fidelity images by construction, not sabotage.
|
| 5 |
+
"""
|
| 6 |
+
import math
|
| 7 |
+
import torch
|
| 8 |
+
import torch.nn as nn
|
| 9 |
+
import torch.nn.functional as F
|
| 10 |
+
from torch.utils.checkpoint import checkpoint
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def timestep_embedding(t: torch.Tensor, dim: int) -> torch.Tensor:
|
| 14 |
+
"""Sinusoidal timestep embeddings, standard DDPM-style."""
|
| 15 |
+
half = dim // 2
|
| 16 |
+
freqs = torch.exp(
|
| 17 |
+
-math.log(10000) * torch.arange(half, device=t.device, dtype=torch.float32) / half
|
| 18 |
+
)
|
| 19 |
+
args = t.float()[:, None] * freqs[None]
|
| 20 |
+
emb = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)
|
| 21 |
+
if dim % 2:
|
| 22 |
+
emb = F.pad(emb, (0, 1))
|
| 23 |
+
return emb
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
class FiLM(nn.Module):
|
| 27 |
+
"""Feature-wise linear modulation — how we inject time + text conditioning
|
| 28 |
+
into each ResBlock without needing cross-attention."""
|
| 29 |
+
|
| 30 |
+
def __init__(self, cond_dim: int, channels: int):
|
| 31 |
+
super().__init__()
|
| 32 |
+
self.proj = nn.Linear(cond_dim, channels * 2)
|
| 33 |
+
|
| 34 |
+
def forward(self, x: torch.Tensor, cond: torch.Tensor) -> torch.Tensor:
|
| 35 |
+
scale, shift = self.proj(cond).chunk(2, dim=-1)
|
| 36 |
+
scale = scale[:, :, None, None]
|
| 37 |
+
shift = shift[:, :, None, None]
|
| 38 |
+
return x * (1 + scale) + shift
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class ResBlock(nn.Module):
|
| 42 |
+
def __init__(self, in_ch: int, out_ch: int, cond_dim: int):
|
| 43 |
+
super().__init__()
|
| 44 |
+
self.norm1 = nn.GroupNorm(8, in_ch)
|
| 45 |
+
self.conv1 = nn.Conv2d(in_ch, out_ch, 3, padding=1)
|
| 46 |
+
self.film = FiLM(cond_dim, out_ch)
|
| 47 |
+
self.norm2 = nn.GroupNorm(8, out_ch)
|
| 48 |
+
self.conv2 = nn.Conv2d(out_ch, out_ch, 3, padding=1)
|
| 49 |
+
self.skip = nn.Conv2d(in_ch, out_ch, 1) if in_ch != out_ch else nn.Identity()
|
| 50 |
+
|
| 51 |
+
def forward(self, x: torch.Tensor, cond: torch.Tensor) -> torch.Tensor:
|
| 52 |
+
h = self.conv1(F.silu(self.norm1(x)))
|
| 53 |
+
h = self.film(h, cond)
|
| 54 |
+
h = self.conv2(F.silu(self.norm2(h)))
|
| 55 |
+
return h + self.skip(x)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
class SelfAttention(nn.Module):
|
| 59 |
+
"""Only used at the lowest spatial resolution to keep param count small."""
|
| 60 |
+
|
| 61 |
+
def __init__(self, channels: int, heads: int = 4):
|
| 62 |
+
super().__init__()
|
| 63 |
+
self.norm = nn.GroupNorm(8, channels)
|
| 64 |
+
self.heads = heads
|
| 65 |
+
self.qkv = nn.Conv2d(channels, channels * 3, 1)
|
| 66 |
+
self.out = nn.Conv2d(channels, channels, 1)
|
| 67 |
+
|
| 68 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 69 |
+
b, c, h, w = x.shape
|
| 70 |
+
qkv = self.qkv(self.norm(x)).reshape(b, 3, self.heads, c // self.heads, h * w)
|
| 71 |
+
q, k, v = qkv.transpose(-1, -2).unbind(1) # each (b, heads, h*w, head_dim)
|
| 72 |
+
# Same math as softmax(q k^T / sqrt(d)) v, via the memory-efficient kernel.
|
| 73 |
+
out = F.scaled_dot_product_attention(q, k, v)
|
| 74 |
+
out = out.transpose(-1, -2).reshape(b, c, h, w)
|
| 75 |
+
return x + self.out(out)
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
class Downsample(nn.Module):
|
| 79 |
+
def __init__(self, ch: int):
|
| 80 |
+
super().__init__()
|
| 81 |
+
self.op = nn.Conv2d(ch, ch, 3, stride=2, padding=1)
|
| 82 |
+
|
| 83 |
+
def forward(self, x):
|
| 84 |
+
return self.op(x)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
class Upsample(nn.Module):
|
| 88 |
+
def __init__(self, ch: int):
|
| 89 |
+
super().__init__()
|
| 90 |
+
self.op = nn.Conv2d(ch, ch, 3, padding=1)
|
| 91 |
+
|
| 92 |
+
def forward(self, x):
|
| 93 |
+
x = F.interpolate(x, scale_factor=2, mode="nearest")
|
| 94 |
+
return self.op(x)
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
class TinyUNet(nn.Module):
|
| 98 |
+
"""
|
| 99 |
+
~15M params at base_ch=64, channel_mults=(1,2,4). Input/output: 3x64x64.
|
| 100 |
+
Conditioning: sinusoidal timestep embedding + pooled CLIP text embedding,
|
| 101 |
+
summed and fed into every ResBlock via FiLM.
|
| 102 |
+
"""
|
| 103 |
+
|
| 104 |
+
def __init__(
|
| 105 |
+
self,
|
| 106 |
+
base_ch: int = 64,
|
| 107 |
+
channel_mults=(1, 2, 4),
|
| 108 |
+
num_res_blocks: int = 2,
|
| 109 |
+
text_embed_dim: int = 512,
|
| 110 |
+
time_embed_dim: int = 256,
|
| 111 |
+
in_ch: int = 3,
|
| 112 |
+
use_checkpoint: bool = False,
|
| 113 |
+
):
|
| 114 |
+
super().__init__()
|
| 115 |
+
# Recompute ResBlock activations in backward instead of storing them.
|
| 116 |
+
# ~30% slower per step, but it's what lets 128px fit on an 8GB card.
|
| 117 |
+
self.use_checkpoint = use_checkpoint
|
| 118 |
+
cond_dim = time_embed_dim
|
| 119 |
+
self.time_embed_dim = time_embed_dim
|
| 120 |
+
self.num_res_blocks = num_res_blocks
|
| 121 |
+
self.time_mlp = nn.Sequential(
|
| 122 |
+
nn.Linear(time_embed_dim, cond_dim),
|
| 123 |
+
nn.SiLU(),
|
| 124 |
+
nn.Linear(cond_dim, cond_dim),
|
| 125 |
+
)
|
| 126 |
+
self.text_proj = nn.Sequential(
|
| 127 |
+
nn.Linear(text_embed_dim, cond_dim),
|
| 128 |
+
nn.SiLU(),
|
| 129 |
+
nn.Linear(cond_dim, cond_dim),
|
| 130 |
+
)
|
| 131 |
+
|
| 132 |
+
self.in_conv = nn.Conv2d(in_ch, base_ch, 3, padding=1)
|
| 133 |
+
|
| 134 |
+
# ---- Down path ----
|
| 135 |
+
chs = [base_ch]
|
| 136 |
+
ch = base_ch
|
| 137 |
+
self.down_blocks = nn.ModuleList()
|
| 138 |
+
self.downsamples = nn.ModuleList()
|
| 139 |
+
for i, mult in enumerate(channel_mults):
|
| 140 |
+
out_ch = base_ch * mult
|
| 141 |
+
for _ in range(num_res_blocks):
|
| 142 |
+
self.down_blocks.append(ResBlock(ch, out_ch, cond_dim))
|
| 143 |
+
ch = out_ch
|
| 144 |
+
chs.append(ch)
|
| 145 |
+
if i != len(channel_mults) - 1:
|
| 146 |
+
self.downsamples.append(Downsample(ch))
|
| 147 |
+
chs.append(ch)
|
| 148 |
+
else:
|
| 149 |
+
self.downsamples.append(nn.Identity())
|
| 150 |
+
|
| 151 |
+
# ---- Bottleneck (with attention — lowest resolution, cheap) ----
|
| 152 |
+
self.mid_block1 = ResBlock(ch, ch, cond_dim)
|
| 153 |
+
self.mid_attn = SelfAttention(ch)
|
| 154 |
+
self.mid_block2 = ResBlock(ch, ch, cond_dim)
|
| 155 |
+
|
| 156 |
+
# ---- Up path ----
|
| 157 |
+
self.up_blocks = nn.ModuleList()
|
| 158 |
+
self.upsamples = nn.ModuleList()
|
| 159 |
+
for i, mult in reversed(list(enumerate(channel_mults))):
|
| 160 |
+
out_ch = base_ch * mult
|
| 161 |
+
for _ in range(num_res_blocks + 1):
|
| 162 |
+
skip_ch = chs.pop()
|
| 163 |
+
self.up_blocks.append(ResBlock(ch + skip_ch, out_ch, cond_dim))
|
| 164 |
+
ch = out_ch
|
| 165 |
+
if i != 0:
|
| 166 |
+
self.upsamples.append(Upsample(ch))
|
| 167 |
+
else:
|
| 168 |
+
self.upsamples.append(nn.Identity())
|
| 169 |
+
|
| 170 |
+
self.out_norm = nn.GroupNorm(8, ch)
|
| 171 |
+
self.out_conv = nn.Conv2d(ch, in_ch, 3, padding=1)
|
| 172 |
+
|
| 173 |
+
def _block(self, block, h, cond):
|
| 174 |
+
if self.use_checkpoint and self.training:
|
| 175 |
+
return checkpoint(block, h, cond, use_reentrant=False)
|
| 176 |
+
return block(h, cond)
|
| 177 |
+
|
| 178 |
+
def forward(
|
| 179 |
+
self, x: torch.Tensor, t: torch.Tensor, text_embed: torch.Tensor
|
| 180 |
+
) -> torch.Tensor:
|
| 181 |
+
cond = self.time_mlp(timestep_embedding(t, self.time_embed_dim))
|
| 182 |
+
cond = cond + self.text_proj(text_embed)
|
| 183 |
+
|
| 184 |
+
h = self.in_conv(x)
|
| 185 |
+
skips = [h]
|
| 186 |
+
|
| 187 |
+
idx = 0
|
| 188 |
+
for i in range(len(self.downsamples)):
|
| 189 |
+
for _ in range(self.num_res_blocks):
|
| 190 |
+
h = self._block(self.down_blocks[idx], h, cond)
|
| 191 |
+
skips.append(h)
|
| 192 |
+
idx += 1
|
| 193 |
+
h = self.downsamples[i](h)
|
| 194 |
+
if not isinstance(self.downsamples[i], nn.Identity):
|
| 195 |
+
skips.append(h)
|
| 196 |
+
|
| 197 |
+
h = self._block(self.mid_block1, h, cond)
|
| 198 |
+
h = self.mid_attn(h)
|
| 199 |
+
h = self._block(self.mid_block2, h, cond)
|
| 200 |
+
|
| 201 |
+
idx = 0
|
| 202 |
+
for i in range(len(self.upsamples)):
|
| 203 |
+
for _ in range(self.num_res_blocks + 1):
|
| 204 |
+
skip = skips.pop()
|
| 205 |
+
h = self._block(self.up_blocks[idx], torch.cat([h, skip], dim=1), cond)
|
| 206 |
+
idx += 1
|
| 207 |
+
h = self.upsamples[i](h)
|
| 208 |
+
|
| 209 |
+
h = self.out_conv(F.silu(self.out_norm(h)))
|
| 210 |
+
return h
|
| 211 |
+
|
| 212 |
+
|
| 213 |
+
if __name__ == "__main__":
|
| 214 |
+
# Quick param count / shape sanity check (run manually, needs torch installed).
|
| 215 |
+
net = TinyUNet()
|
| 216 |
+
n_params = sum(p.numel() for p in net.parameters())
|
| 217 |
+
print(f"TinyUNet params: {n_params / 1e6:.2f}M")
|
| 218 |
+
x = torch.randn(2, 3, 64, 64)
|
| 219 |
+
t = torch.randint(0, 1000, (2,))
|
| 220 |
+
txt = torch.randn(2, 512)
|
| 221 |
+
out = net(x, t, txt)
|
| 222 |
+
print("output shape:", out.shape)
|
prepare_data.py
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Streams COCO captions from Hugging Face and writes it straight to the
|
| 3 |
+
T.I.T.S. data layout at 64x64, so full-size images never hit the disk.
|
| 4 |
+
|
| 5 |
+
COCO over web-scraped sets (CC3M, LAION) because those are full of stock photos
|
| 6 |
+
with watermarks, and the model would happily learn to draw the watermark.
|
| 7 |
+
|
| 8 |
+
The source has one row per caption (~5 per image). Each image is saved once and
|
| 9 |
+
every caption gets its own line pointing at it, so the model sees the same
|
| 10 |
+
picture described several different ways.
|
| 11 |
+
|
| 12 |
+
Usage:
|
| 13 |
+
python prepare_data.py --out_dir data/
|
| 14 |
+
"""
|
| 15 |
+
import argparse
|
| 16 |
+
import io
|
| 17 |
+
import json
|
| 18 |
+
import os
|
| 19 |
+
|
| 20 |
+
from datasets import Image as HFImage, load_dataset
|
| 21 |
+
from PIL import Image
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def main():
|
| 25 |
+
p = argparse.ArgumentParser()
|
| 26 |
+
p.add_argument("--out_dir", type=str, default="data")
|
| 27 |
+
p.add_argument("--num_rows", type=int, default=None, help="caption rows to process (default: all)")
|
| 28 |
+
p.add_argument("--image_size", type=int, default=64)
|
| 29 |
+
p.add_argument("--dataset", type=str, default="jxie/coco_captions")
|
| 30 |
+
p.add_argument("--split", type=str, default="train")
|
| 31 |
+
args = p.parse_args()
|
| 32 |
+
|
| 33 |
+
images_dir = os.path.join(args.out_dir, "images")
|
| 34 |
+
os.makedirs(images_dir, exist_ok=True)
|
| 35 |
+
captions_path = os.path.join(args.out_dir, "captions.jsonl")
|
| 36 |
+
|
| 37 |
+
# Resume support: one line is written per source row, so line count == rows consumed.
|
| 38 |
+
done = 0
|
| 39 |
+
if os.path.exists(captions_path):
|
| 40 |
+
with open(captions_path) as f:
|
| 41 |
+
done = sum(1 for line in f if line.strip())
|
| 42 |
+
|
| 43 |
+
ds = load_dataset(args.dataset, split=args.split, streaming=True)
|
| 44 |
+
# Keep raw bytes so repeat rows of an already-saved image skip JPEG decoding.
|
| 45 |
+
ds = ds.cast_column("image", HFImage(decode=False))
|
| 46 |
+
if done:
|
| 47 |
+
ds = ds.skip(done)
|
| 48 |
+
print(f"resuming at row {done}")
|
| 49 |
+
|
| 50 |
+
s = args.image_size
|
| 51 |
+
n = done
|
| 52 |
+
with open(captions_path, "a") as f:
|
| 53 |
+
for ex in ds:
|
| 54 |
+
if args.num_rows is not None and n >= args.num_rows:
|
| 55 |
+
break
|
| 56 |
+
name = f"{ex['cocoid']:012d}.jpg"
|
| 57 |
+
path = os.path.join(images_dir, name)
|
| 58 |
+
if not os.path.exists(path):
|
| 59 |
+
img = Image.open(io.BytesIO(ex["image"]["bytes"])).convert("RGB")
|
| 60 |
+
w, h = img.size
|
| 61 |
+
side = min(w, h)
|
| 62 |
+
img = img.crop(((w - side) // 2, (h - side) // 2, (w + side) // 2, (h + side) // 2))
|
| 63 |
+
img = img.resize((s, s), Image.BICUBIC)
|
| 64 |
+
img.save(path, quality=95)
|
| 65 |
+
f.write(json.dumps({"file": name, "caption": ex["caption"].strip()}) + "\n")
|
| 66 |
+
n += 1
|
| 67 |
+
if n % 5000 == 0:
|
| 68 |
+
f.flush()
|
| 69 |
+
print(f"{n} rows, {len(os.listdir(images_dir))} images", flush=True)
|
| 70 |
+
print(f"done: {n} caption rows in {args.out_dir}")
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
if __name__ == "__main__":
|
| 74 |
+
main()
|
requirements.txt
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
torch>=2.1
|
| 2 |
+
torchvision>=0.16
|
| 3 |
+
transformers>=4.36
|
| 4 |
+
Pillow>=10.0
|
| 5 |
+
tqdm>=4.66
|
run_training.sh
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# Waits for prepare_data.py to finish, then trains T.I.T.S. at 128px.
|
| 3 |
+
# Resume after a crash/reboot: ./run_training.sh --resume checkpoints/latest.pt
|
| 4 |
+
cd "$(dirname "$0")"
|
| 5 |
+
until grep -q "^done:" prepare.log 2>/dev/null; do
|
| 6 |
+
if grep -q "Traceback" prepare.log 2>/dev/null; then echo "prepare_data failed, see prepare.log"; exit 1; fi
|
| 7 |
+
sleep 30
|
| 8 |
+
done
|
| 9 |
+
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True exec .venv/bin/python -u train.py \
|
| 10 |
+
--data_dir data --image_size 128 --channel_mults 1,1,2,4 --grad_checkpoint \
|
| 11 |
+
--batch_size 32 --epochs 3 "$@"
|
sample.py
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
T.I.T.S. inference — load a checkpoint, feed it a prompt, get back a
|
| 3 |
+
disappointing PNG.
|
| 4 |
+
|
| 5 |
+
Usage:
|
| 6 |
+
python sample.py --checkpoint checkpoints/tits_epoch3.pt --prompt "a dog on a beach" --out out.png
|
| 7 |
+
python sample.py --checkpoint checkpoints/tits_epoch3.pt --prompt "a castle" --num_images 4 --out grid.png
|
| 8 |
+
|
| 9 |
+
--guidance is the classifier-free guidance scale: 1 = off, ~3 = sticks to the
|
| 10 |
+
prompt, higher = more prompt-y but more fried.
|
| 11 |
+
"""
|
| 12 |
+
import argparse
|
| 13 |
+
|
| 14 |
+
import torch
|
| 15 |
+
import torch.nn.functional as F
|
| 16 |
+
from torchvision.utils import save_image
|
| 17 |
+
|
| 18 |
+
from model import TinyUNet
|
| 19 |
+
from diffusion import GaussianDiffusion
|
| 20 |
+
from text_encoder import FrozenCLIPTextEncoder
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def parse_args():
|
| 24 |
+
p = argparse.ArgumentParser()
|
| 25 |
+
p.add_argument("--checkpoint", type=str, required=True)
|
| 26 |
+
p.add_argument("--prompt", type=str, required=True)
|
| 27 |
+
p.add_argument("--num_images", type=int, default=1)
|
| 28 |
+
p.add_argument("--guidance", type=float, default=3.0)
|
| 29 |
+
p.add_argument("--upscale", type=int, default=1, help="nearest-neighbour display upscale, adds no detail")
|
| 30 |
+
p.add_argument("--seed", type=int, default=None)
|
| 31 |
+
p.add_argument("--out", type=str, default="sample.png")
|
| 32 |
+
p.add_argument("--device", type=str, default="cuda" if torch.cuda.is_available() else "cpu")
|
| 33 |
+
return p.parse_args()
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def main():
|
| 37 |
+
args = parse_args()
|
| 38 |
+
device = args.device
|
| 39 |
+
if args.seed is not None:
|
| 40 |
+
torch.manual_seed(args.seed)
|
| 41 |
+
|
| 42 |
+
ckpt = torch.load(args.checkpoint, map_location=device)
|
| 43 |
+
|
| 44 |
+
text_encoder = FrozenCLIPTextEncoder(device=device)
|
| 45 |
+
model = TinyUNet(
|
| 46 |
+
base_ch=ckpt["base_ch"],
|
| 47 |
+
channel_mults=tuple(ckpt.get("channel_mults", (1, 2, 4))),
|
| 48 |
+
text_embed_dim=text_encoder.embed_dim,
|
| 49 |
+
).to(device)
|
| 50 |
+
# Prefer EMA weights when the checkpoint has them.
|
| 51 |
+
model.load_state_dict(ckpt.get("ema_state_dict", ckpt["model_state_dict"]))
|
| 52 |
+
model.eval()
|
| 53 |
+
|
| 54 |
+
diffusion = GaussianDiffusion(timesteps=ckpt["timesteps"], device=device)
|
| 55 |
+
|
| 56 |
+
prompts = [args.prompt] * args.num_images
|
| 57 |
+
text_embed = text_encoder.encode(prompts)
|
| 58 |
+
null_embed = text_encoder.encode([""]) if args.guidance != 1.0 else None
|
| 59 |
+
|
| 60 |
+
imgs = diffusion.sample(
|
| 61 |
+
model, text_embed, image_size=ckpt["image_size"], channels=3,
|
| 62 |
+
null_embed=null_embed, guidance_scale=args.guidance,
|
| 63 |
+
)
|
| 64 |
+
imgs = (imgs + 1) / 2 # [-1,1] -> [0,1] for saving
|
| 65 |
+
if args.upscale > 1:
|
| 66 |
+
imgs = F.interpolate(imgs, scale_factor=args.upscale, mode="nearest")
|
| 67 |
+
|
| 68 |
+
save_image(imgs, args.out, nrow=int(args.num_images**0.5) or 1)
|
| 69 |
+
print(f"saved {args.num_images} terrible image(s) to {args.out}")
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
if __name__ == "__main__":
|
| 73 |
+
main()
|
samples/bird_attempt.png
ADDED
|
Git LFS Details
|
samples/preview_epoch1.png
ADDED
|
Git LFS Details
|
samples/preview_epoch2.png
ADDED
|
Git LFS Details
|
samples/preview_epoch3.png
ADDED
|
Git LFS Details
|
samples/preview_step2000.png
ADDED
|
Git LFS Details
|
text_encoder.py
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Frozen CLIP text encoder for prompt conditioning. We reuse a pretrained encoder
|
| 3 |
+
rather than training our own — the "terrible" in T.I.T.S. lives entirely in the
|
| 4 |
+
image model, not the text understanding, otherwise prompts would just be noise
|
| 5 |
+
and you'd never be able to tell if a bad image was a bad *rendering* of your
|
| 6 |
+
prompt or the model just not understanding the prompt at all.
|
| 7 |
+
"""
|
| 8 |
+
from typing import List
|
| 9 |
+
|
| 10 |
+
import torch
|
| 11 |
+
import torch.nn as nn
|
| 12 |
+
from transformers import CLIPTokenizer, CLIPTextModelWithProjection
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class FrozenCLIPTextEncoder(nn.Module):
|
| 16 |
+
def __init__(self, model_name: str = "openai/clip-vit-base-patch32", device: str = "cuda"):
|
| 17 |
+
super().__init__()
|
| 18 |
+
self.tokenizer = CLIPTokenizer.from_pretrained(model_name)
|
| 19 |
+
self.model = CLIPTextModelWithProjection.from_pretrained(model_name)
|
| 20 |
+
self.model.to(device)
|
| 21 |
+
self.model.eval()
|
| 22 |
+
for p in self.model.parameters():
|
| 23 |
+
p.requires_grad = False
|
| 24 |
+
self.device = device
|
| 25 |
+
self.embed_dim = self.model.config.projection_dim # 512 for base-patch32
|
| 26 |
+
|
| 27 |
+
@torch.no_grad()
|
| 28 |
+
def encode(self, prompts: List[str]) -> torch.Tensor:
|
| 29 |
+
tokens = self.tokenizer(
|
| 30 |
+
prompts, padding=True, truncation=True, max_length=77, return_tensors="pt"
|
| 31 |
+
).to(self.device)
|
| 32 |
+
out = self.model(**tokens)
|
| 33 |
+
return out.text_embeds # (batch, embed_dim), already pooled/projected
|
tits_epoch3.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8d0eb4ea1ad0e0fba1929c2a1526a91b9baa98bf7290449ee18931b9181a5b20
|
| 3 |
+
size 129026103
|
train.py
ADDED
|
@@ -0,0 +1,172 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
T.I.T.S. training script — single GPU, local hardware.
|
| 3 |
+
|
| 4 |
+
Usage:
|
| 5 |
+
python train.py --data_dir data/ --image_size 128 --channel_mults 1,1,2,4 --grad_checkpoint
|
| 6 |
+
|
| 7 |
+
Tuned for a 4060 Ti (8GB): at 128px use --grad_checkpoint and batch_size ~32.
|
| 8 |
+
At 64px, --channel_mults 1,2,4 and no checkpointing is fine.
|
| 9 |
+
"""
|
| 10 |
+
import argparse
|
| 11 |
+
import copy
|
| 12 |
+
import os
|
| 13 |
+
import random
|
| 14 |
+
|
| 15 |
+
import torch
|
| 16 |
+
import torch.nn.functional as F
|
| 17 |
+
from torch.utils.data import DataLoader
|
| 18 |
+
from torch.optim import AdamW
|
| 19 |
+
from torchvision.utils import save_image
|
| 20 |
+
|
| 21 |
+
from model import TinyUNet
|
| 22 |
+
from diffusion import GaussianDiffusion
|
| 23 |
+
from dataset import CaptionedImageDataset, collate_fn
|
| 24 |
+
from text_encoder import FrozenCLIPTextEncoder
|
| 25 |
+
|
| 26 |
+
PREVIEW_PROMPTS = [
|
| 27 |
+
"a motorcycle parked on the street",
|
| 28 |
+
"a dog on a couch",
|
| 29 |
+
"a plate of pizza",
|
| 30 |
+
"a red bus on a city street",
|
| 31 |
+
]
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def parse_args():
|
| 35 |
+
p = argparse.ArgumentParser()
|
| 36 |
+
p.add_argument("--data_dir", type=str, required=True)
|
| 37 |
+
p.add_argument("--image_size", type=int, default=128)
|
| 38 |
+
p.add_argument("--batch_size", type=int, default=32)
|
| 39 |
+
p.add_argument("--epochs", type=int, default=3)
|
| 40 |
+
p.add_argument("--lr", type=float, default=2e-4)
|
| 41 |
+
p.add_argument("--timesteps", type=int, default=1000)
|
| 42 |
+
p.add_argument("--base_ch", type=int, default=64)
|
| 43 |
+
p.add_argument("--channel_mults", type=str, default="1,1,2,4")
|
| 44 |
+
p.add_argument("--grad_checkpoint", action="store_true")
|
| 45 |
+
p.add_argument("--p_uncond", type=float, default=0.1, help="caption dropout rate, enables CFG at sampling time")
|
| 46 |
+
p.add_argument("--ema_decay", type=float, default=0.9995)
|
| 47 |
+
p.add_argument("--out_dir", type=str, default="checkpoints")
|
| 48 |
+
p.add_argument("--save_every", type=int, default=1)
|
| 49 |
+
p.add_argument("--latest_every", type=int, default=2000, help="steps between resumable latest.pt saves")
|
| 50 |
+
p.add_argument("--resume", type=str, default=None)
|
| 51 |
+
p.add_argument("--device", type=str, default="cuda" if torch.cuda.is_available() else "cpu")
|
| 52 |
+
p.add_argument("--num_workers", type=int, default=4)
|
| 53 |
+
return p.parse_args()
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
@torch.no_grad()
|
| 57 |
+
def save_preview(ema, diffusion, text_encoder, image_size, path, guidance_scale=3.0):
|
| 58 |
+
ema.eval()
|
| 59 |
+
emb = text_encoder.encode([p for p in PREVIEW_PROMPTS for _ in range(4)])
|
| 60 |
+
null = text_encoder.encode([""])
|
| 61 |
+
g = torch.random.get_rng_state(), torch.cuda.get_rng_state()
|
| 62 |
+
torch.manual_seed(0)
|
| 63 |
+
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
| 64 |
+
x = diffusion.sample(ema, emb, image_size=image_size, show_progress=False,
|
| 65 |
+
null_embed=null, guidance_scale=guidance_scale)
|
| 66 |
+
torch.random.set_rng_state(g[0]); torch.cuda.set_rng_state(g[1])
|
| 67 |
+
x = F.interpolate((x.float() + 1) / 2, scale_factor=2, mode="nearest")
|
| 68 |
+
save_image(x, path, nrow=4, padding=4, pad_value=1)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def main():
|
| 72 |
+
args = parse_args()
|
| 73 |
+
os.makedirs(args.out_dir, exist_ok=True)
|
| 74 |
+
device = args.device
|
| 75 |
+
channel_mults = tuple(int(c) for c in args.channel_mults.split(","))
|
| 76 |
+
|
| 77 |
+
text_encoder = FrozenCLIPTextEncoder(device=device)
|
| 78 |
+
|
| 79 |
+
model = TinyUNet(
|
| 80 |
+
base_ch=args.base_ch,
|
| 81 |
+
channel_mults=channel_mults,
|
| 82 |
+
text_embed_dim=text_encoder.embed_dim,
|
| 83 |
+
use_checkpoint=args.grad_checkpoint,
|
| 84 |
+
).to(device)
|
| 85 |
+
# EMA of the weights is what gets sampled from — much cleaner than the raw weights.
|
| 86 |
+
ema = copy.deepcopy(model).eval().requires_grad_(False)
|
| 87 |
+
n_params = sum(p.numel() for p in model.parameters())
|
| 88 |
+
print(f"TinyUNet params: {n_params / 1e6:.2f}M")
|
| 89 |
+
|
| 90 |
+
diffusion = GaussianDiffusion(timesteps=args.timesteps, device=device)
|
| 91 |
+
opt = AdamW(model.parameters(), lr=args.lr)
|
| 92 |
+
|
| 93 |
+
step, start_epoch, skip_batches = 0, 0, 0
|
| 94 |
+
if args.resume:
|
| 95 |
+
ck = torch.load(args.resume, map_location=device)
|
| 96 |
+
model.load_state_dict(ck["model_state_dict"])
|
| 97 |
+
ema.load_state_dict(ck["ema_state_dict"])
|
| 98 |
+
opt.load_state_dict(ck["opt_state_dict"])
|
| 99 |
+
step, start_epoch, skip_batches = ck["step"], ck["epoch"], ck["batch_in_epoch"]
|
| 100 |
+
print(f"resumed from {args.resume} at step {step}")
|
| 101 |
+
|
| 102 |
+
dataset = CaptionedImageDataset(args.data_dir, image_size=args.image_size)
|
| 103 |
+
loader = DataLoader(
|
| 104 |
+
dataset,
|
| 105 |
+
batch_size=args.batch_size,
|
| 106 |
+
shuffle=True,
|
| 107 |
+
num_workers=args.num_workers,
|
| 108 |
+
collate_fn=collate_fn,
|
| 109 |
+
drop_last=True,
|
| 110 |
+
pin_memory=True,
|
| 111 |
+
)
|
| 112 |
+
print(f"Dataset size: {len(dataset)} images, {len(loader)} batches/epoch")
|
| 113 |
+
|
| 114 |
+
def ckpt_dict(epoch, batch_in_epoch):
|
| 115 |
+
return {
|
| 116 |
+
"model_state_dict": model.state_dict(),
|
| 117 |
+
"ema_state_dict": ema.state_dict(),
|
| 118 |
+
"opt_state_dict": opt.state_dict(),
|
| 119 |
+
"epoch": epoch,
|
| 120 |
+
"batch_in_epoch": batch_in_epoch,
|
| 121 |
+
"step": step,
|
| 122 |
+
"base_ch": args.base_ch,
|
| 123 |
+
"channel_mults": channel_mults,
|
| 124 |
+
"image_size": args.image_size,
|
| 125 |
+
"timesteps": args.timesteps,
|
| 126 |
+
}
|
| 127 |
+
|
| 128 |
+
for epoch in range(start_epoch, args.epochs):
|
| 129 |
+
model.train()
|
| 130 |
+
running_loss, n_batches = 0.0, 0
|
| 131 |
+
for i, (imgs, captions) in enumerate(loader):
|
| 132 |
+
if epoch == start_epoch and i < skip_batches:
|
| 133 |
+
continue # resume mid-epoch (different shuffle, but same number of steps)
|
| 134 |
+
imgs = imgs.to(device, non_blocking=True)
|
| 135 |
+
# Caption dropout: the empty prompt is the "unconditional" input for CFG.
|
| 136 |
+
captions = ["" if random.random() < args.p_uncond else c for c in captions]
|
| 137 |
+
text_embed = text_encoder.encode(captions)
|
| 138 |
+
|
| 139 |
+
# bf16 autocast: roughly halves activation memory on an 8GB card, no GradScaler needed.
|
| 140 |
+
with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=device.startswith("cuda")):
|
| 141 |
+
loss = diffusion.training_loss(model, imgs, text_embed)
|
| 142 |
+
|
| 143 |
+
opt.zero_grad(set_to_none=True)
|
| 144 |
+
loss.backward()
|
| 145 |
+
opt.step()
|
| 146 |
+
|
| 147 |
+
with torch.no_grad():
|
| 148 |
+
for pe, pm in zip(ema.parameters(), model.parameters()):
|
| 149 |
+
pe.lerp_(pm, 1 - args.ema_decay)
|
| 150 |
+
|
| 151 |
+
running_loss += loss.item()
|
| 152 |
+
n_batches += 1
|
| 153 |
+
step += 1
|
| 154 |
+
if step % 50 == 0:
|
| 155 |
+
print(f"epoch {epoch} step {step} loss {loss.item():.4f}", flush=True)
|
| 156 |
+
if step % args.latest_every == 0:
|
| 157 |
+
torch.save(ckpt_dict(epoch, i + 1), os.path.join(args.out_dir, "latest.pt"))
|
| 158 |
+
|
| 159 |
+
print(f"== epoch {epoch} done, avg loss {running_loss / max(n_batches, 1):.4f} ==")
|
| 160 |
+
|
| 161 |
+
if (epoch + 1) % args.save_every == 0:
|
| 162 |
+
ckpt_path = os.path.join(args.out_dir, f"tits_epoch{epoch+1}.pt")
|
| 163 |
+
torch.save(ckpt_dict(epoch + 1, 0), ckpt_path)
|
| 164 |
+
torch.save(ckpt_dict(epoch + 1, 0), os.path.join(args.out_dir, "latest.pt"))
|
| 165 |
+
print(f"saved checkpoint: {ckpt_path}")
|
| 166 |
+
preview = os.path.join(args.out_dir, f"preview_epoch{epoch+1}.png")
|
| 167 |
+
save_preview(ema, diffusion, text_encoder, args.image_size, preview)
|
| 168 |
+
print(f"saved preview: {preview}", flush=True)
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
if __name__ == "__main__":
|
| 172 |
+
main()
|