Codeminute commited on
Commit
a4a2b68
·
verified ·
1 Parent(s): 015db08

T.I.T.S. — 16M param text-to-image model, 3 epochs on COCO at 128px

Browse files
.gitattributes CHANGED
@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ samples/bird_attempt.png filter=lfs diff=lfs merge=lfs -text
37
+ samples/preview_epoch1.png filter=lfs diff=lfs merge=lfs -text
38
+ samples/preview_epoch2.png filter=lfs diff=lfs merge=lfs -text
39
+ samples/preview_epoch3.png filter=lfs diff=lfs merge=lfs -text
40
+ samples/preview_step2000.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,135 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ datasets:
4
+ - jxie/coco_captions
5
+ language:
6
+ - en
7
+ tags:
8
+ - text-to-image
9
+ - diffusion
10
+ - ddpm
11
+ - unet
12
+ - pytorch
13
+ - trained-from-scratch
14
+ - smol
15
+ pipeline_tag: text-to-image
16
+ library_name: pytorch
17
+ ---
18
+
19
+ # T.I.T.S. — Terrible Imagegen TranSformer
20
+
21
+ A 16M-parameter text-to-image diffusion model that draws **excellent buses**, **passable pizza**,
22
+ and **no birds whatsoever**. Trained from scratch in 7.5 hours on one RTX 4060 Ti, for roughly
23
+ the electricity cost of a load of laundry.
24
+
25
+ It is bad on purpose, honestly. The math is textbook DDPM, nothing was sabotaged, and no corners
26
+ were cut except the one that matters: it has 16 million parameters, which in 2026 is a rounding
27
+ error. Stable Diffusion has 860M. Flux has 12B. T.I.T.S. has slightly fewer parameters than the
28
+ number of pixels in the images it is trying to make.
29
+
30
+ Also, despite the name, **it is not a transformer**. It's a UNet with a single attention layer
31
+ bolted to the bottom. The acronym came first. The architecture was never consulted and has no
32
+ comment.
33
+
34
+ ![samples](samples/preview_epoch3.png)
35
+
36
+ *Final samples. Rows: "a motorcycle parked on the street", "a dog on a couch", "a plate of pizza",
37
+ "a red bus on a city street". Four samples each, guidance 3.0. Note that row 4 is doing great and
38
+ row 2 is a war crime.*
39
+
40
+ ## What it can and cannot do
41
+
42
+ | Prompt | Result | Grade |
43
+ |---|---|---|
44
+ | Buses, streets, vehicles | Windows. Wheels. A red front end. An actual bus. | A- |
45
+ | Pizza, plates of food | Melted cheese, red toppings, crust. You would eat it. | B |
46
+ | Motorcycles | Two wheels, handlebars, roughly the correct number of each | C+ |
47
+ | Dogs, cats, people | Fleshy blobs arranged on furniture | F |
48
+ | Birds | Immaculate foliage. Zero birds. | F |
49
+ | Dragons, castles, anime | Abstract expressionism (unintentional) | N/A |
50
+
51
+ The pattern is **boxes yes, animals no**. A bus is a rectangle with windows in it. A dog is a
52
+ deformable mammal photographed from four hundred angles in four hundred lighting conditions.
53
+ Sixteen million parameters can afford exactly one of those.
54
+
55
+ ![bird](samples/bird_attempt.png)
56
+
57
+ *"a blue tit on a branch" at guidance 6. It rendered the branch, the leaves, the dappled light,
58
+ and the general feeling of being in a garden. Then it clocked off for the day.*
59
+
60
+ ## Specs
61
+
62
+ | | |
63
+ |---|---|
64
+ | Parameters | 16.10M (image model) + frozen CLIP text encoder |
65
+ | Architecture | TinyUNet — `base_ch=64`, mults `(1,1,2,4)`, 2 res blocks/stage, one lonely self-attention layer at 16×16 |
66
+ | Resolution | 128×128, pixel space (no VAE, no latents, no shortcuts) |
67
+ | Text conditioning | Frozen CLIP ViT-B/32 pooled embedding → FiLM in every ResBlock |
68
+ | Diffusion | Linear-beta DDPM, 1000 steps, ε-prediction, ancestral sampling with x₀ clipping |
69
+ | Guidance | Classifier-free, 10% caption dropout during training. Default scale 3.0 |
70
+ | Training data | COCO Captions — 113,287 images, 566,747 captions, all of it from 2014 |
71
+ | Training | 3 epochs / 53,130 steps / batch 32 / AdamW @ 2e-4 / bf16 / EMA 0.9995 |
72
+ | Hardware | 1× RTX 4060 Ti (8GB), 7.5 hours, ~117W, shared with a browser |
73
+ | Final loss | 0.0209 |
74
+ | Inference | ~13s per image, which is 13s longer than you'll want to wait |
75
+
76
+ ## Usage
77
+
78
+ ```bash
79
+ pip install -r requirements.txt
80
+ python sample.py --checkpoint tits_epoch3.pt --prompt "a red bus on a city street" \
81
+ --num_images 4 --guidance 3.0 --upscale 2 --out bus.png
82
+ ```
83
+
84
+ Ask it for a bus. It's really good at buses. Let it have this.
85
+
86
+ `--guidance` is the classifier-free guidance scale: 1 = off, 3 = good, 6 = saturated,
87
+ 10 = radioactive. Raising it does not summon a bird. This was tested thoroughly and at some
88
+ personal cost.
89
+
90
+ The checkpoint ships `ema_state_dict` (used by default, much cleaner) and `model_state_dict`.
91
+ Optimizer state was stripped, so this is for inference, not for resuming. You do not want to
92
+ resume this.
93
+
94
+ ## Training your own
95
+
96
+ ```bash
97
+ python prepare_data.py --out_dir data --image_size 128 # streams COCO, saves at 128px
98
+ ./run_training.sh # 3 epochs, ~7.5h on a 4060 Ti
99
+ ```
100
+
101
+ `prepare_data.py` streams from HF and only ever writes the downscaled images, so the 19GB source
102
+ never touches your disk. Resumable, because it will be interrupted.
103
+ `run_training.sh --resume checkpoints/latest.pt` picks up from the last save (every 2000 steps).
104
+
105
+ ## Things learned the hard way
106
+
107
+ - **Web-scraped alt-text data comes with watermarks.** Attempt one used CC3M. Roughly 1 in 10
108
+ images had an Alamy or Shutterstock watermark visible even at 64×64, and the model was well on
109
+ its way to concluding that a professional photograph has a grey bar across the middle. Switched
110
+ to COCO. COCO is clean, if permanently stuck in 2014.
111
+ - **Vanilla DDPM ancestral sampling drifts.** Without clipping predicted x₀ each step, a small
112
+ model's errors compound over 1000 steps until every image is the same shade of burnt orange.
113
+ One `.clamp(-1, 1)` fixes it. Attempt one looked like a house fire.
114
+ - **Classifier-free guidance is not optional.** Without caption dropout the prompt gently suggests
115
+ rather than instructs, and every output converges on the platonic average COCO photograph, which
116
+ is a beige room with something in it.
117
+ - **EMA goes through a blue phase.** Early in training the averaged weights still contain a few
118
+ percent of the random initialization, and everything comes out smurf-colored. It passes.
119
+ - **Raising the resolution made it dangerously competent.** The README originally warned that
120
+ bumping past 64×64 would quietly turn this into a real model. At 128×128 it started producing
121
+ buses you could identify. Consider this a cautionary tale.
122
+
123
+ ![evolution](samples/preview_step2000.png)
124
+
125
+ *Step 2,000 of 53,130, back when it only made weather.*
126
+
127
+ ## Limitations and intended use
128
+
129
+ For entertainment, education, and finding out exactly where 16M parameters runs out of road.
130
+ Not for production. Inherits every bias in COCO, and its worldview ends in 2014 — it has never
131
+ seen a modern phone, an e-scooter, or a Tesla, and it is happier that way.
132
+
133
+ ## Citation
134
+
135
+ Please don't.
dataset.py ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Expects a directory of images plus a captions file mapping filename -> prompt.
3
+
4
+ captions format (JSONL, one object per line):
5
+ {"file": "000001.jpg", "caption": "a photo of a dog on a beach"}
6
+
7
+ Point --data_dir at a folder like:
8
+ data/
9
+ images/
10
+ 000001.jpg
11
+ 000002.jpg
12
+ captions.jsonl
13
+
14
+ A LAION-Aesthetics subset or Conceptual Captions slice both work fine — just
15
+ get everything into this filename/caption shape first.
16
+ """
17
+ import json
18
+ import os
19
+ from typing import Optional
20
+
21
+ from PIL import Image
22
+ from torch.utils.data import Dataset
23
+ from torchvision import transforms
24
+
25
+
26
+ class CaptionedImageDataset(Dataset):
27
+ def __init__(self, data_dir: str, image_size: int = 64, captions_file: Optional[str] = None):
28
+ self.images_dir = os.path.join(data_dir, "images")
29
+ captions_file = captions_file or os.path.join(data_dir, "captions.jsonl")
30
+
31
+ self.entries = []
32
+ with open(captions_file, "r") as f:
33
+ for line in f:
34
+ line = line.strip()
35
+ if not line:
36
+ continue
37
+ obj = json.loads(line)
38
+ self.entries.append((obj["file"], obj["caption"]))
39
+
40
+ self.transform = transforms.Compose(
41
+ [
42
+ transforms.Resize(image_size),
43
+ transforms.CenterCrop(image_size),
44
+ transforms.ToTensor(),
45
+ transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]), # -> [-1, 1]
46
+ ]
47
+ )
48
+
49
+ def __len__(self):
50
+ return len(self.entries)
51
+
52
+ def __getitem__(self, idx):
53
+ filename, caption = self.entries[idx]
54
+ img = Image.open(os.path.join(self.images_dir, filename)).convert("RGB")
55
+ img = self.transform(img)
56
+ return img, caption
57
+
58
+
59
+ def collate_fn(batch):
60
+ imgs = [b[0] for b in batch]
61
+ captions = [b[1] for b in batch]
62
+ import torch
63
+
64
+ return torch.stack(imgs, dim=0), captions
diffusion.py ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ DDPM math: noise schedule, forward (q) process for training, reverse (p) sampling.
3
+ Standard linear-beta DDPM — nothing exotic, the "terrible" comes from the tiny
4
+ UNet and low training budget, not from a broken sampler.
5
+ """
6
+ import torch
7
+ import torch.nn.functional as F
8
+ from tqdm import tqdm
9
+
10
+
11
+ class GaussianDiffusion:
12
+ def __init__(
13
+ self,
14
+ timesteps: int = 1000,
15
+ beta_start: float = 1e-4,
16
+ beta_end: float = 0.02,
17
+ device: str = "cuda",
18
+ ):
19
+ self.timesteps = timesteps
20
+ self.device = device
21
+
22
+ betas = torch.linspace(beta_start, beta_end, timesteps, device=device)
23
+ alphas = 1.0 - betas
24
+ alphas_cumprod = torch.cumprod(alphas, dim=0)
25
+ alphas_cumprod_prev = F.pad(alphas_cumprod[:-1], (1, 0), value=1.0)
26
+
27
+ self.betas = betas
28
+ self.alphas = alphas
29
+ self.alphas_cumprod = alphas_cumprod
30
+ self.alphas_cumprod_prev = alphas_cumprod_prev
31
+ self.sqrt_alphas_cumprod = torch.sqrt(alphas_cumprod)
32
+ self.sqrt_one_minus_alphas_cumprod = torch.sqrt(1.0 - alphas_cumprod)
33
+ self.posterior_variance = (
34
+ betas * (1.0 - alphas_cumprod_prev) / (1.0 - alphas_cumprod)
35
+ )
36
+
37
+ def _extract(self, a: torch.Tensor, t: torch.Tensor, shape) -> torch.Tensor:
38
+ out = a.gather(0, t)
39
+ return out.reshape(t.shape[0], *((1,) * (len(shape) - 1)))
40
+
41
+ def q_sample(self, x0: torch.Tensor, t: torch.Tensor, noise: torch.Tensor):
42
+ """Forward diffusion: add noise to a clean image at timestep t."""
43
+ sqrt_ac = self._extract(self.sqrt_alphas_cumprod, t, x0.shape)
44
+ sqrt_1m_ac = self._extract(self.sqrt_one_minus_alphas_cumprod, t, x0.shape)
45
+ return sqrt_ac * x0 + sqrt_1m_ac * noise
46
+
47
+ def training_loss(self, model, x0: torch.Tensor, text_embed: torch.Tensor):
48
+ """Standard epsilon-prediction MSE loss, uniform-random t per sample."""
49
+ b = x0.shape[0]
50
+ t = torch.randint(0, self.timesteps, (b,), device=x0.device).long()
51
+ noise = torch.randn_like(x0)
52
+ x_t = self.q_sample(x0, t, noise)
53
+ pred_noise = model(x_t, t, text_embed)
54
+ return F.mse_loss(pred_noise, noise)
55
+
56
+ @torch.no_grad()
57
+ def p_sample(
58
+ self,
59
+ model,
60
+ x_t: torch.Tensor,
61
+ t: int,
62
+ text_embed: torch.Tensor,
63
+ null_embed: torch.Tensor = None,
64
+ guidance_scale: float = 1.0,
65
+ ):
66
+ """Single reverse step: x_t -> x_{t-1}.
67
+
68
+ With null_embed and guidance_scale > 1, uses classifier-free guidance: run the
69
+ model with and without the prompt and push the prediction away from the
70
+ unconditional one. Needs a model trained with caption dropout.
71
+ """
72
+ b = x_t.shape[0]
73
+ t_batch = torch.full((b,), t, device=x_t.device, dtype=torch.long)
74
+ if null_embed is not None and guidance_scale != 1.0:
75
+ null = null_embed.expand(b, -1)
76
+ eps = model(torch.cat([x_t, x_t]), torch.cat([t_batch, t_batch]), torch.cat([text_embed, null]))
77
+ eps_cond, eps_uncond = eps.chunk(2)
78
+ pred_noise = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
79
+ else:
80
+ pred_noise = model(x_t, t_batch, text_embed)
81
+
82
+ alpha_cumprod = self.alphas_cumprod[t]
83
+ alpha_cumprod_prev = self.alphas_cumprod_prev[t]
84
+ beta = self.betas[t]
85
+
86
+ # Predict x0 and clip it to the valid pixel range before computing the
87
+ # posterior mean. Without this, small errors from a tiny model compound over
88
+ # 1000 steps and images drift to oversaturated orange / pure black / pure white.
89
+ pred_x0 = (x_t - torch.sqrt(1 - alpha_cumprod) * pred_noise) / torch.sqrt(alpha_cumprod)
90
+ pred_x0 = pred_x0.clamp(-1, 1)
91
+ mean = (
92
+ torch.sqrt(alpha_cumprod_prev) * beta / (1 - alpha_cumprod) * pred_x0
93
+ + torch.sqrt(1 - beta) * (1 - alpha_cumprod_prev) / (1 - alpha_cumprod) * x_t
94
+ )
95
+ if t == 0:
96
+ return mean
97
+ noise = torch.randn_like(x_t)
98
+ var = self.posterior_variance[t]
99
+ return mean + torch.sqrt(var) * noise
100
+
101
+ @torch.no_grad()
102
+ def sample(
103
+ self,
104
+ model,
105
+ text_embed: torch.Tensor,
106
+ image_size: int = 64,
107
+ channels: int = 3,
108
+ show_progress: bool = True,
109
+ null_embed: torch.Tensor = None,
110
+ guidance_scale: float = 1.0,
111
+ ) -> torch.Tensor:
112
+ """Full reverse process from pure noise -> image, DDPM ancestral sampling."""
113
+ b = text_embed.shape[0]
114
+ x = torch.randn(b, channels, image_size, image_size, device=self.device)
115
+ steps = range(self.timesteps - 1, -1, -1)
116
+ if show_progress:
117
+ steps = tqdm(steps, desc="sampling")
118
+ for t in steps:
119
+ x = self.p_sample(model, x, t, text_embed, null_embed, guidance_scale)
120
+ return x.clamp(-1, 1)
model.py ADDED
@@ -0,0 +1,222 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ T.I.T.S. — Terrible Imagegen TranSformer
3
+ Tiny text-conditioned pixel-space UNet for DDPM. Deliberately small (~15M params)
4
+ so it trains fast and produces low-fidelity images by construction, not sabotage.
5
+ """
6
+ import math
7
+ import torch
8
+ import torch.nn as nn
9
+ import torch.nn.functional as F
10
+ from torch.utils.checkpoint import checkpoint
11
+
12
+
13
+ def timestep_embedding(t: torch.Tensor, dim: int) -> torch.Tensor:
14
+ """Sinusoidal timestep embeddings, standard DDPM-style."""
15
+ half = dim // 2
16
+ freqs = torch.exp(
17
+ -math.log(10000) * torch.arange(half, device=t.device, dtype=torch.float32) / half
18
+ )
19
+ args = t.float()[:, None] * freqs[None]
20
+ emb = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)
21
+ if dim % 2:
22
+ emb = F.pad(emb, (0, 1))
23
+ return emb
24
+
25
+
26
+ class FiLM(nn.Module):
27
+ """Feature-wise linear modulation — how we inject time + text conditioning
28
+ into each ResBlock without needing cross-attention."""
29
+
30
+ def __init__(self, cond_dim: int, channels: int):
31
+ super().__init__()
32
+ self.proj = nn.Linear(cond_dim, channels * 2)
33
+
34
+ def forward(self, x: torch.Tensor, cond: torch.Tensor) -> torch.Tensor:
35
+ scale, shift = self.proj(cond).chunk(2, dim=-1)
36
+ scale = scale[:, :, None, None]
37
+ shift = shift[:, :, None, None]
38
+ return x * (1 + scale) + shift
39
+
40
+
41
+ class ResBlock(nn.Module):
42
+ def __init__(self, in_ch: int, out_ch: int, cond_dim: int):
43
+ super().__init__()
44
+ self.norm1 = nn.GroupNorm(8, in_ch)
45
+ self.conv1 = nn.Conv2d(in_ch, out_ch, 3, padding=1)
46
+ self.film = FiLM(cond_dim, out_ch)
47
+ self.norm2 = nn.GroupNorm(8, out_ch)
48
+ self.conv2 = nn.Conv2d(out_ch, out_ch, 3, padding=1)
49
+ self.skip = nn.Conv2d(in_ch, out_ch, 1) if in_ch != out_ch else nn.Identity()
50
+
51
+ def forward(self, x: torch.Tensor, cond: torch.Tensor) -> torch.Tensor:
52
+ h = self.conv1(F.silu(self.norm1(x)))
53
+ h = self.film(h, cond)
54
+ h = self.conv2(F.silu(self.norm2(h)))
55
+ return h + self.skip(x)
56
+
57
+
58
+ class SelfAttention(nn.Module):
59
+ """Only used at the lowest spatial resolution to keep param count small."""
60
+
61
+ def __init__(self, channels: int, heads: int = 4):
62
+ super().__init__()
63
+ self.norm = nn.GroupNorm(8, channels)
64
+ self.heads = heads
65
+ self.qkv = nn.Conv2d(channels, channels * 3, 1)
66
+ self.out = nn.Conv2d(channels, channels, 1)
67
+
68
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
69
+ b, c, h, w = x.shape
70
+ qkv = self.qkv(self.norm(x)).reshape(b, 3, self.heads, c // self.heads, h * w)
71
+ q, k, v = qkv.transpose(-1, -2).unbind(1) # each (b, heads, h*w, head_dim)
72
+ # Same math as softmax(q k^T / sqrt(d)) v, via the memory-efficient kernel.
73
+ out = F.scaled_dot_product_attention(q, k, v)
74
+ out = out.transpose(-1, -2).reshape(b, c, h, w)
75
+ return x + self.out(out)
76
+
77
+
78
+ class Downsample(nn.Module):
79
+ def __init__(self, ch: int):
80
+ super().__init__()
81
+ self.op = nn.Conv2d(ch, ch, 3, stride=2, padding=1)
82
+
83
+ def forward(self, x):
84
+ return self.op(x)
85
+
86
+
87
+ class Upsample(nn.Module):
88
+ def __init__(self, ch: int):
89
+ super().__init__()
90
+ self.op = nn.Conv2d(ch, ch, 3, padding=1)
91
+
92
+ def forward(self, x):
93
+ x = F.interpolate(x, scale_factor=2, mode="nearest")
94
+ return self.op(x)
95
+
96
+
97
+ class TinyUNet(nn.Module):
98
+ """
99
+ ~15M params at base_ch=64, channel_mults=(1,2,4). Input/output: 3x64x64.
100
+ Conditioning: sinusoidal timestep embedding + pooled CLIP text embedding,
101
+ summed and fed into every ResBlock via FiLM.
102
+ """
103
+
104
+ def __init__(
105
+ self,
106
+ base_ch: int = 64,
107
+ channel_mults=(1, 2, 4),
108
+ num_res_blocks: int = 2,
109
+ text_embed_dim: int = 512,
110
+ time_embed_dim: int = 256,
111
+ in_ch: int = 3,
112
+ use_checkpoint: bool = False,
113
+ ):
114
+ super().__init__()
115
+ # Recompute ResBlock activations in backward instead of storing them.
116
+ # ~30% slower per step, but it's what lets 128px fit on an 8GB card.
117
+ self.use_checkpoint = use_checkpoint
118
+ cond_dim = time_embed_dim
119
+ self.time_embed_dim = time_embed_dim
120
+ self.num_res_blocks = num_res_blocks
121
+ self.time_mlp = nn.Sequential(
122
+ nn.Linear(time_embed_dim, cond_dim),
123
+ nn.SiLU(),
124
+ nn.Linear(cond_dim, cond_dim),
125
+ )
126
+ self.text_proj = nn.Sequential(
127
+ nn.Linear(text_embed_dim, cond_dim),
128
+ nn.SiLU(),
129
+ nn.Linear(cond_dim, cond_dim),
130
+ )
131
+
132
+ self.in_conv = nn.Conv2d(in_ch, base_ch, 3, padding=1)
133
+
134
+ # ---- Down path ----
135
+ chs = [base_ch]
136
+ ch = base_ch
137
+ self.down_blocks = nn.ModuleList()
138
+ self.downsamples = nn.ModuleList()
139
+ for i, mult in enumerate(channel_mults):
140
+ out_ch = base_ch * mult
141
+ for _ in range(num_res_blocks):
142
+ self.down_blocks.append(ResBlock(ch, out_ch, cond_dim))
143
+ ch = out_ch
144
+ chs.append(ch)
145
+ if i != len(channel_mults) - 1:
146
+ self.downsamples.append(Downsample(ch))
147
+ chs.append(ch)
148
+ else:
149
+ self.downsamples.append(nn.Identity())
150
+
151
+ # ---- Bottleneck (with attention — lowest resolution, cheap) ----
152
+ self.mid_block1 = ResBlock(ch, ch, cond_dim)
153
+ self.mid_attn = SelfAttention(ch)
154
+ self.mid_block2 = ResBlock(ch, ch, cond_dim)
155
+
156
+ # ---- Up path ----
157
+ self.up_blocks = nn.ModuleList()
158
+ self.upsamples = nn.ModuleList()
159
+ for i, mult in reversed(list(enumerate(channel_mults))):
160
+ out_ch = base_ch * mult
161
+ for _ in range(num_res_blocks + 1):
162
+ skip_ch = chs.pop()
163
+ self.up_blocks.append(ResBlock(ch + skip_ch, out_ch, cond_dim))
164
+ ch = out_ch
165
+ if i != 0:
166
+ self.upsamples.append(Upsample(ch))
167
+ else:
168
+ self.upsamples.append(nn.Identity())
169
+
170
+ self.out_norm = nn.GroupNorm(8, ch)
171
+ self.out_conv = nn.Conv2d(ch, in_ch, 3, padding=1)
172
+
173
+ def _block(self, block, h, cond):
174
+ if self.use_checkpoint and self.training:
175
+ return checkpoint(block, h, cond, use_reentrant=False)
176
+ return block(h, cond)
177
+
178
+ def forward(
179
+ self, x: torch.Tensor, t: torch.Tensor, text_embed: torch.Tensor
180
+ ) -> torch.Tensor:
181
+ cond = self.time_mlp(timestep_embedding(t, self.time_embed_dim))
182
+ cond = cond + self.text_proj(text_embed)
183
+
184
+ h = self.in_conv(x)
185
+ skips = [h]
186
+
187
+ idx = 0
188
+ for i in range(len(self.downsamples)):
189
+ for _ in range(self.num_res_blocks):
190
+ h = self._block(self.down_blocks[idx], h, cond)
191
+ skips.append(h)
192
+ idx += 1
193
+ h = self.downsamples[i](h)
194
+ if not isinstance(self.downsamples[i], nn.Identity):
195
+ skips.append(h)
196
+
197
+ h = self._block(self.mid_block1, h, cond)
198
+ h = self.mid_attn(h)
199
+ h = self._block(self.mid_block2, h, cond)
200
+
201
+ idx = 0
202
+ for i in range(len(self.upsamples)):
203
+ for _ in range(self.num_res_blocks + 1):
204
+ skip = skips.pop()
205
+ h = self._block(self.up_blocks[idx], torch.cat([h, skip], dim=1), cond)
206
+ idx += 1
207
+ h = self.upsamples[i](h)
208
+
209
+ h = self.out_conv(F.silu(self.out_norm(h)))
210
+ return h
211
+
212
+
213
+ if __name__ == "__main__":
214
+ # Quick param count / shape sanity check (run manually, needs torch installed).
215
+ net = TinyUNet()
216
+ n_params = sum(p.numel() for p in net.parameters())
217
+ print(f"TinyUNet params: {n_params / 1e6:.2f}M")
218
+ x = torch.randn(2, 3, 64, 64)
219
+ t = torch.randint(0, 1000, (2,))
220
+ txt = torch.randn(2, 512)
221
+ out = net(x, t, txt)
222
+ print("output shape:", out.shape)
prepare_data.py ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Streams COCO captions from Hugging Face and writes it straight to the
3
+ T.I.T.S. data layout at 64x64, so full-size images never hit the disk.
4
+
5
+ COCO over web-scraped sets (CC3M, LAION) because those are full of stock photos
6
+ with watermarks, and the model would happily learn to draw the watermark.
7
+
8
+ The source has one row per caption (~5 per image). Each image is saved once and
9
+ every caption gets its own line pointing at it, so the model sees the same
10
+ picture described several different ways.
11
+
12
+ Usage:
13
+ python prepare_data.py --out_dir data/
14
+ """
15
+ import argparse
16
+ import io
17
+ import json
18
+ import os
19
+
20
+ from datasets import Image as HFImage, load_dataset
21
+ from PIL import Image
22
+
23
+
24
+ def main():
25
+ p = argparse.ArgumentParser()
26
+ p.add_argument("--out_dir", type=str, default="data")
27
+ p.add_argument("--num_rows", type=int, default=None, help="caption rows to process (default: all)")
28
+ p.add_argument("--image_size", type=int, default=64)
29
+ p.add_argument("--dataset", type=str, default="jxie/coco_captions")
30
+ p.add_argument("--split", type=str, default="train")
31
+ args = p.parse_args()
32
+
33
+ images_dir = os.path.join(args.out_dir, "images")
34
+ os.makedirs(images_dir, exist_ok=True)
35
+ captions_path = os.path.join(args.out_dir, "captions.jsonl")
36
+
37
+ # Resume support: one line is written per source row, so line count == rows consumed.
38
+ done = 0
39
+ if os.path.exists(captions_path):
40
+ with open(captions_path) as f:
41
+ done = sum(1 for line in f if line.strip())
42
+
43
+ ds = load_dataset(args.dataset, split=args.split, streaming=True)
44
+ # Keep raw bytes so repeat rows of an already-saved image skip JPEG decoding.
45
+ ds = ds.cast_column("image", HFImage(decode=False))
46
+ if done:
47
+ ds = ds.skip(done)
48
+ print(f"resuming at row {done}")
49
+
50
+ s = args.image_size
51
+ n = done
52
+ with open(captions_path, "a") as f:
53
+ for ex in ds:
54
+ if args.num_rows is not None and n >= args.num_rows:
55
+ break
56
+ name = f"{ex['cocoid']:012d}.jpg"
57
+ path = os.path.join(images_dir, name)
58
+ if not os.path.exists(path):
59
+ img = Image.open(io.BytesIO(ex["image"]["bytes"])).convert("RGB")
60
+ w, h = img.size
61
+ side = min(w, h)
62
+ img = img.crop(((w - side) // 2, (h - side) // 2, (w + side) // 2, (h + side) // 2))
63
+ img = img.resize((s, s), Image.BICUBIC)
64
+ img.save(path, quality=95)
65
+ f.write(json.dumps({"file": name, "caption": ex["caption"].strip()}) + "\n")
66
+ n += 1
67
+ if n % 5000 == 0:
68
+ f.flush()
69
+ print(f"{n} rows, {len(os.listdir(images_dir))} images", flush=True)
70
+ print(f"done: {n} caption rows in {args.out_dir}")
71
+
72
+
73
+ if __name__ == "__main__":
74
+ main()
requirements.txt ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ torch>=2.1
2
+ torchvision>=0.16
3
+ transformers>=4.36
4
+ Pillow>=10.0
5
+ tqdm>=4.66
run_training.sh ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+ # Waits for prepare_data.py to finish, then trains T.I.T.S. at 128px.
3
+ # Resume after a crash/reboot: ./run_training.sh --resume checkpoints/latest.pt
4
+ cd "$(dirname "$0")"
5
+ until grep -q "^done:" prepare.log 2>/dev/null; do
6
+ if grep -q "Traceback" prepare.log 2>/dev/null; then echo "prepare_data failed, see prepare.log"; exit 1; fi
7
+ sleep 30
8
+ done
9
+ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True exec .venv/bin/python -u train.py \
10
+ --data_dir data --image_size 128 --channel_mults 1,1,2,4 --grad_checkpoint \
11
+ --batch_size 32 --epochs 3 "$@"
sample.py ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ T.I.T.S. inference — load a checkpoint, feed it a prompt, get back a
3
+ disappointing PNG.
4
+
5
+ Usage:
6
+ python sample.py --checkpoint checkpoints/tits_epoch3.pt --prompt "a dog on a beach" --out out.png
7
+ python sample.py --checkpoint checkpoints/tits_epoch3.pt --prompt "a castle" --num_images 4 --out grid.png
8
+
9
+ --guidance is the classifier-free guidance scale: 1 = off, ~3 = sticks to the
10
+ prompt, higher = more prompt-y but more fried.
11
+ """
12
+ import argparse
13
+
14
+ import torch
15
+ import torch.nn.functional as F
16
+ from torchvision.utils import save_image
17
+
18
+ from model import TinyUNet
19
+ from diffusion import GaussianDiffusion
20
+ from text_encoder import FrozenCLIPTextEncoder
21
+
22
+
23
+ def parse_args():
24
+ p = argparse.ArgumentParser()
25
+ p.add_argument("--checkpoint", type=str, required=True)
26
+ p.add_argument("--prompt", type=str, required=True)
27
+ p.add_argument("--num_images", type=int, default=1)
28
+ p.add_argument("--guidance", type=float, default=3.0)
29
+ p.add_argument("--upscale", type=int, default=1, help="nearest-neighbour display upscale, adds no detail")
30
+ p.add_argument("--seed", type=int, default=None)
31
+ p.add_argument("--out", type=str, default="sample.png")
32
+ p.add_argument("--device", type=str, default="cuda" if torch.cuda.is_available() else "cpu")
33
+ return p.parse_args()
34
+
35
+
36
+ def main():
37
+ args = parse_args()
38
+ device = args.device
39
+ if args.seed is not None:
40
+ torch.manual_seed(args.seed)
41
+
42
+ ckpt = torch.load(args.checkpoint, map_location=device)
43
+
44
+ text_encoder = FrozenCLIPTextEncoder(device=device)
45
+ model = TinyUNet(
46
+ base_ch=ckpt["base_ch"],
47
+ channel_mults=tuple(ckpt.get("channel_mults", (1, 2, 4))),
48
+ text_embed_dim=text_encoder.embed_dim,
49
+ ).to(device)
50
+ # Prefer EMA weights when the checkpoint has them.
51
+ model.load_state_dict(ckpt.get("ema_state_dict", ckpt["model_state_dict"]))
52
+ model.eval()
53
+
54
+ diffusion = GaussianDiffusion(timesteps=ckpt["timesteps"], device=device)
55
+
56
+ prompts = [args.prompt] * args.num_images
57
+ text_embed = text_encoder.encode(prompts)
58
+ null_embed = text_encoder.encode([""]) if args.guidance != 1.0 else None
59
+
60
+ imgs = diffusion.sample(
61
+ model, text_embed, image_size=ckpt["image_size"], channels=3,
62
+ null_embed=null_embed, guidance_scale=args.guidance,
63
+ )
64
+ imgs = (imgs + 1) / 2 # [-1,1] -> [0,1] for saving
65
+ if args.upscale > 1:
66
+ imgs = F.interpolate(imgs, scale_factor=args.upscale, mode="nearest")
67
+
68
+ save_image(imgs, args.out, nrow=int(args.num_images**0.5) or 1)
69
+ print(f"saved {args.num_images} terrible image(s) to {args.out}")
70
+
71
+
72
+ if __name__ == "__main__":
73
+ main()
samples/bird_attempt.png ADDED

Git LFS Details

  • SHA256: 9e8d4c632910f1400d03119b31fe66d565f7818984dcf8c839dfc91159db7583
  • Pointer size: 131 Bytes
  • Size of remote file: 166 kB
samples/preview_epoch1.png ADDED

Git LFS Details

  • SHA256: efd02a70c44810b89dc540cbe9db852bbab60d980c2c2180718f12fe4fb2bdcb
  • Pointer size: 131 Bytes
  • Size of remote file: 666 kB
samples/preview_epoch2.png ADDED

Git LFS Details

  • SHA256: ff95ecc08aa3a39f403da1cf38b4e7047f5358a0c205538031975cedd93f172b
  • Pointer size: 131 Bytes
  • Size of remote file: 649 kB
samples/preview_epoch3.png ADDED

Git LFS Details

  • SHA256: dddadb80ab5c8a846ea34af590134c9bde3e9211b9d819043f38eecb69cc7239
  • Pointer size: 131 Bytes
  • Size of remote file: 638 kB
samples/preview_step2000.png ADDED

Git LFS Details

  • SHA256: 080e053c0743c56f3cba437803915d97ee4ced1c75d2de58df3fc832c9411b68
  • Pointer size: 131 Bytes
  • Size of remote file: 755 kB
text_encoder.py ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Frozen CLIP text encoder for prompt conditioning. We reuse a pretrained encoder
3
+ rather than training our own — the "terrible" in T.I.T.S. lives entirely in the
4
+ image model, not the text understanding, otherwise prompts would just be noise
5
+ and you'd never be able to tell if a bad image was a bad *rendering* of your
6
+ prompt or the model just not understanding the prompt at all.
7
+ """
8
+ from typing import List
9
+
10
+ import torch
11
+ import torch.nn as nn
12
+ from transformers import CLIPTokenizer, CLIPTextModelWithProjection
13
+
14
+
15
+ class FrozenCLIPTextEncoder(nn.Module):
16
+ def __init__(self, model_name: str = "openai/clip-vit-base-patch32", device: str = "cuda"):
17
+ super().__init__()
18
+ self.tokenizer = CLIPTokenizer.from_pretrained(model_name)
19
+ self.model = CLIPTextModelWithProjection.from_pretrained(model_name)
20
+ self.model.to(device)
21
+ self.model.eval()
22
+ for p in self.model.parameters():
23
+ p.requires_grad = False
24
+ self.device = device
25
+ self.embed_dim = self.model.config.projection_dim # 512 for base-patch32
26
+
27
+ @torch.no_grad()
28
+ def encode(self, prompts: List[str]) -> torch.Tensor:
29
+ tokens = self.tokenizer(
30
+ prompts, padding=True, truncation=True, max_length=77, return_tensors="pt"
31
+ ).to(self.device)
32
+ out = self.model(**tokens)
33
+ return out.text_embeds # (batch, embed_dim), already pooled/projected
tits_epoch3.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8d0eb4ea1ad0e0fba1929c2a1526a91b9baa98bf7290449ee18931b9181a5b20
3
+ size 129026103
train.py ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ T.I.T.S. training script — single GPU, local hardware.
3
+
4
+ Usage:
5
+ python train.py --data_dir data/ --image_size 128 --channel_mults 1,1,2,4 --grad_checkpoint
6
+
7
+ Tuned for a 4060 Ti (8GB): at 128px use --grad_checkpoint and batch_size ~32.
8
+ At 64px, --channel_mults 1,2,4 and no checkpointing is fine.
9
+ """
10
+ import argparse
11
+ import copy
12
+ import os
13
+ import random
14
+
15
+ import torch
16
+ import torch.nn.functional as F
17
+ from torch.utils.data import DataLoader
18
+ from torch.optim import AdamW
19
+ from torchvision.utils import save_image
20
+
21
+ from model import TinyUNet
22
+ from diffusion import GaussianDiffusion
23
+ from dataset import CaptionedImageDataset, collate_fn
24
+ from text_encoder import FrozenCLIPTextEncoder
25
+
26
+ PREVIEW_PROMPTS = [
27
+ "a motorcycle parked on the street",
28
+ "a dog on a couch",
29
+ "a plate of pizza",
30
+ "a red bus on a city street",
31
+ ]
32
+
33
+
34
+ def parse_args():
35
+ p = argparse.ArgumentParser()
36
+ p.add_argument("--data_dir", type=str, required=True)
37
+ p.add_argument("--image_size", type=int, default=128)
38
+ p.add_argument("--batch_size", type=int, default=32)
39
+ p.add_argument("--epochs", type=int, default=3)
40
+ p.add_argument("--lr", type=float, default=2e-4)
41
+ p.add_argument("--timesteps", type=int, default=1000)
42
+ p.add_argument("--base_ch", type=int, default=64)
43
+ p.add_argument("--channel_mults", type=str, default="1,1,2,4")
44
+ p.add_argument("--grad_checkpoint", action="store_true")
45
+ p.add_argument("--p_uncond", type=float, default=0.1, help="caption dropout rate, enables CFG at sampling time")
46
+ p.add_argument("--ema_decay", type=float, default=0.9995)
47
+ p.add_argument("--out_dir", type=str, default="checkpoints")
48
+ p.add_argument("--save_every", type=int, default=1)
49
+ p.add_argument("--latest_every", type=int, default=2000, help="steps between resumable latest.pt saves")
50
+ p.add_argument("--resume", type=str, default=None)
51
+ p.add_argument("--device", type=str, default="cuda" if torch.cuda.is_available() else "cpu")
52
+ p.add_argument("--num_workers", type=int, default=4)
53
+ return p.parse_args()
54
+
55
+
56
+ @torch.no_grad()
57
+ def save_preview(ema, diffusion, text_encoder, image_size, path, guidance_scale=3.0):
58
+ ema.eval()
59
+ emb = text_encoder.encode([p for p in PREVIEW_PROMPTS for _ in range(4)])
60
+ null = text_encoder.encode([""])
61
+ g = torch.random.get_rng_state(), torch.cuda.get_rng_state()
62
+ torch.manual_seed(0)
63
+ with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
64
+ x = diffusion.sample(ema, emb, image_size=image_size, show_progress=False,
65
+ null_embed=null, guidance_scale=guidance_scale)
66
+ torch.random.set_rng_state(g[0]); torch.cuda.set_rng_state(g[1])
67
+ x = F.interpolate((x.float() + 1) / 2, scale_factor=2, mode="nearest")
68
+ save_image(x, path, nrow=4, padding=4, pad_value=1)
69
+
70
+
71
+ def main():
72
+ args = parse_args()
73
+ os.makedirs(args.out_dir, exist_ok=True)
74
+ device = args.device
75
+ channel_mults = tuple(int(c) for c in args.channel_mults.split(","))
76
+
77
+ text_encoder = FrozenCLIPTextEncoder(device=device)
78
+
79
+ model = TinyUNet(
80
+ base_ch=args.base_ch,
81
+ channel_mults=channel_mults,
82
+ text_embed_dim=text_encoder.embed_dim,
83
+ use_checkpoint=args.grad_checkpoint,
84
+ ).to(device)
85
+ # EMA of the weights is what gets sampled from — much cleaner than the raw weights.
86
+ ema = copy.deepcopy(model).eval().requires_grad_(False)
87
+ n_params = sum(p.numel() for p in model.parameters())
88
+ print(f"TinyUNet params: {n_params / 1e6:.2f}M")
89
+
90
+ diffusion = GaussianDiffusion(timesteps=args.timesteps, device=device)
91
+ opt = AdamW(model.parameters(), lr=args.lr)
92
+
93
+ step, start_epoch, skip_batches = 0, 0, 0
94
+ if args.resume:
95
+ ck = torch.load(args.resume, map_location=device)
96
+ model.load_state_dict(ck["model_state_dict"])
97
+ ema.load_state_dict(ck["ema_state_dict"])
98
+ opt.load_state_dict(ck["opt_state_dict"])
99
+ step, start_epoch, skip_batches = ck["step"], ck["epoch"], ck["batch_in_epoch"]
100
+ print(f"resumed from {args.resume} at step {step}")
101
+
102
+ dataset = CaptionedImageDataset(args.data_dir, image_size=args.image_size)
103
+ loader = DataLoader(
104
+ dataset,
105
+ batch_size=args.batch_size,
106
+ shuffle=True,
107
+ num_workers=args.num_workers,
108
+ collate_fn=collate_fn,
109
+ drop_last=True,
110
+ pin_memory=True,
111
+ )
112
+ print(f"Dataset size: {len(dataset)} images, {len(loader)} batches/epoch")
113
+
114
+ def ckpt_dict(epoch, batch_in_epoch):
115
+ return {
116
+ "model_state_dict": model.state_dict(),
117
+ "ema_state_dict": ema.state_dict(),
118
+ "opt_state_dict": opt.state_dict(),
119
+ "epoch": epoch,
120
+ "batch_in_epoch": batch_in_epoch,
121
+ "step": step,
122
+ "base_ch": args.base_ch,
123
+ "channel_mults": channel_mults,
124
+ "image_size": args.image_size,
125
+ "timesteps": args.timesteps,
126
+ }
127
+
128
+ for epoch in range(start_epoch, args.epochs):
129
+ model.train()
130
+ running_loss, n_batches = 0.0, 0
131
+ for i, (imgs, captions) in enumerate(loader):
132
+ if epoch == start_epoch and i < skip_batches:
133
+ continue # resume mid-epoch (different shuffle, but same number of steps)
134
+ imgs = imgs.to(device, non_blocking=True)
135
+ # Caption dropout: the empty prompt is the "unconditional" input for CFG.
136
+ captions = ["" if random.random() < args.p_uncond else c for c in captions]
137
+ text_embed = text_encoder.encode(captions)
138
+
139
+ # bf16 autocast: roughly halves activation memory on an 8GB card, no GradScaler needed.
140
+ with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=device.startswith("cuda")):
141
+ loss = diffusion.training_loss(model, imgs, text_embed)
142
+
143
+ opt.zero_grad(set_to_none=True)
144
+ loss.backward()
145
+ opt.step()
146
+
147
+ with torch.no_grad():
148
+ for pe, pm in zip(ema.parameters(), model.parameters()):
149
+ pe.lerp_(pm, 1 - args.ema_decay)
150
+
151
+ running_loss += loss.item()
152
+ n_batches += 1
153
+ step += 1
154
+ if step % 50 == 0:
155
+ print(f"epoch {epoch} step {step} loss {loss.item():.4f}", flush=True)
156
+ if step % args.latest_every == 0:
157
+ torch.save(ckpt_dict(epoch, i + 1), os.path.join(args.out_dir, "latest.pt"))
158
+
159
+ print(f"== epoch {epoch} done, avg loss {running_loss / max(n_batches, 1):.4f} ==")
160
+
161
+ if (epoch + 1) % args.save_every == 0:
162
+ ckpt_path = os.path.join(args.out_dir, f"tits_epoch{epoch+1}.pt")
163
+ torch.save(ckpt_dict(epoch + 1, 0), ckpt_path)
164
+ torch.save(ckpt_dict(epoch + 1, 0), os.path.join(args.out_dir, "latest.pt"))
165
+ print(f"saved checkpoint: {ckpt_path}")
166
+ preview = os.path.join(args.out_dir, f"preview_epoch{epoch+1}.png")
167
+ save_preview(ema, diffusion, text_encoder, args.image_size, preview)
168
+ print(f"saved preview: {preview}", flush=True)
169
+
170
+
171
+ if __name__ == "__main__":
172
+ main()