Conditional GANs & Pix2Pix
The first big unlock of 2014-2017 was controlling what a GAN makes. Attach a label, or an image, or a sentence. Pix2Pix did the image version and it still beats every generic text-to-image model on narrow image-to-image tasks.
The Problem
An unconditional GAN samples arbitrary faces. Useful for a demo, useless in production. You want: map a sketch to a photomap a map to an aerial photomap a daytime scene to nighttimecolorize a grayscale image. In all of these, you are given an input image $x$ and must output $y$ with some semantic correspondence. There are many plausible $y$'s per $x$. Mean-squared error flattens them into mush. An adversarial loss doesn't, because "looks real" is sharp.
Conditional GAN (Mirza & Osindero, 2014) adds a condition $c$ as an input to both $G$ and $D$. Pix2Pix (Isola et al., 2017) specialized this: condition is a full input image, generator is a U-Net, discriminator is a patch-based classifier (PatchGAN), and loss is adversarial + L1. That recipe outperforms from-scratch text-to-image models on narrow image-to-image domains even in 2026 because it is trained on paired datayou have exactly the signal you need.
The Concept
Conditional G. $G(x, z) \to y$. In Pix2Pix, $z$ is dropout inside G (no input noise, Isola found explicit noise got ignored).
Conditional D. $D(x, y) \to [0, 1]$. Input is the pair (condition, output). This is the key difference: D must judge whether $y$ is consistent with $x$, not just whether $y$ looks real.
U-Net generator. Encoder-decoder with skip connections across the bottleneck. Critical for tasks where input and output share low-level structure (edges, silhouette). Without the skips, high-frequency detail vanishes.
PatchGAN discriminator. Instead of outputting a single real/fake score, D outputs an $N \times N$ grid where each cell judges a receptive field of ~70×70 pixels. Averaged. This is a Markov random field assumption: realism is local. Much faster to train, fewer parameters, sharper output.
Loss.
The L1 term stabilizes training and pushes G toward the known target. L1 gives sharper edges than L2 (medians, not means). $\lambda = 100$ was the Pix2Pix default.
CycleGAN, when you don't have pairs
Pix2Pix needs paired $(x, y)$ data. CycleGAN (Zhu et al., 2017) drops this requirement at the cost of an extra loss: the cycle consistency loss. Two generators $G: X \to Y$ and $F: Y \to X$. Train them so $F(G(x)) \approx x$ and $G(F(y)) \approx y$. This lets you translate horses to zebras, summer to winter, without paired examples.
In 2026, unpaired image-to-image is mostly done via diffusion (ControlNet, IP-Adapter) rather than CycleGAN, but the cycle-consistency idea survives in almost every unpaired domain adaptation paper.
Build It
code/main.py implements a tiny conditional GAN on 1-D data. The condition $c$ is a class label (0 or 1). The task: produce a sample from the conditional distribution for the given class.
Step 1: append condition to both G and D inputs
def G(z, c, params):
return mlp(concat([z, one_hot(c)]), params)
def D(x, c, params):
return mlp(concat([x, one_hot(c)]), params)One-hot encoding is the simplest way. Larger models use learned embeddings, FiLM modulation, or cross-attention.
Step 2: train conditional
for step in range(steps):
x, c = sample_real_conditional()
noise = sample_noise()
update_D(x_real=x, x_fake=G(noise, c), c=c)
update_G(noise, c)The generator must match the real distribution for the given conditionnot the marginal.
Step 3: verify per-class output
for c in [0, 1]:
samples = [G(noise, c) for noise in batch]
mean_c = mean(samples)
assert_near(mean_c, real_mean_for_class_c)Pitfalls
- Condition ignored. G learns to marginalize, D never penalizes because condition signal is weak. Fix: condition D more aggressively (early layer, not just late), use projection discriminator (Miyato & Koyama 2018).
- L1 weight too low. G drifts to arbitrary real-looking outputs, not faithful ones. Start $\lambda \approx 100$ for Pix2Pix-style tasks.
- L1 weight too high. G produces blurry outputs because L1 is still an $L_p$ norm. Anneal down once training stabilizes.
- Ground-truth leakage in D. Concatenate $(x, y)$ as D input, not just $y$. Without this D cannot check consistency.
- Mode collapse per class. Each class can collapse independently. Run class-conditional diversity checks.
Use It
2026 state of image-to-image tasks:
| Task | Best approach |
|---|---|
| Sketch → photo, same domain, paired data | Pix2Pix / Pix2PixHD (still fast, still sharp) |
| Sketch → photo, unpaired | ControlNet with a Scribble conditioning model |
| Semantic seg → photo | SPADE / GauGAN2 or SD + ControlNet-Seg |
| Style transfer | Diffusion with IP-Adapter or LoRA; GAN methods are legacy |
| Depth → photo | ControlNet-Depth over Stable Diffusion |
| Super-resolution | Real-ESRGAN (GAN), ESRGAN-Plus, or SD-Upscale (diffusion) |
| Colorization | ColTran, diffusion-based colorizers, or Pix2Pix-color |
| Daytime → nighttime, seasons, weather | CycleGAN or ControlNet-based |
Pix2Pix remains the right tool when (a) you have thousands of paired examples, (b) the task is narrow and repeatable, and (c) you need fast inference. On generic open-domain tasks, diffusion wins.
Ship It
Save outputs/skill-img2img-chooser.md. Skill takes a task description, data availability (paired vs unpaired, N samples), and latency/quality budget, then outputs: approach (Pix2Pix, CycleGAN, ControlNet variant, SDXL + IP-Adapter), training data requirements, inference cost, and eval protocol (LPIPS, FID, task-specific).
Exercises
- Easy. Modify
code/main.pyto add a third class. Confirm G still maps each class's noise to the correct mode. - Medium. Replace L1 with a perceptual-style loss in the 1-D setting (e.g. a small frozen D acting as feature extractor). Does it change sharpness of the conditional distribution?
- Hard. Sketch a CycleGAN in the 1-D setting: two distributions, two generators, cycle loss. Show that it learns to map between them with no paired data.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Conditional GAN | "GAN with labels" | G(z, c), D(x, c). Both networks see the condition. |
| Pix2Pix | "Image-to-image GAN" | Paired cGAN with U-Net G and PatchGAN D + L1 loss. |
| U-Net | "Encoder-decoder with skips" | Symmetric conv network; skips preserve high-freq. |
| PatchGAN | "Local-realism classifier" | D outputs per-patch score instead of global score. |
| CycleGAN | "Unpaired image translation" | Two G's + cycle-consistency loss; no paired data. |
| SPADE | "GauGAN" | Normalizes intermediate activations with the semantic map; segmentation-to-image. |
| FiLM | "Feature-wise linear modulation" | Per-feature affine transform from the condition; cheap conditioning. |
Production note: Pix2Pix as a latency-bound baseline
When you have paired data and a narrow task (sketch → render, semantic map → photo, day → night), Pix2Pix's one-shot inference beats diffusion by an order of magnitude on latency. The production comparison is usually:
| Path | Steps | Typical latency at 512² on a single L4 |
|---|---|---|
| Pix2Pix (U-Net forward) | 1 | ~30 ms |
| SD-Inpaint or SD-Img2Img | 20 | ~1.2 s |
| SDXL-Turbo Img2Img | 1-4 | ~0.15-0.35 s |
| ControlNet + SDXL base | 20-30 | ~3-5 s |
Pix2Pix wins on throughput in static batches (every request is the same FLOPs). Diffusion wins on quality and generalization. The modern play is often to ship a Pix2Pix-style distilled model for the narrow task and a diffusion fallback for tail inputs.
Further Reading
- Mirza & Osindero (2014). Conditional Generative Adversarial Netsthe cGAN paper.
- Isola et al. (2017). Image-to-Image Translation with Conditional Adversarial NetworksPix2Pix.
- Zhu et al. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial NetworksCycleGAN.
- Wang et al. (2018). High-Resolution Image Synthesis with Conditional GANsPix2PixHD.
- Park et al. (2019). Semantic Image Synthesis with Spatially-Adaptive NormalizationSPADE / GauGAN.
- Miyato & Koyama (2018). cGANs with Projection Discriminatorthe projection D.
The full code
Every file for this lesson. Copy it, run it, then change it.
import math
import random
def sigmoid(x):
if x >= 0:
z = math.exp(-x)
return 1 / (1 + z)
z = math.exp(x)
return z / (1 + z)
def leaky(x, a=0.2):
return x if x > 0 else a * x
def leaky_grad(x, a=0.2):
return 1.0 if x > 0 else a
def randn_matrix(rows, cols, rng, scale=0.3):
return [[rng.gauss(0, scale) for _ in range(cols)] for _ in range(rows)]
def matmul(W, x):
return [sum(w * xi for w, xi in zip(row, x)) for row in W]
def add(a, b):
return [x + y for x, y in zip(a, b)]
def one_hot(c, num):
v = [0.0] * num
v[c] = 1.0
return v
def init_mlp(in_dim, hidden, out_dim, rng):
return {
"W1": randn_matrix(hidden, in_dim, rng),
"b1": [0.0] * hidden,
"W2": randn_matrix(out_dim, hidden, rng),
"b2": [0.0] * out_dim,
}
def g_forward(z, c, G, num_classes):
inp = z + one_hot(c, num_classes)
pre1 = add(matmul(G["W1"], inp), G["b1"])
h = [leaky(v) for v in pre1]
out = add(matmul(G["W2"], h), G["b2"])
return out, h, pre1, inp
def d_forward(x, c, D, num_classes):
inp = x + one_hot(c, num_classes)
pre1 = add(matmul(D["W1"], inp), D["b1"])
h = [leaky(v) for v in pre1]
logit = add(matmul(D["W2"], h), D["b2"])[0]
return sigmoid(logit), h, pre1, inp, logit
def sample_real_conditional(n, num_classes, rng):
out = []
for _ in range(n):
c = rng.randrange(num_classes)
if c == 0:
x = rng.gauss(-2.0, 0.3)
else:
x = rng.gauss(2.0, 0.3)
out.append(([x], c))
return out
def update_d(reals, fakes, D, num_classes, lr):
grads = init_grads(D)
for (x, c) in reals:
accumulate_d_grad(x, c, 1.0, D, num_classes, grads)
for (x, c) in fakes:
accumulate_d_grad(x, c, 0.0, D, num_classes, grads)
n = len(reals) + len(fakes)
apply_grads(D, grads, lr, n)
def accumulate_d_grad(x, c, target, D, num_classes, grads):
p, h, pre1, inp, _ = d_forward(x, c, D, num_classes)
dL_dpre2 = p - target
grads["b2"][0] += dL_dpre2
for j in range(len(h)):
grads["W2"][0][j] += dL_dpre2 * h[j]
dh = [D["W2"][0][j] * dL_dpre2 for j in range(len(h))]
dpre1 = [dh[j] * leaky_grad(pre1[j]) for j in range(len(h))]
for j in range(len(h)):
grads["b1"][j] += dpre1[j]
for k in range(len(inp)):
grads["W1"][j][k] += dpre1[j] * inp[k]
def update_g(noise, cs, G, D, num_classes, lr, l1_w=0.0, targets=None):
"""Non-saturating G loss + optional conditional L1 toward a target."""
grads = init_grads(G)
for i, z in enumerate(noise):
c = cs[i]
x_hat, g_h, g_pre1, g_inp = g_forward(z, c, G, num_classes)
p, d_h, d_pre1, d_inp, d_logit = d_forward(x_hat, c, D, num_classes)
dL_dpre2 = p - 1.0
dh_D = [D["W2"][0][j] * dL_dpre2 for j in range(len(d_h))]
dpre1_D = [dh_D[j] * leaky_grad(d_pre1[j]) for j in range(len(d_h))]
dL_dxhat = [0.0] * len(x_hat)
for j in range(len(d_h)):
for k in range(len(x_hat)):
dL_dxhat[k] += D["W1"][j][k] * dpre1_D[j]
if l1_w > 0 and targets is not None:
for k in range(len(x_hat)):
dL_dxhat[k] += l1_w * (1.0 if x_hat[k] > targets[i][k] else -1.0)
grads["b2"] = [grads["b2"][i] + dL_dxhat[i] for i in range(len(x_hat))]
for a in range(len(x_hat)):
for b in range(len(g_h)):
grads["W2"][a][b] += dL_dxhat[a] * g_h[b]
dh_G = [sum(G["W2"][a][b] * dL_dxhat[a] for a in range(len(x_hat)))
for b in range(len(g_h))]
dpre1_G = [dh_G[j] * leaky_grad(g_pre1[j]) for j in range(len(g_h))]
for j in range(len(g_h)):
grads["b1"][j] += dpre1_G[j]
for k in range(len(g_inp)):
grads["W1"][j][k] += dpre1_G[j] * g_inp[k]
apply_grads(G, grads, lr, len(noise))
def init_grads(net):
grads = {}
for k, v in net.items():
if isinstance(v[0], list):
grads[k] = [[0.0] * len(v[0]) for _ in v]
else:
grads[k] = [0.0] * len(v)
return grads
def apply_grads(net, grads, lr, n):
for k, v in net.items():
if isinstance(v[0], list):
for i in range(len(v)):
for j in range(len(v[i])):
v[i][j] -= lr * grads[k][i][j] / n
else:
for i in range(len(v)):
v[i] -= lr * grads[k][i] / n
def mean(xs):
return sum(xs) / max(len(xs), 1)
def main():
rng = random.Random(5)
num_classes, z_dim, hidden = 2, 4, 16
G = init_mlp(z_dim + num_classes, hidden, 1, rng)
D = init_mlp(1 + num_classes, hidden, 1, rng)
batch, g_lr, d_lr = 32, 0.02, 0.01
print("=== conditional GAN on two-mode mixture (class 0 -> -2, class 1 -> +2) ===")
for step in range(1, 601):
reals = sample_real_conditional(batch, num_classes, rng)
cs = [c for _, c in reals]
noise = [[rng.gauss(0, 1) for _ in range(z_dim)] for _ in range(batch)]
fakes = [(g_forward(noise[i], cs[i], G, num_classes)[0], cs[i]) for i in range(batch)]
update_d(reals, fakes, D, num_classes, d_lr)
noise = [[rng.gauss(0, 1) for _ in range(z_dim)] for _ in range(batch)]
cs = [rng.randrange(num_classes) for _ in range(batch)]
update_g(noise, cs, G, D, num_classes, g_lr)
if step % 150 == 0:
probes = {c: [] for c in range(num_classes)}
for _ in range(300):
c = rng.randrange(num_classes)
z = [rng.gauss(0, 1) for _ in range(z_dim)]
probes[c].append(g_forward(z, c, G, num_classes)[0][0])
line = f"step {step:4d}:"
for c in range(num_classes):
line += f" class {c}: mean {mean(probes[c]):+.2f} (n={len(probes[c])})"
print(line)
print()
print("=== sampling per class ===")
for c in range(num_classes):
z_batch = [[rng.gauss(0, 1) for _ in range(z_dim)] for _ in range(6)]
outs = [g_forward(z, c, G, num_classes)[0][0] for z in z_batch]
print(f" class {c}: " + " ".join(f"{v:+.2f}" for v in outs))
print()
print("takeaway: G(z, c) learns a class-specific sampler.")
print(" same architecture, one extra input, totally different samples.")
if __name__ == "__main__":
main()