GANime: Generating Anime Characters from Sketches with Deep Learning

Overview

Published as arXiv:2508.09207 and developed as a Stanford CS230 project, GANime systematically benchmarks four neural approaches to anime sketch colorization — optimization-based Neural Style Transfer (NST), single-pass Fast NST, paired conditional GAN (Pix2Pix), and unpaired cycle-consistent GAN (CycleGAN) — on 17,769 sketch-color image pairs at 256×256 resolution. The primary technical contribution is the addition of Total Variation (TV) regularization to the standard Pix2Pix objective, suppressing color-bleeding artifacts at region boundaries and yielding a 3.3% additional FID improvement (227.9 → 220.5) over vanilla Pix2Pix. The TV-regularized Pix2Pix achieves an FID of 220.5 and SSIM of 0.756 — a 36.2% FID reduction over the NST baseline (345.5) — while CycleGAN reaches FID=272.6 without requiring paired supervision. All architectures share a modular Pix2Pix_Net backbone providing U-Net generators and PatchGAN discriminators, with a custom InstanceNormalization Keras layer for CycleGAN’s per-instance normalization. Implemented in TensorFlow 2.1 with @tf.function graph-mode training.

Four Colorization Approaches

U-Net Generator Architecture

The generator follows a modified U-Net with 8 encoder blocks and 7 decoder blocks + 1 final transposed convolution, accepting flexible-resolution 3-channel inputs. All convolutional layers use tf.random_normal_initializer(0., 0.02) with use_bias=False.

Encoder (downsampling path) — each block: Conv2D(k=4, s=2, 'same') → Normalization → LeakyReLU:

Decoder (upsampling path) — each block: Conv2DTranspose(k=4, s=2, 'same') → Normalization → [Dropout(0.5)] → ReLU → Concatenate with corresponding encoder skip connection:

Final layer: Conv2DTranspose(3, k=4, s=2, activation='tanh') → output (256, 256, 3) in [−1, 1]. The bottleneck output (down8) is excluded from skip connections, yielding 7 skip connections total that preserve fine-grained spatial details like hair strands, eye highlights, and clothing edges.

70×70 PatchGAN Discriminator

Rather than classifying the entire image, the discriminator produces a 30×30 grid of patch predictions, each with a ~70×70 pixel receptive field — enforcing high-frequency structural accuracy. For Pix2Pix (conditional), input and target/generated images are concatenated along the channel dimension (6-channel input); for CycleGAN, only a single 3-channel image is input.

No sigmoid activation — raw logits used with BinaryCrossentropy(from_logits=True).

BatchNorm vs. InstanceNorm: Normalization Design

A critical architectural distinction: Pix2Pix uses BatchNormalization while CycleGAN uses a custom InstanceNormalization Keras layer that normalizes per-instance across spatial dimensions [1, 2] independently per sample and per channel:

class InstanceNormalization(keras.layers.Layer):
    def build(self, input_shape):
        self.scale = self.add_weight('scale', shape=input_shape[-1:],
                                     initializer=tf.random_normal_initializer(1., 0.02))
        self.offset = self.add_weight('offset', shape=input_shape[-1:],
                                      initializer='zeros')
    def call(self, x):
        mean, variance = tf.nn.moments(x, axes=[1, 2], keepdims=True)
        return self.scale * (x - mean) * tf.math.rsqrt(variance + 1e-5) + self.offset

Learnable affine parameters: scale ~ N(1.0, 0.02), offset = 0. This avoids the TensorFlow Addons dependency and provides full control over initialization — important because InstanceNorm prevents cross-sample style leakage in CycleGAN’s unpaired training regime.

Loss Functions

Pix2Pix: Adversarial + L1 + TV Regularization

L_G = L_adversarial + 100 · L_L1 + 1e-4 · L_TV

TV regularization (the novel contribution) computes L1-norm finite differences between horizontally and vertically adjacent pixels, suppressing artifacts and color bleeding at region boundaries. Supports both L1 (anisotropic) and L2 (isotropic) variants via the --norm-tv-loss flag. The 3.3% FID improvement (227.9 → 220.5) validates its effectiveness for anime colorization where sharp color boundaries between hair, skin, and clothing are critical.

CycleGAN: Adversarial + Cycle Consistency + Identity

L_G = L_adversarial(G) + 10 · L_cycle(X→Y→X) + 10 · L_cycle(Y→X→Y) + 5 · L_identity(G)

Cycle consistency loss enforces F(G(x)) ≈ x and G(F(y)) ≈ y (L1 distance), enabling unpaired training. Identity loss \|G(y) − y\| preserves color when the input is already in the target domain. Discriminator losses use a 0.5 scaling factor (distinct from Pix2Pix’s unscaled discriminators).

Neural Style Transfer: Gram Matrix Optimization

L = (1e-2 / 5) · Σ MSE(Gram(style_out), Gram(style_target))
  + (1e4 / 1) · Σ MSE(content_out, content_target)

Style layers: block{1-5}_conv1 (5 layers). Content layer: block5_conv2. Gram matrices computed via Einstein summation (tf.linalg.einsum('bijc,bijd->bcd')) normalized by spatial dimensions. Optimized via Adam (lr=0.02, β₁=0.99, ε=0.1) for 1,000 steps × 100 epochs per image.

Dataset: 17,769 Anime Sketch-Color Pairs

Training augmentation (via @tf.function-decorated random_jitter()): resize to 286×286 → random crop to 256×256 → random horizontal flip (50%). All pixels normalized to [−1, 1] via (pixel / 127.5) − 1.

Training Configuration

CycleGAN uses tf.GradientTape(persistent=True) since gradients for all four networks must be computed from the same forward pass. Both GAN models use @tf.function-decorated training steps for graph-mode execution with TensorBoard logging for all loss components.

Evaluation

FID (Frechet Inception Distance): InceptionV3 (include_top=False, pooling='avg') extracts 2048-dimensional features; FID = ‖μ₁ − μ₂‖² + Tr(Σ₁ + Σ₂ − 2√(Σ₁Σ₂)) via scipy.linalg.sqrtm.

SSIM: tf.image.ssim(max_val=255, filter_size=11, filter_sigma=1.5, k1=0.01, k2=0.03), averaged across all test pairs.

Key findings: (1) Paired supervision (Pix2Pix) dramatically outperforms unpaired approaches (CycleGAN) with a 55.7 FID gap. (2) TV regularization provides consistent improvement across both FID and SSIM. (3) SSIM plateaus around epoch 10 while FID continues improving until epoch 35, suggesting structural fidelity converges before perceptual quality. (4) PatchGAN effectively preserves fine details in hair, eyes, and clothing.

Demo

Tech Stack

Python (3.7+), TensorFlow (2.1), Keras, VGG19 (feature extraction), InceptionV3 (FID evaluation), TensorFlow Hub (Fast NST), SciPy (matrix square root), OpenCV, Kaggle API, TensorBoard, AWS EC2, Google Colab