How Not to Give a FLOP: Combining Regularization and Structured Pruning

Overview

Published as arXiv:2003.13593, this project investigates the synergy between data-level regularization and structured filter pruning for efficient deep network inference. The central finding is that Cutout + Soft Filter Pruning achieves Pareto dominance over unpruned baselines — simultaneously improving accuracy by up to +2.03% while reducing FLOPs by ~15% across five ResNet depths (20/32/44/56/110) on CIFAR-10. The core insight is that regularization redistributes discriminative capacity more uniformly across filters, so that when the lowest-norm filters are zeroed by pruning, they are truly redundant rather than containing critical specialized features. The study evaluates a 5×6 experiment matrix (5 architectures × 6 configurations: baseline, Mixup-only, Cutout-only, pruning-only, Mixup+pruning, Cutout+pruning) totaling 30 experiments, revealing that Cutout+Pruning consistently outperforms Mixup+Pruning by 0.93–1.53%, despite Mixup being the stronger standalone regularizer. Implemented in PyTorch with PyTorch Lightning and Weights & Biases experiment tracking.

ResNet Architectures: CIFAR-Scale Family

All architectures follow He et al. (2015) with 3 stages, channel widths 16 → 32 → 64, and spatial resolutions 32×32 → 16×16 → 8×8. Each residual block uses two 3×3 convolutions with BatchNorm + ReLU and Option A shortcut connections (identity with zero-padding for dimension changes — no learnable projection parameters):

self.shortcut = LambdaLayer(lambda x:
    F.pad(x[:, :, ::2, ::2], (0, 0, 0, 0, planes//4, planes//4), "constant", 0))

Initial layer: single 3×3 Conv2D (16 channels, no max pooling) → global average pooling → linear classifier. All weights initialized via Kaiming Normal (init.kaiming_normal_).

Two Regularization Techniques

Mixup (Zhang et al., 2018)

Applied during training_step() by creating convex combinations of input-label pairs within each minibatch:

x̃ = λ·xᵢ + (1−λ)·xⱼ,   λ ~ Beta(α, α),  α = 1.0
L_mixup = λ·L(pred, yₐ) + (1−λ)·L(pred, yᵦ)

Permutation index generated via torch.randperm(batch_size). When α=1.0, lambda follows a uniform distribution over [0, 1].

Cutout (DeVries & Taylor, 2017)

Applied as a torchvision transform (after normalization, generating a different random mask per image access):

• n_holes=1 square patch per image, length=16 pixels per side
• Random center (x, y) sampled uniformly; patch extends beyond boundaries (clipped by np.clip)
• Zeros out normalized pixel values, forcing the network to recognize objects from partial observations

Design distinction: Cutout operates at the input level (spatial masking), while Mixup operates at the label level (inter-class blending). This difference proves critical for pruning compatibility.

Soft Filter Pruning (He et al., 2018)

A structured pruning method operating at the filter level — entire convolutional filters are zeroed, not individual weights.

Algorithm (executed at end of each epoch):

1. L2-norm ranking: For each Conv2D layer, compute ‖Fᵢ‖₂ = √(Σ w²) across all kernel dimensions per filter
2. Selection: Sort by L2-norm; select the bottom (1 − pruning_rate) fraction for zeroing
3. Binary codebook masking: Multiply filter weights element-wise by {0, 1} mask
4. Soft reset: Unlike hard pruning, zeroed filters remain in the architecture and can be restored by backpropagation in subsequent epochs

The “soft” aspect is key: the network continuously reallocates capacity, re-evaluating which filters to keep at every epoch — analogous to a dynamic lottery ticket search. With default pruning_rate=0.9 (90% retained), ~15% FLOP reduction is achieved uniformly across architectures.

Layer selection: Pruning applies to every convolutional layer except downsample shortcuts and the final classifier. For CIFAR-scale ResNets using Option A (zero-padding) shortcuts, no skip lists are needed since shortcuts have no learnable parameters.

Training Configuration

Results: 5×6 Experiment Matrix on CIFAR-10

Accuracy Deltas vs. Baseline

Key Findings

1. Pruning alone degrades accuracy by 0.16–1.07% across all depths — removing filters without regularization loses critical features.

2. Cutout + Pruning achieves Pareto dominance: Better accuracy AND ~15% fewer FLOPs vs. unpruned baselines at every depth.

3. Cutout > Mixup for pruning compatibility: Cutout+Pruning outperforms Mixup+Pruning by 0.93–1.53% despite Mixup being the stronger standalone regularizer (+1.55% avg vs. +1.07% avg). This is because Cutout’s spatial masking forces filter-level redundancy (each filter must be robust to missing input regions), while Mixup’s inter-class blending spreads information across classes but not necessarily across filters.

4. Benefits scale with depth: Cutout+Pruning improvement grows from +1.24% (ResNet-20) to +2.03% (ResNet-56), suggesting deeper networks have more prunable redundancy when properly regularized.

5. Near-free compression: The accuracy gap between Cutout-only and Cutout+Pruning averages only 0.53%, meaning ~15% FLOP reduction comes nearly “for free.”

FLOP Reduction Analysis

Analytical FLOP computation models the per-layer reduction: for the first convolution per block (only output channels pruned), FLOPs scale by retention rate r; for the second convolution (both input and output pruned), FLOPs scale by r².

Demo

Tech Stack

Python (3.7+), PyTorch, PyTorch Lightning, Weights & Biases (experiment tracking), TensorBoard, NumPy