Transfer Learning

Key ideas

• Transfer learning starts a new model from the weights of a model pretrained on a large source dataset (e.g. ImageNet) instead of from random weights, so what it learned carries over¹. Copying only the architecture transfers nothing.
• Fine-tuning recipe¹: copy every layer except the output layer, add a new output layer sized for the target classes and initialized randomly, then train on the target data, the copied layers with a small learning rate (they are already good) and the new layer with a larger one (d2l uses 10×). Inputs must be preprocessed as in pretraining; d2l normalizes the inputs of its ImageNet-pretrained ResNet-18 with per-channel mean $(0.485, 0.456, 0.406)$ and std $(0.229, 0.224, 0.225)$¹.
• Fixed feature extractor: run the frozen network, read the activations of one layer (e.g. AlexNet's 4096-D codes just before the classifier) and train a linear classifier, such as a linear SVM or softmax, on them². Cheap, and it works with few labels. Any layer can serve; the vector length is that layer's size (a flattened conv map keeps spatial information; a globally pooled one does not).
• Which strategy (rules of thumb)²:

• Why it works³: first-layer features (Gabor-like edges, colour blobs) are general and later layers become specific to the source task; transferability drops as the tasks grow apart, yet even distant-task features beat random ones, and a transferred initialization still helps after fine-tuning. Freezing copied middle layers also hurts because neighbouring layers had co-adapted; letting the copied layers fine-tune removes that drop³.
• Discriminative fine-tuning (ULMFiT): every layer gets its own learning rate, 2.6 times smaller per step down, $\eta^{l-1} = \eta^l / 2.6$, so the general lower layers change least; ULMFiT pairs it with slanted triangular learning rates (short linear increase, long linear decay) and gradual unfreezing from the last layer down, which avoids catastrophic forgetting⁴.
• Domain adaptation: keep training a pretrained model on data from the target domain (e.g. a language model on legal or medical text) so it fits that domain better; ULMFiT fine-tunes its language model on the target task's text before training the classifier⁴.
• Linear probing vs. fine-tuning: with a random head, the backbone changes while the head is still being learned, which distorts the pretrained features. On 10 distribution-shift datasets, fine-tuning was about 2% better in-distribution but 7% worse out-of-distribution than linear probing; linear probing first and full fine-tuning afterwards (LP-FT) beat both⁵.
• Distribution shift⁶, with $q$ the training and $p$ the target distribution: covariate shift changes $p(\mathbf{x})$ but keeps $p(y \mid \mathbf{x})$ (photos at training, cartoons at test); reweight training examples by $p(\mathbf{x})/q(\mathbf{x})$, estimated with a classifier that separates target from source inputs. Label shift changes $p(y)$ but keeps $p(\mathbf{x} \mid y)$ (a disease becomes more common, its symptoms look the same); reweight by $p(y)/q(y)$, estimating the target $p(y)$ by solving $\mathbf{C}\,p(\mathbf{y}) = \mu(\hat{\mathbf{y}})$ with the validation confusion matrix $\mathbf{C}$ and the mean test prediction. Concept shift changes what the labels mean.
• Self-supervised pretraining learns features without labels. SimCLR makes two random augmentations of every image in a batch of $N$; for each view the other view of the same image is the positive and the other $2(N - 1)$ views are negatives; the loss (NT-Xent) is a softmax cross-entropy over cosine similarities divided by a temperature $\tau$⁷. The augmentations define the task, so the features learn to ignore what they change. A constant encoder would satisfy the positive term alone; the negatives push different images apart, and more of them (larger batches, longer training) help⁷. A linear classifier on SimCLR features matched a supervised ResNet-50 on ImageNet, and fine-tuning on 1% of the labels reached 85.8% top-5 accuracy⁷.
• Neural style transfer keeps a pretrained VGG-19 frozen and optimizes the pixels of a synthesized image⁸: content loss compares feature maps at a layer near the output, style loss compares Gram matrices $XX^\top$ (channel × channel inner products summed over positions, so "which features co-occur", not where) at several layers, and a total-variation term removes high-frequency noise⁸.

Formulas

New head parameters: $n_{\text{features}} \cdot n_{\text{classes}} + n_{\text{classes}}$. Label-shift weight: $\beta_i = p(y_i)/q(y_i)$. Gram matrix of a $c \times h \times w$ map: $c \times c$, independent of $h$ and $w$. For unit vectors, $\lVert\mathbf{u} - \mathbf{v}\rVert = \sqrt{2 - 2\cos\theta}$.

Worked example

ResNet-18 ends in global average pooling to 512 features. For 5 target classes the new head Linear(512, 5) has $512 \cdot 5 + 5 = 2565$ parameters, the only trainable ones if the backbone is frozen. Unfreezing the last stage (layer4, $8{,}393{,}728$ parameters) as well makes $8{,}396{,}293$ of the $11{,}179{,}077$ parameters trainable.

Contrastive batch with $N = 8$ images: 16 views; each anchor has 1 positive and 14 negatives, so 15 terms in the denominator. With unit vectors and $\tau = 1$, even a perfect encoder (positive similarity 1, negatives $-1$) has loss $\log(1 + 14e^{-2}) \approx 1.06$; with $\tau = 0.1$ it drops to about $3 \times 10^{-8}$, which is why the temperature is needed.

In code

import torch
import torchvision
from torch import nn
model = torchvision.models.resnet18(weights=None)  # weights="IMAGENET1K_V1" in practice
for p in model.parameters():
    p.requires_grad = False                         # 1) freeze the backbone ...
model.fc = nn.Linear(model.fc.in_features, 5)        # 2) ... then add a new, trainable head
print(sum(p.numel() for p in model.parameters() if p.requires_grad))  # 2565
opt = torch.optim.SGD([p for p in model.parameters() if p.requires_grad], lr=1e-3)
print(model(torch.zeros(2, 3, 224, 224)).shape)     # torch.Size([2, 5])

Common pitfalls

• Freezing with a loop over model.parameters() after replacing the head also freezes the head; backward() then fails because nothing requires a gradient.
• model.eval() does not freeze weights; set requires_grad=False (PyTorch) or trainable = False (Keras).
• Passing only model.classifier.parameters() to the optimizer trains all layers of that classifier, not just the new last layer.
• Use the pretraining normalization statistics, not those of your new dataset, and keep random augmentation out of the validation set.
• Fine-tuning everything with a large learning rate on 300 images overfits and wipes out the pretrained features.
• For data very different from the source, features from the last layers are the least transferable (earlier layers are more general), and if such data are also plentiful, pretraining helps least².
• Style transfer trains the image, not the network; the Gram matrix discards where features occur.
• Label shift is not covariate shift: when the class frequencies change but each class looks the same, $p(y \mid \mathbf{x})$ does change, so reweight by $p(y)/q(y)$⁶.