Transfer learning

Transfer learning is a technique where a model trained on one task (the source task) is reused as the starting point for a model on a different but related task (the target task). Instead of learning from scratch, the model transfers knowledge such as learned features, which saves time and data. It is especially useful when the target task has limited labeled data.

Training a deep model from scratch needs a large amount of labeled data to learn useful features; with little data the model would overfit. A pretrained model already learned general features from a huge dataset, so you only need to adapt the final layers using your small dataset. This lets you achieve good performance even with limited examples.

Early convolutional layers capture low-level, general features such as edges, colors, and simple textures. These features are common across almost all images, which is why they transfer well to new image tasks. Later layers capture more task-specific, high-level features like object parts or whole objects.

In feature extraction, you freeze the pretrained layers and only train a new classifier head on top, using the pretrained network as a fixed feature generator. In fine-tuning, you also allow some or all of the pretrained weights to update during training so they adapt to the new task. Fine-tuning can give higher accuracy but needs more data and care to avoid overfitting.

Freezing a layer means its weights are not updated during training; gradients are not used to change them. This keeps the pretrained knowledge in those layers intact and reduces the number of trainable parameters. Freezing is common for early layers when you only want to train a new output head.

# PyTorch: freeze all layers
for param in model.parameters():
    param.requires_grad = False

ImageNet contains millions of labeled images across a thousand diverse categories, so models trained on it learn rich, general-purpose visual features. These features (edges, shapes, textures, object parts) are useful for a wide range of other image tasks. As a result, an ImageNet-pretrained model transfers effectively even to images it has never seen before.

The pretrained weights are already close to a good solution, so large updates could destroy the useful features they encode. A small learning rate makes gentle adjustments that adapt the weights to the new task without erasing prior knowledge. This stabilizes training and helps avoid catastrophic forgetting.

Catastrophic forgetting happens when a model, while learning a new task, overwrites the weights that encoded knowledge from the original task, losing its previous abilities. In transfer learning it can occur if you fine-tune too aggressively with a high learning rate. Techniques like freezing layers, using small learning rates, or gradual unfreezing help reduce it.

With a large dataset and a similar task, fine-tuning the entire network (or most of it) is recommended because you have enough data to safely update the weights without overfitting. The strong starting point from pretraining usually leads to faster convergence and higher accuracy than training from scratch. Full fine-tuning lets the model fully adapt its high-level features to your data.

A small dataset makes fine-tuning risky because the model can easily overfit, but a very different task means the pretrained high-level features may not be relevant, so frozen feature extraction may also work poorly. You often must rely on early/general features only and use heavy regularization or data augmentation. This combination gives the weakest knowledge overlap and the least data to compensate.

The classifier head is a new set of layers, usually ending in a softmax sized to your number of target classes, placed on top of the pretrained feature extractor. It maps the general features produced by the base model to the specific categories of your task. Since the original output layer was built for the source classes, you replace it with this new head.

A pretrained model is a network whose weights have already been learned by training on a large dataset for some task. You can obtain them from libraries and model hubs such as Keras Applications, PyTorch torchvision, TensorFlow Hub, or Hugging Face. Using these saves the enormous compute and data needed to train from scratch.

from torchvision import models
model = models.resnet18(weights='IMAGENET1K_V1')

Models like BERT are first pretrained on huge text corpora using self-supervised objectives (e.g., predicting masked words), learning general language representations. For a target task such as sentiment analysis, you add a small task-specific head and fine-tune the model on your labeled data. This pretrain-then-fine-tune approach achieves strong results with relatively little task data.

Self-supervised pretraining creates training labels automatically from the data itself, for example by hiding (masking) some words and asking the model to predict them. This lets models learn from massive amounts of unlabeled text without manual annotation. The general language knowledge gained transfers to many downstream tasks via fine-tuning.

Domain adaptation is a form of transfer learning where the task stays the same but the data distribution (domain) differs between source and target, such as recognizing objects in daytime photos then adapting to nighttime photos. The goal is to make a model trained on the source domain perform well on the target domain. Techniques try to align the feature distributions of the two domains.

A pretrained model learned its weights under specific input conditions, such as a fixed image size and a particular normalization (mean/standard deviation per channel). Feeding inputs in a different format would mismatch what the network's filters expect, degrading performance. Matching the original preprocessing ensures the learned features activate correctly.

# torchvision ImageNet normalization
normalize = transforms.Normalize(
  mean=[0.485, 0.456, 0.406],
  std=[0.229, 0.224, 0.225])

Gradual unfreezing means you start by training only the new top layers while the rest stay frozen, then progressively unfreeze and train deeper layers over successive stages. This prevents large, destabilizing updates to the pretrained weights early on. It helps preserve general features while still allowing the model to adapt deeply to the new task.

Because the pretrained model already encodes useful features, you only need to train a small head or fine-tune for a few epochs instead of learning everything from random initialization. This requires far fewer training iterations and less data, cutting GPU time and energy use significantly. It is a key reason transfer learning is popular in practice.

The pretrained network acts as a fixed feature extractor: its frozen layers transform inputs into a vector of high-level features, and only the new classifier on top is trained on those features. Because the base does not change, you can even precompute and cache its outputs to speed up training. This is efficient and works well when the source and target are similar.

The model is overfitting: it memorized the tiny training set rather than learning to generalize, which is common when fine-tuning many parameters on very little data. Remedies include freezing more layers (feature extraction only), adding data augmentation, using regularization/dropout, and reducing the learning rate. Collecting more data would also help.

The source domain is the original data and task the model was trained on, while the target domain is the new data and task you want to apply the model to. Transfer learning aims to carry knowledge from the source to the target. The more related the two domains are, the more effective the transfer typically is.

Negative transfer occurs when the source and target tasks are too dissimilar, so the transferred features mislead the model and it performs worse than training from scratch. Irrelevant prior knowledge can bias learning in the wrong direction. Choosing a source task related to the target is important to avoid this.

The original output layer has a fixed number of units matching the source classes (e.g., 1000 for ImageNet), but your task has a different number of categories. You replace it with a new output layer sized to your classes and randomly initialize it before training. The rest of the network can be kept and reused as the feature extractor.

import torch.nn as nn
model.fc = nn.Linear(model.fc.in_features, num_classes)

Few-shot learning aims to learn a new task from only a handful of labeled examples per class. It commonly relies on transfer learning: a model pretrained on large data provides strong general representations so that just a few examples suffice to adapt it. Without such transferred knowledge, learning from so few examples would be nearly impossible.

Setting include_top=False loads the convolutional feature-extraction base of the network but excludes its original fully connected classification layers. This lets you attach your own custom classifier head suited to your task. It is a standard step when repurposing a pretrained CNN for new classes.

from tensorflow.keras.applications import VGG16
base = VGG16(weights='imagenet', include_top=False,
             input_shape=(224,224,3))

Data augmentation artificially expands the training set by applying label-preserving transformations such as flips, rotations, crops, and brightness changes. This adds variety, reducing overfitting that is otherwise common when fine-tuning on small data. Combined with transfer learning, it helps the adapted model generalize better.

Deep networks learn a hierarchy of features, and the lower/general features (edges, shapes, basic patterns) are not specific to any one task, so they are useful for many problems sharing the same data type. This generality is what makes the representations transferable. Only the higher, task-specific features need to be replaced or adapted.

In inductive transfer learning the target task is different from the source task and labeled target data is available to learn the new task. In transductive transfer learning the task is the same but the domains differ, and typically labeled data exists only in the source domain (domain adaptation is an example). The key distinction is whether the task or the domain is what changes.

The new head starts with random weights, which produce large, noisy gradients early on; if the base were unfrozen, those gradients could corrupt the valuable pretrained features. By first training the head until it is reasonable, you ensure that later fine-tuning makes only gentle, beneficial adjustments. This two-phase approach leads to more stable and accurate results.

Just as a person who already knows how to ride a bicycle can learn to ride a motorcycle more easily by reusing balance and steering skills, a model reuses knowledge from a previous task to learn a new related one faster. Both leverage prior experience instead of starting from zero. This analogy captures the core intuition of transfer learning.

You must resize your images to 224x224 to match the expected spatial dimensions, and convert grayscale to 3-channel format (e.g., by duplicating the single channel) so the input has the RGB shape the model expects. You should also apply the same normalization the model was trained with. These steps make your data compatible with the pretrained network's input requirements.

You would compare the target-task performance (e.g., validation accuracy or F1) of a model trained from scratch versus one using transfer learning, on the same small dataset. If the transfer-learned model reaches higher accuracy and/or converges in fewer epochs, that demonstrates the benefit. Plotting learning curves for both is a clear way to show this.

Medical image datasets are often small and expensive to label because they require expert annotation, so training large models from scratch is impractical. Transfer learning lets practitioners start from models pretrained on large general image datasets and fine-tune them on limited medical data. This achieves strong performance despite the scarcity of labeled examples.

Since the frozen base never changes, its output for each image is constant across epochs, so computing it once and caching the feature vectors avoids redundant forward passes. You can then train the small classifier head directly on these stored features, making each epoch much faster and cheaper. This is a practical speed-up for feature-extraction transfer learning.