Landmark Architectures

Key ideas

Counts are for torchvision's ImageNet models (see the code below); MobileNet's is from its paper.

• AlexNet made the 2012 breakthrough: ReLU instead of sigmoid (cheaper, trains more easily), dropout in the fully connected layers, heavy data augmentation, and an $11 \times 11$ first convolution¹. In d2l's version (one $224 \times 224$ input channel, 10 classes) the two hidden FC layers need $6400 \times 4096$ and $4096 \times 4096$ weight matrices, over 40 million parameters and 164 MB¹. Counted over that whole model, the FC layers hold about 92% of its 46.8 million parameters, while the convolutions, whose kernels are reused at every position, do about 95% of its 0.94 billion multiply–adds.
• VGG builds the network from blocks: several $3 \times 3$ convolutions with stride 1 and padding 1 (size kept), then $2 \times 2$ max-pooling with stride 2 (size halved)². A deep net of small filters beat a shallow net of larger ones: two stacked $3 \times 3$ layers see a $5 \times 5$ region, three see $7 \times 7$ with $27C^2$ instead of $49C^2$ weights and three ReLUs instead of one². VGG-11 has 8 conv + 3 FC weight layers, VGG-16 13 + 3, VGG-19 16 + 3². Five poolings turn $224$ into $7$, and the first FC layer, fed by the $512 \times 7 \times 7$ map, holds most of the parameters.
• NiN follows each ordinary convolution with two $1 \times 1$ convolutions (a small MLP applied at every pixel) and ends with a block that outputs one channel per class, then global average pooling, with no FC layer at all³. Dropping the giant FC layers is the main reason it has far fewer parameters than AlexNet and VGG³.
• Inception (GoogLeNet): four parallel branches ($1 \times 1$; $1 \times 1$ then $3 \times 3$; $1 \times 1$ then $5 \times 5$; $3 \times 3$ max-pool then $1 \times 1$) whose outputs are concatenated along channels; the $1 \times 1$ convs of the middle branches first reduce the channels to cut cost⁴. Like NiN it ends with global average pooling, followed by one FC layer⁴. Pooling instead of 4096-wide FC layers plus these channel reductions keep GoogLeNet small.
• ResNet: a residual block computes $\mathbf{x} + g(\mathbf{x})$, so pushing the last layer's weights to zero gives the identity; extra blocks can always fall back to "do nothing", and the function classes are nested⁵. Each conv is followed by batch norm and ReLU; a $1 \times 1$ conv on the shortcut matches channels or stride; the network ends with global average pooling and one FC layer⁵. The shortcut also passes gradients straight back. Nesting guarantees a training fit at least as good, not better test accuracy.
• ResNeXt uses grouped $3 \times 3$ convolutions ($g$ times cheaper; groups exchange no information) between $1 \times 1$ convs that mix the channels again⁵.
• DenseNet concatenates each layer's output to its input instead of adding it. Each layer adds only $k$ channels (the growth rate), and transition layers ($1 \times 1$ conv + average pooling) cut the channels and halve the size⁶. The concatenations make it memory-hungry⁶.
• MobileNet replaces each standard convolution by a depthwise one (one $k \times k$ filter per input channel) followed by a pointwise $1 \times 1$ convolution that mixes the channels. With $3 \times 3$ kernels this needs 8 to 9 times less computation; MobileNet has 4.2 M parameters and 569 M multiply–adds, against 29.3 M and 4866 M for the same network with full convolutions⁷.
• Graph neural networks pass messages: each node aggregates its neighbours' features (sum, mean or max) and updates its own. Such GNNs can tell two graphs apart at most as well as the Weisfeiler–Lehman test, and sum aggregation with injective functions reaches that bound⁸. With identical starting features, every node of a 6-cycle and of two disjoint triangles has two neighbours, so no message-passing GNN separates these graphs.

Formulas

Worked example

torchvision's VGG-16 on a $224 \times 224$ image: the conv part outputs $512 \times 7 \times 7 = 25{,}088$ values. The first FC layer to 4096 units has $25{,}088 \cdot 4096 + 4096 = 102{,}764{,}544$ parameters, 74% of the network's $138{,}357{,}544$; the three FC layers together hold 89%. At 4 bytes per float32 the whole model is about 553 MB. Stacking three $3 \times 3$ convs with $C = 256$ uses $27 \cdot 256^2 = 1{,}769{,}472$ weights, against $49 \cdot 256^2 = 3{,}211{,}264$ for one $7 \times 7$ conv with the same receptive field. A dense block with 64 input channels, 4 layers and growth rate 32 feeds its fourth layer $64 + 3 \cdot 32 = 160$ channels and outputs $192$.

In code

import torchvision.models as M
names = ["alexnet", "vgg16", "resnet18", "resnet50", "densenet121"]
counts = {n: sum(p.numel() for p in getattr(M, n)(weights=None).parameters()) for n in names}
print(counts)  # {'alexnet': 61100840, 'vgg16': 138357544, 'resnet18': 11689512, 'resnet50': 25557032, 'densenet121': 7978856}
vgg = M.vgg16(weights=None)                          # architecture only, no download
print(sum(p.numel() for p in vgg.classifier[0].parameters()))  # 102764544

Common pitfalls

• ResNet adds the shortcut; DenseNet and Inception concatenate along channels.
• ResNet-50 has about 25.6 million parameters, not billions, and ends with global average pooling plus one FC layer, not two 4096-unit layers as VGG does.
• The original ResNet uses batch normalization, not layer normalization, and ReLU, not tanh; ResNeXt keeps that recipe.
• Highway networks (gated shortcuts) came before ResNet, which uses plain identity shortcuts instead⁵.
• In AlexNet and VGG the FC layers hold most parameters, but the conv layers do most of the computation.
• VGG's depth counts weight layers only; pooling, ReLU and dropout are not counted.
• A ViT has much less of a CNN's built-in locality and translation bias, so on small datasets a ResNet of similar size usually wins.
• A GNN that sees only the bond graph cannot distinguish a molecule from its mirror image; geometry must be part of the input.