CNN from 0 — why convolutional networks exist and what every layer actually does

This is a from-scratch tour of the convolutional neural network (CNN): the model architecture behind almost every image classifier. We will start with why CNNs exist at all, then build up the picture one layer at a time — each with a diagram and a real-world example — and finish by reading a real VGG implementation line by line.

The running example throughout: a raw photo of a cat goes in, the label cat comes out. Every layer below is a step in turning those pixels into that word.

Why do we need a CNN?

Imagine doing image classification without one. You would have to:

  • Hand-design the features. Sit down and decide what numbers describe a cat — edge counts, color histograms, corner positions — and write code to extract each one.
  • Lay them out in a table. One row per image, one column per feature you invented, exactly like a spreadsheet. The model never sees the picture; it only sees your guesses about the picture.
  • Train in two separate stages. First the feature-extraction pipeline, then a separate classifier on top of it. Two systems to build, tune, and keep in sync.

This is slow, brittle, and only as good as the features a human thought to write down.

A CNN collapses all of that into one model fed nothing but raw pixels:

  • Feature extraction is learned, not hand-written — the network discovers which edges, textures, and shapes matter.
  • Classification sits on top of those learned features in the same network.
  • One model for everything. You feed in pixels, you get out a label, and a single training run optimizes both halves together.
The old way: manual feature engineeringRaw imagepixelsHuman writesfeature codeedges, histograms…Classifierstage 2two stages · features capped by human imaginationThe CNN way: one model, raw pixels inRaw imagepixelsCNNlearned feature extraction + classificationtrained end-to-end in one run”cat”

The one formula behind every layer

Strip away the names and almost every layer is doing the same arithmetic:

output = features · weight + bias

A layer multiplies its input by a set of learnable weights, adds a bias, and passes the result on. What makes layers different is how they apply those weights — across the whole flattened input, across a small sliding window, across channels, and so on. Keep this formula in mind; everything below is a variation on it.

Fully-connected layer

What: flattens the input into one long vector and connects every input value to every output value (a full matrix multiply, nn.Linear in PyTorch). Why: it mixes all information together, which makes it the natural choice for the final classification step. Real-world: the very last layer of the cat classifier — it takes the high-level features and produces one score per class (cat, dog, bird…). The highest score wins.

The catch: “connect everything to everything” throws away where things were in the image. That spatial blindness is exactly why we don’t build the whole network out of these — but it is perfect for the final verdict.

Features (flattened)x₁x₂x₃x₄w₁w₂w₃w₄y₁cat ✓y₂dogy₃birdy₁ = w₁x₁ + w₂x₂ + w₃x₃ + w₄x₄ + b₁Lines drawn for the”cat” output only —every output (class) hasits own row of weights. 
y = W x + b
shapes:   x: (in,)     W: (out × in)     b: (out,)     y: (out,)
one class score:   yᵢ = bᵢ + Σⱼ Wᵢⱼ · xⱼ      (a weighted sum of every input)

Convolution layer

What: slides a small kernel (e.g. 3×3) across the image, computing features · weight + bias at every position. nn.Conv2d in PyTorch. Why: unlike a fully-connected layer, convolution preserves spatial structure — it only ever looks at a small patch of neighboring pixels at a time, so the relationship between adjacent pixels survives. That locality is exactly what makes it great at feature extraction. Real-world: the first convolution layers of the cat classifier learn to fire on edges and textures — a whisker, the curve of an ear — wherever they appear in the frame. Because the same kernel slides everywhere, a feature learned in one corner is recognized in every corner.

2D convolution — one output pixel of one output channel:
   Y[i,j] = b + Σ_c Σ_{u,v}  X[c, i+u, j+v] · K[c, u, v]
            (sum over input channels c and the k×k window u, v)

output spatial size:   O = ⌊(I − k + 2p) / s⌋ + 1
   I input · k kernel · p padding · s stride
   k=3, p=1, s=1  →  O = I   (VGG keeps size; pooling does the shrinking)

learnable parameters:  C_out · (C_in · k² + 1)    (+1 = bias; grouped conv divides this by g)

The diagram below shows one 3×3 kernel sliding over the image. At each stop it multiplies the 9 pixels under it by its 9 weights, sums them, and writes a single number into the output feature map. Here the kernel has learned a vertical-edge detector, so it lights up along the cat’s edge.

Input image (6 × 6)3×3 kernel slides →· weights+ bias, sumFeature map (4 × 4)9.2each cell = one kernel positionThe same 9 weights are reused at every position — that weight sharing is why CNNs need so few parameters.

Why did the map shrink (6×6 → 4×4)? Padding

Notice the feature map above came out smaller than the input — even though this is convolution, not a pooling/downsampling step. That shrinkage is a border effect, not intentional downsampling: a 3×3 kernel only produces an output where all 9 of its cells land on real pixels. Centered on an edge pixel, part of the kernel would hang off the image with nothing to multiply — so the kernel’s center can only visit the interior positions, dropping k − 1 pixels along each axis (1 per side for a 3×3 → 6 → 4). This is called “valid” convolution.

The fix is padding — add a ring of zeros around the image so the kernel can be centered on the edge pixels. With padding = (k−1)/2 (= 1 for a 3×3), the output matches the input size; this is “same” convolution.

”valid” — no paddingkernel must fit fully → border lostvalid 4×4✗6×6 → 4×4O = (6 − 3)/1 + 1 = 4”same” — padding = 1zero ring lets the kernel reach the edge✓6×6 → 6×60 = padding

This is exactly why VGG sets padding=1 on every 3×3 conv (nn.Conv2d(in, out, 3, padding=1) in the code): the convolutions preserve spatial size, so the clean 32 → 16 → 8 → 4 → 2 progression in the end-to-end diagram comes entirely from the pooling layers — never from the convolutions. (The diagram above uses the no-padding case just to make the sliding mechanic visible.)

Why not just use a fully-connected layer on the image? Two reasons, both rooted in the weight sharing above. Parameters: one fully-connected layer mapping a 224×224×3 image to 4096 units needs 224·224·3·4096 ≈ 616 million weights; a single 3×3 conv with 64 filters needs 3·3·3·64 = 1,728. The kernel is reused at every position instead of learning a separate weight per pixel. Generalization: because the same kernel slides everywhere, a feature learned in one spot is detected everywhere — the layer is translation-equivariant and never has to re-learn “cat ear” for each pixel.

A feature map is a 3D volume — a cluster of cubes

The 2D grid above is a simplification. A convolution layer doesn’t produce one feature map; it produces one per kernel. Stack them and a feature map is really a 3D volume of numbers with three axes: width and height (the spatial layout, inherited from the image) and channels (one slice per learned feature). It helps to picture it literally as a cluster of cubes, where every little cube holds one activation value.

shape:   (C, H, W)        with a batch dimension:   (N, C, H, W)
total values held:   N · C · H · W
width (W)height (H)channels (C)one slice per featureone (x, y) location,all C channels= the feature vector for that pixelIn PyTorch this volume has shape (C, H, W) — and with a batch dimension, (N, C, H, W).

Receptive field: how a deep neuron sees the whole image

A single 3×3 convolution only looks at 9 neighboring pixels. So how does a CNN ever reason about a whole cat? The answer is the receptive field: the region of the original input that influences one value deep in the network. Each layer you stack widens it.

Two 3×3 convolutions in a row mean a value in the second layer sees a 3×3 patch of the first layer — and each of those came from a 3×3 patch of the input. The windows overlap and add up: the second-layer value depends on a 5×5 patch of the input. Stack a third and it grows to 7×7; add pooling and it leaps further. After enough layers a single deep neuron’s receptive field covers the entire image — which is exactly why the deepest layers can recognize whole objects while the early ones only catch edges.

stacked stride-1 k×k conv layers:   RF(n) = 1 + n·(k − 1)
   k=3, n=2  →  RF = 1 + 2·2 = 5      (the 5×5 patch below)

with strides / pooling:   RF_L = RF_{L−1} + (k_L − 1) · Πᵢ sᵢ
   earlier strides sᵢ multiply the growth, so pooling makes the field jump
Input5 × 5 regionAfter conv 13 × 3 regionAfter conv 21 valueOne value after two stacked 3×3 convolutions depends on a 5×5 patch of the input.Keep stacking layers and pooling, and the receptive field eventually spans the whole image.

Grouped convolution

What: instead of letting every output channel look at every input channel, you split the channels into g groups and convolve each group independently. Why — and why is the model smaller? This is the part worth doing the arithmetic on.

A normal convolution from C_in input channels to C_out output channels with a k×k kernel has:

params = C_in × C_out × k × k

With g groups, each group only maps C_in / g inputs to C_out / g outputs, and there are g such groups:

params = g × (C_in / g) × (C_out / g) × k × k
       = C_in × C_out × k × k ÷ g

So the parameter count (and the compute) drops by a factor of g. That is the whole reason grouped convolution makes models smaller: each output channel is wired to only a slice of the input channels instead of all of them, so there are simply fewer connections to store and multiply. Taken to the extreme (g = C_in, one group per channel) this becomes a depthwise convolution, the trick at the heart of lightweight mobile architectures.

The trade-off: groups can’t share information with each other inside the layer, so designs usually mix the groups back together afterwards (e.g. with a 1×1 convolution).

Normal convolutionall input channels → every output channelinput · C_inevery channel mixesoutput · C_outC_in × C_out × k² params(e.g. 4 × 4 = 16 connections)Grouped convolution (g = 2)channels split into 2 groups, each convolved aloneinput · 2 groupsoutput · 2 groupsgroup B → Bgroup A → AC_in × C_out × k² ÷ g params2 × (2 × 2) = 8 connections — half (÷ g)The depth of each cuboid is the channel axis. Normal conv wires the whole depth together;grouped conv slices the depth into g independent bands — fewer wires, fewer parameters.

The 1×1 convolution

A kernel can be 1×1 — it covers a single pixel but spans all channels. It does no spatial mixing at all; instead it computes a weighted combination across channels at each location. That makes it the standard tool for two jobs: changing the channel count (e.g. squeezing 256 channels down to 64 to save compute — the “bottleneck” in ResNet and Inception) and mixing grouped-convolution groups back together. It is cheap (C_in · C_out params, no k² factor) and turns up everywhere in modern architectures. Mentally, a 1×1 conv is a small fully-connected layer applied independently at every pixel.

Pooling layer

What: downsamples a feature map, shrinking its height and width. nn.MaxPool2d keeps the largest value in each window; nn.AvgPool2d keeps the average. Why: it reduces the size of the feature maps, which cuts compute and gives the network a little translation tolerance — a feature that shifts by a pixel still lands in the same pooled cell. Real-world: after the early convolutions detect whiskers and ear-edges everywhere, max-pooling keeps the strongest response in each region and discards the precise pixel location. The classifier cares that a whisker is present, not that it was at pixel (213, 88).

window of size k, stride s:
   max pool:  Y[i,j] = max  over the window of  X
   avg pool:  Y[i,j] = mean over the window of  X
output size:   O = ⌊(I − k)/s⌋ + 1      (k = s = 2  →  O = I/2, halves H and W)
no learnable parameters — pooling is a fixed reduction.
Feature map (4 × 4)15328410621739242×2 max-poolPooled (2 × 2)5879keep the max of each colored 2×2 block

Equivariance vs invariance: convolution is translation-equivariant — shift the input and the feature map shifts the same way. Pooling adds a dose of translation invariance — a feature that moves within a pooling window gives the same output. A CNN’s robustness to small shifts comes from this combination, not from convolution alone.

Normalization layer

What: re-centers and re-scales the feature maps so their values have a consistent distribution (roughly zero mean, unit variance) before the next layer. Why: without it, the scale of activations drifts as it passes through layers, which makes training slow and unstable. Normalization keeps the numbers in a healthy range so the network learns faster.

The four common variants differ only in which dimensions they average over — and that single choice is what distinguishes them:

  • Batch norm — normalize each channel across the whole batch. Great for large-batch image training; sensitive to small batch sizes.
  • Layer norm — normalize across all channels of a single sample. The default in Transformers; batch-size independent.
  • Instance norm — normalize each channel of each sample on its own. Popular in style transfer.
  • Group norm — split channels into groups and normalize within each group. A middle ground that works well when batches are small.
every variant does the same two steps, then differs only in the averaging axes:
   x̂ = (x − μ) / √(σ² + ε)        # subtract mean, divide by std (ε avoids ÷0)
   y  = γ · x̂ + β                 # learnable scale γ and shift β

axes that μ and σ² are computed over (tensor (N, C, H, W)):
   Batch norm     →  (N, H, W)       per channel
   Layer norm     →  (C, H, W)       per sample
   Instance norm  →  (H, W)          per sample, per channel
   Group norm     →  (C/g, H, W)     per sample, per channel-group

The diagram shows a tensor of shape (N batch, C channels, H×W spatial) and shades which slice each method averages over.

Batch normper channel, across batch→ batch (N)channels (C)Layer normper sample, across channelsone sample, all CInstance normper channel, per sampleone cell onlyGroup normchannel groups, per samplegroup of channels

Batch norm: train vs eval: during training, batch norm normalizes using the current batch’s mean and variance and updates a running average of them. At inference it switches to those frozen running statistics, so an image’s output doesn’t depend on whatever else happens to be in its batch. Forgetting to call model.eval() is a very common bug. This batch dependence is also exactly why batch norm degrades with tiny batches — and why layer norm and group norm exist.

Activation function

What: applies a non-linear function f to the layer’s output. With y_pre = W · X + b, the activation produces output = f(y_pre). ReLU — f(x) = max(0, x) — is the workhorse. Why: stacking linear layers only ever produces another linear function, no matter how many you pile up. The non-linearity is what lets the network bend and fold its decision surface to fit the messy, non-linear reality of real images. Without it, a 50-layer network is no more expressive than a single layer. Real-world: ReLU after each convolution lets the cat classifier represent “this region has a strong vertical edge and a curved texture but not a straight horizontal line” — the kind of conditional, non-linear logic that distinguishes a cat’s ear from a triangle.

applied elementwise to the linear output:   output = f(W x + b)
   ReLU       f(x) = max(0, x)
   LeakyReLU  f(x) = max(αx, x)              (small α, e.g. 0.01)
   Sigmoid    f(x) = 1 / (1 + e^(−x))
   Tanh       f(x) = (e^x − e^(−x)) / (e^x + e^(−x))
inputf(x)f(x) = 0f(x) = xReLU: max(0, x) — negatives clipped to zero, positives pass through

A practical note: ReLU can “die.” If a neuron’s pre-activation is always negative, ReLU outputs 0 and its gradient is 0, so it never updates again. Too-high learning rates make this worse. LeakyReLU (a small slope for negatives), careful initialization, and batch norm all help keep neurons alive.

How a CNN learns

Everything so far is the forward pass — pixels flowing forward into scores. But where do the kernel weights come from? They start as random numbers and are learned: the network repeatedly compares its guess to the true label and nudges every weight to make the error smaller.

From scores to probabilities: softmax

The fully-connected layer outputs raw scores called logits. Softmax turns them into a probability distribution over the classes — every value in (0, 1), all summing to 1:

softmax(z)_i = e^(z_i) / Σ_j e^(z_j)
   logits [2.0, 0.5, -1.0]  →  probs [0.78, 0.17, 0.05]   ("cat" wins)

Measuring the error: cross-entropy loss

Cross-entropy compares the predicted distribution to the true label and collapses it to one number — large when the prediction is confidently wrong, near zero when confidently right:

L = − Σ_i  y_i · log(p_i)        y = one-hot true label, p = softmax output
  = − log(p_correct)             (only the true class survives the one-hot sum)

Learning: backpropagation + gradient descent

Training is one loop, repeated over batch after batch:

  1. forward pass → prediction
  2. loss → a number saying how wrong it was
  3. backpropagation → the chain rule walks the error backward through every layer, computing ∂L/∂w for each weight — how much that weight contributed to the mistake
  4. gradient descent → step each weight against its gradient: w ← w − η · ∂L/∂w (η = learning rate)
1 · Forwardpixels → logits2 · Losssoftmax + cross-entropy3 · Backprop∂L/∂w, chain rule4 · Updatew ← w − η·∂L/∂wtrue label ↓repeat over many batches — the loop is how the kernels get trainedbackprop direction (steps 3–4) runs right-to-left, opposite the forward pass

Over many passes the kernels organize themselves — early ones into edge and texture detectors, deep ones into object-part detectors — with no human ever defining what a “feature” is. That self-organization is the entire payoff of the “one model, raw pixels in” pitch from the top of this post.

Putting it together: reading a VGG

VGG stacks the layers above in a simple, repeating pattern: a few convolution blocks (each = Conv → BatchNorm → ReLU), then a pooling layer to halve the resolution, repeated until the spatial size is small, then a fully-connected layer to classify. Here is a compact implementation, annotated:

from collections import OrderedDict, defaultdict
import torch.nn as nn

class VGG(nn.Module):
    # numbers = output channels of a conv block; 'M' = max-pool (downsample)
    ARCH = [64, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M']

    def __init__(self) -> None:
        super().__init__()

        layers = []
        counts = defaultdict(int)  # gives each layer a unique name: conv0, conv1, ...

        def add(name, layer) -> None:
            layers.append((f"{name}{counts[name]}", layer))
            counts[name] += 1

        in_channels = 3  # RGB input
        for x in self.ARCH:
            if x != 'M':
                # one conv block: extract features → normalize → non-linearity
                add("conv", nn.Conv2d(in_channels, x, 3, padding=1, bias=False))
                add("bn",   nn.BatchNorm2d(x))   # normalization layer
                add("relu", nn.ReLU(True))       # activation function
                in_channels = x
            else:
                add("pool", nn.MaxPool2d(2))     # pooling: halve H and W

        self.backbone   = nn.Sequential(OrderedDict(layers))
        self.classifier = nn.Linear(512, 10)     # fully-connected: 512 features → 10 classes

    def forward(self, x):
        x = self.backbone(x)      # conv/bn/relu/pool stack → feature maps (N, 512, H, W)
        x = x.mean([2, 3])        # global average pool over H, W → (N, 512)
        x = self.classifier(x)    # → (N, 10) class scores
        return x

Fixes from the original notes: count[name] was a typo for counts[name] (the defaultdict we actually defined), and the add helper indexes the same dict it increments — so the order matters: append with the current count, then increment. The imports (OrderedDict, defaultdict, nn) also need to be present for the snippet to run.

Read top to bottom, every layer from this post is in there:

  • Conv2d — feature extraction with weight-shared sliding kernels.
  • BatchNorm2d — normalization to keep activations well-scaled.
  • ReLU — the non-linearity that gives the stack its expressive power.
  • MaxPool2d — downsampling between blocks.
  • x.mean([2, 3]) — a global average pool that collapses the spatial dimensions, the same downsampling idea taken to its limit.
  • Linear — the fully-connected classifier that turns features into 10 class scores.

That is a complete CNN: raw pixels go into forward, and ten numbers — one score per class — come out. The highest one is the network’s answer.

The end-to-end picture

Tracing the ARCH list as 3D volumes makes the whole network legible at a glance. Watch the two trends as data flows left to right: the spatial size shrinks every time a MaxPool2d halves height and width (32 → 16 → 8 → 4 → 2), while the channel depth grows as each conv block learns more features (3 → 64 → 128 → 256 → 512). The network trades where for what: it gives up spatial resolution to build up an ever-richer description of content. At the end, a global average pool flattens the volume to a 512-vector and the fully-connected layer turns it into 10 class scores.

layer →volume →input image32²×3Conv ×2+ BN + ReLU32²×128MaxPool+ Conv ×216²×256MaxPool+ Conv ×28²×512MaxPool+ Conv ×24²×512MaxPool2²×512Globalavg pool512-vecLinear(FC)10 logitsSoftmax→ probabilitiescat .71dog .2610 probs (Σ=1)argmaxcatsoftmax is applied after the network —at inference, or folded into the loss in trainingEach Conv = 3×3, pad 1, stride 1 (size-preserving) + BN + ReLU. Numbers under each cube = output channels.spatial shrinks: 32 → 16 → 8 → 4 → 2 (each MaxPool halves H×W)channels grow: 3 → 64 → 128 → 256 → 512 (more features learned)trade wherefor what

What do the shrinking feature maps mean?

A natural worry: after all that downsampling, does the 2²×512 volume still represent the image? Yes — it still represents it, it just stops looking like it. Representation is not resemblance.

As the spatial grid shrinks, each remaining cell’s receptive field grows, so one cell now summarizes a large patch of the original pixels. And at every cell you don’t have a single number but a 512-long vector — one value per channel — that answers “which of the 512 learned concepts are present in this region?”

8²×512    each cell ≈ a medium patch  →  mid-level concepts (textures, small parts)
4²×512    each cell ≈ a large patch   →  high-level concepts (ears, eyes, paws)
2²×512    each cell ≈ a quadrant      →  near-whole-object concepts
512-vec   one cell = whole image      →  pure "what is it" (no location left)

The network deliberately trades where exactly (pixel positions) for what (object identity): for “is this a cat?” the precise pixel of a whisker doesn’t matter, only that whiskers and ears and fur are present. It’s like summarizing an article into a few keywords — not the article, but still a faithful representation of its meaning.

From the last feature map to the answer: GAP → Linear

The final two steps turn that volume into a prediction.

1 · Global average pooling averages each channel’s 2×2 grid down to one number (this is x.mean([2, 3]) in the code), collapsing 512×2×2 into a flat 512-vector — no parameters, just “how strongly was each feature present on average.”

2 · The Linear layer is one y = W x + b. Each of the 10 outputs is the dot product of one row of W with the whole 512-vector, plus a bias:

x: 512-vec        W: 10 × 512        b: 10-vec        →   y: 10 logits
yᵢ = bᵢ + Σⱼ Wᵢⱼ · xⱼ       (row i of W = the "template" for class i)

   (10 × 512) · (512 × 1) + (10 × 1) = (10 × 1)
       W            x          b          y
weights here: 10 × 512 = 5,120  (+ 10 biases)

A tiny worked version (shrink 512→4, 10→3 so it’s hand-checkable):

x = [1.0, 2.0, 0.5, 3.0]
row "cat"  = [ 0.2, 0.1, 0.0, 0.5],  b=0.1  →  0.1 + (0.2+0.2+0+1.5) = 2.0
row "dog"  = [-0.3, 0.4, 0.2, 0.1],  b=0.0  →  0.0 + (-0.3+0.8+0.1+0.3) = 0.9
row "bird" = [ 0.1,-0.2, 0.6, 0.0],  b=-0.2 →  -0.2 + (0.1-0.4+0.3+0) = -0.2

logits = [2.0, 0.9, -0.2]  → softmax → [0.71, 0.26, 0.08]  → argmax → "cat"

So 2²×512 → GAP → 512-vec → Linear → 10 logits → softmax → label: the global average pool flattens, and the linear layer scores each class as a weighted vote over the 512 features.

Where exactly is softmax applied? It’s the very last step — right after the final linear layer — turning the 10 logits into probabilities that sum to 1, after which argmax picks the winner. One subtlety worth knowing: softmax is not inside the VGG forward() shown earlier. During training, PyTorch’s nn.CrossEntropyLoss applies (log-)softmax internally for numerical stability, so the model deliberately outputs raw logits. You only call softmax yourself at inference, when you want human-readable probabilities. Either way it lives at the very end of the pipeline, never between the conv layers.

Recap

  • CNNs exist to replace hand-built feature engineering with features learned end-to-end from raw pixels in a single model.
  • Every layer is a flavor of output = features · weight + bias.
  • Convolution extracts features while preserving spatial structure; pooling downsamples; normalization keeps activations stable; activation functions add the non-linearity that makes depth worthwhile; the fully-connected layer makes the final call.
  • A conv only shrinks the map by its border (k−1) unless you add padding — padding=1 on a 3×3 keeps the size, so in VGG only pooling downsamples.
  • Deep feature maps still represent the image without resembling it: each cell holds a 512-value “concept vector” over a large receptive field, until the final vector is pure “what is it.”
  • A feature map is a 3D volume (channels × height × width) — a cluster of cubes, one value per cell.
  • The receptive field is the slice of the input that one neuron sees; it grows with depth until deep neurons see the whole image.
  • Grouped convolution wires each output to only a slice of the inputs, cutting parameters by a factor of g; a 1×1 conv mixes channels with no spatial extent.
  • Across the network, spatial size shrinks while channel depth grows — the model trades where for what.
  • The network learns by a loop: forward pass → softmax + cross-entropy loss → backprop → gradient-descent update, repeated over many batches until the kernels become feature detectors on their own.
  • Stack them in a repeating pattern and you get an architecture like VGG — pixels in, label out.