--- ## Overview A **Convolutional Neural Network (CNN)** is a class of deep neural network that applies the mathematical operation of *discrete convolution* to exploit spatial structure in data. CNNs are characterised by local connectivity, weight sharing, and hierarchical feature extraction. --- ## High-Level Intuition Before diving into the mathematics, it helps to understand *what a CNN is actually doing* at each stage. ### The Core Idea: Scanning for Patterns Imagine you are looking at a photograph of a cat. You don't recognise it by examining every pixel in isolation — you look for **local patterns**: whiskers, eyes, ears. A CNN works the same way. A small matrix called a **kernel** (or filter) is slid across the image. At each position, it computes a dot product with the underlying pixel patch. If the kernel is shaped like a horizontal edge detector, it will produce a high value wherever a horizontal edge exists, and a low value elsewhere. The result is a **feature map** — a spatial map of where that pattern occurs. > [!tip] Analogy > Think of each kernel as a **stamp** being pressed across the image. The feature map is the "impression" left behind — high values where the stamp matched, low values where it didn't. --- ### Overall Architecture A typical CNN stacks multiple types of layers. The image flows through them from left to right, being progressively transformed from raw pixels into a final prediction (e.g. "cat" or "dog"). The key stages are: | Stage | What it does | Intuition | |-------|-------------|-----------| | **Conv layer** | Applies $F$ learnable filters | Detects local patterns (edges, textures, shapes) | | **Activation (ReLU)** | Sets negative values to zero | Introduces non-linearity; keeps strong signals | | **Pooling layer** | Shrinks the spatial size | Keeps the most important signals; adds robustness | | **Fully connected** | Flattens and classifies | Combines all features to make the final prediction | --- ### Hierarchical Feature Learning One of the most powerful properties of CNNs is that they learn features **hierarchically**. Early layers respond to simple, low-level patterns; later layers combine these into complex, high-level concepts. - **Layer 1–2:** Detects oriented edges and colour gradients - **Layer 3–4:** Combines edges into textures and simple shapes - **Layer 5–6:** Assembles shapes into object parts (e.g. eyes, wheels) - **Final layers:** Recognises whole objects and scenes This hierarchy emerges **automatically** from training — the network discovers which features are useful, without being told explicitly. --- ### The Convolution Step, Visually Here is a concrete example of a single convolution step. The full input is $5 \times 5$; the $3 \times 3$ kernel is placed at the top-left corner, operating on that $3 \times 3$ patch (stride = 1, no padding): ``` Input patch Kernel Output value ┌─────────────┐ ┌─────────┐ │ 1 0 1 │ │ 1 0 -1│ │ 0 1 0 │ ✦ │ 1 0 -1│ = (element-wise multiply & sum) → 0 │ 1 0 1 │ │ 1 0 -1│ └─────────────┘ └─────────┘ ``` The kernel slides one step to the right and the calculation repeats, building up the feature map one value at a time. --- ### Pooling, Visually After convolution and activation, **max pooling** down-samples the feature map by taking the maximum value in each non-overlapping $2 \times 2$ window: ``` Feature map (4×4) After 2×2 Max Pooling (2×2) ┌────┬────┬────┬────┐ ┌────┬────┐ │ 1 │ 3 │ 2 │ 4 │ │ 6 │ 4 │ ├────┼────┼────┼────┤ → ├────┼────┤ │ 5 │ 6 │ 1 │ 2 │ │ 3 │ 3 │ ├────┼────┼────┼────┤ └────┴────┘ │ 3 │ 2 │ 3 │ 1 │ ├────┼────┼────┼────┤ │ 1 │ 0 │ 2 │ 1 │ └────┴────┴────┴────┘ ``` This halves the spatial dimensions while retaining the strongest activations. Crucially, if a feature shifts by one pixel, max pooling still returns the same value — giving the network **translation robustness**. --- ### Putting It All Together A CNN trained on images of handwritten digits might learn filters like these in its first layer: - A filter that fires on `/` (diagonal edges going right) - A filter that fires on `—` (horizontal edges) - A filter that fires on `|` (vertical edges) By stacking many such filters across many layers, the network builds up increasingly abstract representations until it can distinguish a `3` from an `8` with high confidence. > [!note] Why not just use a fully connected network? > A fully connected network on a $224 \times 224$ RGB image would require $224 \times 224 \times 3 = 150{,}528$ input weights — *per neuron*. A CNN uses a $3 \times 3 \times 3$ kernel with just **27 weights**, shared across the entire image. This **weight sharing** is what makes CNNs feasible for visual data. --- ## The Convolution Operation ### Continuous Convolution The continuous convolution of two functions $f, g : \mathbb{R} \to \mathbb{R}$ is defined as: $ (f * g)(t) = \int_{-\infty}^{\infty} f(\tau)\, g(t - \tau)\, d\tau $ ### Discrete 2D Convolution For a discrete 2D input (e.g. an image) $I : \mathbb{Z}^2 \to \mathbb{R}$ and a kernel $K : \mathbb{Z}^2 \to \mathbb{R}$: $ (I * K)(i, j) = \sum_{m} \sum_{n} I(i - m,\, j - n)\, K(m, n) $ > [!note] Cross-Correlation vs. Convolution > In practice, CNNs use **cross-correlation**, not strict convolution. The kernel is *not* flipped: > $(I \star K)(i,j) = \sum_{m} \sum_{n} I(i + m,\, j + n)\, K(m, n)$ > The distinction is immaterial during training since the kernel weights are learned, but it is worth noting for mathematical rigour. --- ## A Single Convolutional Layer ### Inputs and Notation | Symbol | Description | |--------|-------------| | $\mathbf{X} \in \mathbb{R}^{H \times W \times C}$ | Input tensor (height × width × channels) | | $\mathbf{K} \in \mathbb{R}^{h \times w \times C \times F}$ | Kernel tensor ($F$ filters of size $h \times w \times C$) | | $\mathbf{b} \in \mathbb{R}^{F}$ | Bias vector | | $\mathbf{Z} \in \mathbb{R}^{H' \times W' \times F}$ | Pre-activation output | | $\sigma$ | Non-linear activation function | ### Forward Pass The pre-activation for output feature map $f$ at spatial position $(i, j)$ is: $ Z_{i,j,f} = b_f + \sum_{c=1}^{C} \sum_{m=0}^{h-1} \sum_{n=0}^{w-1} X_{(i \cdot s + m),\, (j \cdot s + n),\, c} \cdot K_{m,n,c,f} $ where $s$ is the **stride**. The activated output is: $ A_{i,j,f} = \sigma(Z_{i,j,f}) $ ### Output Spatial Dimensions Given padding $p$ and stride $s$: $ H' = \left\lfloor \frac{H - h + 2p}{s} \right\rfloor + 1, \qquad W' = \left\lfloor \frac{W - w + 2p}{s} \right\rfloor + 1 $ --- ## Learnable Parameters The total number of learnable parameters in a convolutional layer (excluding biases) is: $ |\theta_{\text{conv}}| = h \times w \times C \times F $ This is independent of the spatial dimensions $H$ and $W$ — the key advantage of **weight sharing**. --- ## Pooling Pooling reduces spatial dimensions and introduces translation invariance. ### Max Pooling $ P_{i,j} = \max_{(m,n) \in \mathcal{R}_{i,j}} A_{m,n} $ ### Average Pooling $ P_{i,j} = \frac{1}{|\mathcal{R}_{i,j}|} \sum_{(m,n) \in \mathcal{R}_{i,j}} A_{m,n} $ where $\mathcal{R}_{i,j}$ is the pooling receptive field centred at $(i, j)$. --- ## Backpropagation Through a Conv Layer Let $\mathcal{L}$ be the scalar loss. The gradients required for training are: ### Gradient w.r.t. the Kernel $ \frac{\partial \mathcal{L}}{\partial K_{m,n,c,f}} = \sum_{i} \sum_{j} \frac{\partial \mathcal{L}}{\partial Z_{i,j,f}} \cdot X_{(i \cdot s + m),\, (j \cdot s + n),\, c} $ ### Gradient w.r.t. the Input (for upstream layers) $ \frac{\partial \mathcal{L}}{\partial X_{i',j',c}} = \sum_{f} \sum_{m} \sum_{n} \frac{\partial \mathcal{L}}{\partial Z_{\lfloor(i'-m)/s\rfloor,\, \lfloor(j'-n)/s\rfloor,\, f}} \cdot K_{m,n,c,f} $ This operation is equivalent to a **full convolution** of the upstream gradient with the *flipped* kernel. --- ## Full CNN Architecture (Formal Definition) A CNN with $L$ layers can be described as a composition of functions: $ \hat{y} = f_L \circ f_{L-1} \circ \cdots \circ f_1(\mathbf{X}) $ where each $f_\ell$ is one of: - **Convolutional layer:** $f_\ell(\mathbf{A}) = \sigma(\mathbf{A} \star \mathbf{K}^{(\ell)} + \mathbf{b}^{(\ell)})$ - **Pooling layer:** $f_\ell(\mathbf{A}) = \text{Pool}(\mathbf{A})$ - **Fully connected layer:** $f_\ell(\mathbf{a}) = \sigma(\mathbf{W}^{(\ell)}\mathbf{a} + \mathbf{b}^{(\ell)})$ - **Normalisation layer:** $f_\ell(\mathbf{A}) = \text{BatchNorm}(\mathbf{A})$ The network is trained by minimising an empirical risk: $ \mathcal{L}(\theta) = \frac{1}{N} \sum_{k=1}^{N} \ell\!\left(\hat{y}^{(k)},\, y^{(k)}\right) + \lambda\, \Omega(\theta) $ where $\Omega(\theta)$ is a regularisation term (e.g. $\ell_2$: $\|\theta\|^2$) and $\lambda \geq 0$. --- ## Key Properties | Property | Explanation | |----------|-------------| | **Local connectivity** | Each unit depends only on a local patch of the input | | **Weight sharing** | The same kernel $\mathbf{K}$ is applied across all spatial locations | | **Translation equivariance** | $f(T_\delta \mathbf{X}) = T_\delta f(\mathbf{X})$ for shift $T_\delta$ | | **Hierarchical features** | Early layers learn edges; deeper layers learn semantic features | --- ## Common Activation Functions | Function | Definition | Range | |----------|------------|-------| | ReLU | $\sigma(z) = \max(0, z)$ | $[0, \infty)$ | | Sigmoid | $\sigma(z) = \frac{1}{1+e^{-z}}$ | $(0, 1)$ | | Tanh | $\sigma(z) = \tanh(z)$ | $(-1, 1)$ | | GELU | $\sigma(z) = z \cdot \Phi(z)$ | $\approx(-0.17, \infty)$ | --- ## References - LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). *Gradient-based learning applied to document recognition.* Proceedings of the IEEE. - Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning.* MIT Press. [deeplearningbook.org](https://www.deeplearningbook.org) - Dumoulin, V. & Visin, F. (2016). *A guide to convolution arithmetic for deep learning.* arXiv:1603.07285.