convolution

Just go read udl book + https://openreview.net/pdf?id=-oeKiM9lD9h, rlly

x = nn.Conv1d(in_channels=1, out_channels=2, dilation=1, stride=1, kernel_size=3, padding=0) # a) + b)
x = nn.Conv1d(in_channels=2, out_channels=1, dilation=1, stride=1, kernel_size=3, padding=0) # c)

Raw convolution operation

def conv1d(  
	input_signal: list[float], kernel: list[float], stride:int=1, dilation: int=1
	) -> list[float]:  
	input_signal = [0] * (len(kernel) - 1) + input_signal + [0] * (len(kernel) - 1)  
	output_size = (len(input_signal) - len(kernel) * dilation) // stride + 1  
	output_signal = [0] * output_size  
	for i in range(output_size):  
		for j in range(len(kernel)):  
			output_signal[i] += input_signal[i * stride + j * dilation] * kernel[j]  
	return output_signal

Properties

Convolutions are translation invariant

Special types of convolutions

1x1 Convolution

E.g.: Input feature map of size (H,W,C ), apply 1x1 convolution with F filters
→ output feature map (H,W,F)

$$x_{i,j}\in {\mathbb{R}}^C\stackrel{W\in {\mathbb{R}}^{F\times C},b}{\to }{y}_{i,j}=W{x}_{i,j}+b\in {\mathbb{R}}^F$$

This is essentially a linear layer applied to the pixel (or token) with dimensionality $C$.
So if you stack 1x1 convs, you get an MLP:

$$x\stackrel{{\text{Linear}}_1}{\to }\varphi \stackrel{{\text{Linear}}_2}{\to }$$

with weights shared across all inputs at each layer.
This keeps translation equivariance compared to regular MLPs, and uses far fewer parameters, while increasing network expressivity.

It mixes info only across channels, wheras regular convs also mix information across space.

It’s often used as a cheap channel reshape operation.

Use when you want to change channel count while preserving spatial structure independently at each position.
Anytime positions don’t need to share information.

1x1 conv is also known as pointwise conv

Network In Network

Depthwise conv = only spatial mixing, no channel mixing

Depthwise separable has depthwise, then pointwise (factorized the channel mixing, for efficiency)

Visualizations

https://medium.com/hitchhikers-guide-to-deep-learning/10-introduction-to-deep-learning-with-computer-vision-types-of-convolutions-atrous-convolutions-3cf142f77bc0

“valid”, p=2, “same”, p=2+stride 1 (or <1?)

Playlist with visualizations of all properties (feature count, kernel size, …).

Halving height and width but doubling number of features → volume cut in half: Net has to compress:

padding="same" → input tensor is padded so that the output tensor has the same resolution.
padding="valid" → there must be no padding pixels in any of the filter window patches

padding styles (5×5 input, 3×3 kernel)

Blue = input, gray = zero-padding, red box = one kernel position.

valid — no padding, kernel only where it fully fits. Output shrinks: $5\times 5\to 3\times 3$.

same — pad 1 ring of zeros (3×3 kernel, stride 1). Output keeps input size: $5\times 5\to 5\times 5$.

full — pad so every input cell is visited by the kernel (ring of $k-1=2$). Output grows: $5\times 5\to 7\times 7$.

causal (1D, kernel size 3) — pad only on the left so output at position $t$ depends only on inputs $\le t$ (no leakage from the future). Used in WaveNet / temporal convs.

Pooling / Downsampling

Reduces resolution of image by taking the max of e.g. 4 pixels (nn.MaxPool2d(kernel_size=4) or averaging them, while keeping the number of features the same.

weight-sharing