ResNet

motivation / intuition

Learning $y=x+f(x)$ is easier than learning $y=f(x)$
Instead of learning the mapping $x\mapsto y$, we only learn the small difference / “residual”.

To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.

• Most of the time, the input will be close to the output (think resolution upscaling task).
• The network does not need to worry about memorizing the input.
• vanishing gradients are prevented

Classic resnet block, size is preserved:

Increasing the stride (and proportionally also the conv channels) reduces the spatial dim and is often done in classification.

Resnet is a discrete, less general case of NODEs, it is an euler integrator, but there are much better ones:

overview and intuition

A ResNet can be thought of as a discrete-time analogs to an ODE:

$$h_{t+1}=h_t+f(h_t,{\theta }_t)$$

You update the representation in discrete steps layer by layer. Neural ODE views this as an infinite process with continous time.

The regular NN has a discrete gradient field, whereas the gradient field of the ODE Network is continous. The black dots are data points.

Again, with resnet you only model the distance (residual) of

$$x_{k+1}=x_k+\underset{\text{residual}}{\underbrace{f(x_k)}}$$

… this is a textbook euler numerical integrator (However euler integration, esp. with large time steps is a bad numerical integration method). So the resnet is doing euler integration over a vector field $f$, so what the neural network is learning, $x$, can also be thought of as a vector field, a differential equation $\dot{x}=f$

With NODE, we model the differential equation itself, instead of just a one-timestep-update or flow-map of the differential equation.

$$\frac{d}{dt}x=f(x)$$

So it is a generalization of resnets to continuous time, where you also use a fancier integrator and smallar or arbitrary time-steps to model the vector field very accurately.
The mathematician’s solution to this would be:

$$x_{k+1}=x_k{\int }_{t_k}^{t_{k+1}}f(x(\tau ))d\tau$$

That’s hard to compute, hence we approximate it (and you can use any integrator for that).

---

Classic dynamical system:

$$\dot{x}=f(x)⟹x(t_0+\Delta )=\underset{\text{flow map, }\varphi }{\underbrace{x(t_0)+{\int }_{t_0}^{t_0+\Delta t}d\tau }}$$

With NODE, $f$ is parametrized by a neural network

$$\dot{x}(t)=\underset{\text{neural net}}{\underbrace{f_{\theta }(x(t),t)}}⟹x(\tau )=\underset{\text{used in loss function, }\mathcal{L}}{\underbrace{\Phi (x(t_0),t_0,\tau ;\theta )}}$$

Free parameters:
$f_{\theta }\dots$ network parameters
$t_0\dots$ initial condition
$\Delta t\dots$ timestep

Hidden state: $x(\tau )$
… data is samples at discrete points in time (doesn’t need to be uniformly in time), so between any two data / measurement points, there is a continous state and the flow map integrates along that hidden state $x(\tau )$ from $t_0$ to $t_0+\Delta t$.
And we want information about this hidden state, which is not in the training data!

$$\underset{\text{lagrange multiplier}}{\underbrace{a(t):=-\frac{\partial \mathcal{L}}{\partial x(t)}}}⟹\frac{d}{dt}a(t)=-a(t)\underset{\text{auto-diff}}{\underbrace{\frac{\partial f_0}{\partial x(t)}}}$$

TODO

steve skips over the details here “lagrange multiplier, adjoint equation”, “we been knowing how to do this the past 20 years”, … TODO steve’s optimization bootcamp)

tldr: we can keep track of the $a$ variable which keeps track of $x(\tau )$ with AD, in the paper called reverse-mode differentiation

Link to original

References

Official pytorch code
Residual Connection | PapersWithCode