energy-based models

EBMs stand in contrast with causal models

Goal: minimize the energy of a system according to some energy function
The core use-case is learning a good representation of something. A representation is good if you can do useful stuff with it.

$$E(x,y)=\kappa (x,y)$$

… wehere $\kappa$ is some sort of compatibility measure (I picked $\kappa$ arbitrarily as symbol).
$E(x,x)=0$
$-E(x,y)=\mathcal{L}(x,y)$ … energy is just the like the negative loss. We use energy in in the test-time compute context. The difference with EBM is, that we learn the energy function $E_{\theta }\mapsto \mathbb{R}$. So it’s like a reward model?

$f(\cdot )=E(\mathcal{X},\mathcal{Y})$ … energy landscape, which you want to minimize.

EBMs are a generalization of diffusion, where diffusion models amortize the backward pass of an EBM essentially.

The EBM outputs the “loss” (which you backpropagate through during inference time) while the diffusion directly outputs the gradient to transform the datapoint.

Inference by energy minimization is intrinsically more powerful than inference by feed-forward propagation. - Lecun

“EBTs can technically continue for an unbounded amount of time, meaning they can literally compute problems a feed-forward transformer cannot.” - Author

How to train an EBM?

0]

Make input X corresponding answer pair Y should produce 0 energy

This alone would make it collapse (always predict 0 energy).
→ Need a way to make the energy higher everywhere X!=Y

1]

EBL using contrastive learning (showing negative pairs, where energy should go up)

$\mathcal{L}=E_{\theta }(x_1,y_1)+E_{\theta }(x_2,y_2)-E_{\theta }(x_1,y_2)-E_{\theta }(x_2,y_1)$, for unrelated $y_1,y_2$
Curse of dimensionality: infeasible to collect negative examples to push the energy down everywhere for high dim spaces.
You get a jagged energy landscape, with occasional troughs around datapoints.

2]

regularizing methods that minimize the volume of space that can take low energy

• LLMs do this too actually, but very implicitly (finite probability/tokens budget).
  • There is no latent variable $z$ which can vary the input / optimize it over the joint probability of symbols in a text, but LLMs factorize the probabilities in terms of conditional probabilities over successive tokens.
  • What you want to do for LLMs is not to search for the string of tokens that minimizes the energy, but the abstract representation that minimizes it, given a reward model for good answers.