few-shot learning

Few shot learning is a training paradigm, where the model is trained on fulfilling tasks given only a few examples, at test time.
In contrast to in-context learning, few-short learning is an explicit training (meta learning) paradigm, while ICL is an emergent capability (from scale, …), without being meta-trained for it.

Few-Shot Classification

Training a classifier

We train a classifier $f_{\theta }$ with parameters $\theta$ on a dataset $\mathcal{D}=\{({\mathbf{x}}_i,y_i){\}}_{i=1}^n$ to output the probability of a datapoint belonging to the class $y$ given the feature vector $x$, $P_{\theta }(y|\mathbf{x})$.
The optimal parameters maximize the probability of the true labels:

$$\begin{array}{r}{\theta }^{\ast }=\arg \underset{\theta }{\max }{\mathbb{E}}_{(\mathbf{x},y)\in \mathcal{D}}[P_{\theta }(y|\mathbf{x})])]\end{array}$$

When training with mini-batches $B\subset \mathcal{D}$:

$$\begin{array}{r}{\theta }^{\ast }=\arg \underset{\theta }{\max }{\mathbb{E}}_{B\subset \mathcal{D}}\left[\sum _{(\mathbf{x},y)\in B}{P}_{\theta }(y|\mathbf{x})\right]\end{array}$$

Link to original

In few-shot classification, we sample two disjoint sets from ${\mathcal{D}}^T\subset \mathcal{D}$ each epoch, where $T\subset \mathcal{T}$ is a subset of labels (tasks) of $D$: A support set $S\subset \mathcal{D}$ and a training batch $B\subset \mathcal{D}$.
The support set is part of the model input. The goal is that the model learns to generalize to other datasets, by learning how to learn from the support set.

$$\begin{array}{r}\theta =\arg \underset{\theta }{\max }{\mathbb{E}}_{T\subset \mathcal{T}}{\mathbb{E}}_{(S,B)\subset {\mathcal{D}}^T}\left[\sum _{(\mathbf{x},y)\in B}{P}_{\theta }(y|\mathbf{x},S)\right]\end{array}$$

How would such a network represent never-seen classes? It would need to output embeddings.