Every machine learning model requires some type of architecture design and possibly some initial assumptions about the data we want to analyze.
Generally, every building block and every belief that we make about the data is a form of inductive bias.
The more data you have, the less you want to use inductive biases.
Nearest neighbors
Assume that most of the points in a small neighborhood in feature space belong to the same class. Given a point for which the class is unknown, guess that it belongs to the same class as the majority in its immediate neighborhood. This is the bias used in the k-nearest neighbours algorithm. The assumption is that cases that are near each other tend to belong to the same class.
The transformer has a very minimal inductive bias.
In the core transformer inductive biases are mostly factored out. Self-attention is a very minimal (and verry useful) inductive bias, with a most general connectivity, where everything attends to everything..
Without positional encodings, there is no notion of space.
If you want to have a notion of space, or other constraints, you need to specifically add them.
Positional encodings, for example, are a type of inductive bias, same as for example the Swin Transformer, where you limit the attention node-connectivity to local windows, somewhat like the biologically inspired inductive bias of CNNs.
Causal attention is another example of an inductive bias, where tokens can only attend to previous tokens in the sequence.
Link to originalTransformers and The Bitter Lesson (from The Bittersweet Lesson)
→ If it’s possible for a human domain expert to discover from the data the basis of a useful inductive bias, it should be obvious for your model to learn about it too, so no need for an inductive bias.
→ Instead focus on building biases that improve either scale, learning or search .
→ In the case of sequence models, any bias towards short-term dependency is needless, and may inhibit learning (about long-term dependency).
→ Skip connections are good because they promote learning.
→ MHSA is good because it enables the Transformer to (learn to) perform an online, feed-forward, parallel ‘search’ over possible interpretations of a sequence.