activation space

Activation Space

Activation space is the high-dimensional space where the outputs of neurons (their activations) live. For a layer with $m$ neurons, the activation space for that layer is ${\mathbb{R}}^m$. The activation space captures the internal representations learned by the network for different inputs.

While weight space is fixed for a given architecture (depends solely on network topology), activation space is dynamic and depends on both the weights and the input data. The weights determine a mapping from input space to activation space, transforming data representations layer by layer.