dropout

Dropout is a regularization technique that introduces dynamic sparcity, turning a fully connected layer into a sparsely connected one.

With dropout, every training step, each entire neuron/activation/output vector has $p$ probability of being $0$.
At inference time, weights are scaled to $w\to pw$.

DropConnect

DropConnect is a regularization technique where each weight+bias has a probability $p$ of being $0$.

$$r=a((M\cdot W)v)$$

$r\dots$ layer output
$v\dots$ layer input
$W\dots$ weight matrix
$M\dots$ binary marix with connection information

$$M_{ij}\sim \text{Bernoulli}(p)$$

References

https://paperswithcode.com/method/dropconnect

Link to original