>[!tldr] Activation function >An *activation function* $a: \mathbb{R} \rightarrow \mathbb{R}$ gets applied in a [[Perceptron (neural network)|perceptron]] after the weighted sum of the inputs. It is the non-linear term. Classic activation functions are >- Identity function, >- [[Binary step]], >- [[Sigmoid function]], >- [[Rectified linear unit (ReLU)]], or >- [[Hyperbolic tangent (tanh)]].