Its differential is \(1\) for values of \(z\) above \(0\), and \(0\) for values of \(z\) below \(0\).
The differential is undefined at \(z=0\), however this is unlikely to occur in practice.
The ReLU activation function induces sparcity.
Its derivative is the sigmoid function:
The softplus function is a smooth approximation of the ReLU function.
Unlike the ReLU function, Softplus does not induce sparcity.