LRTanH: Substitution for the Activation Function Derivative during Back Propagation

Back-propagation algorithm is one of the key stages of neural network training. At this stage, the derivation of an activation function is employed to adjust the weights of neurons in the hidden layer. A successful back-propagation depends on a good activation function to converge desired values and minimize the classification error. This paper presents a new approach for training on neural networks by substituting a different activation function derivative to investigate its affects during back-propagation. We modify the derivation of hyperbolic tangent activation function (TanH) by adding a variable that depends on a linear model. Our modified function is called Linearized Hyperbolic Tangent function (LRTanH). We tested LRTanH on MNIST Digits and MNIST Fashion datasets. LRTanH increased accuracy to 88.4% versus 81.06% with 64 nodes on MNIST Digits dataset.