GPU-based empirical evaluation of activation functions in convolutional neural networks

Activation functions are important components of Convolutional Neural Networks (CNN) that introduces nonlinearity in the model to compute complex functions. There are different types of activation functions used with CNNs in different applications, however it turns out that an effective activation function yields better results and improves performance of the model. In this study four of the widely used activation functions are chosen to analyze and evaluate to figure out their efficiency in terms of the model's accuracy. Sigmoid, hyperbolic tangent, rectified linear unit (ReLU) and exponential linear unit (ELU) activation functions have been used with most of the successful models. A CNN model has been implemented on the MNIST dataset to perform the analysis task. The experiments have been performed on Nvidia GPU 940MX to accelerate the training and testing of the CNN model. It has been observed that ReLU the most popular activation function performs better than sigmoid and tanh and a recent activation function ELU performs better than ReLU.