Design and Implementation of an Approximate Softmax Layer for Deep Neural Networks

Deep neural networks (DNNs) have been widely used in classification due to their high accuracy. The softmax function is one of the important non-linear functions in DNNs. Therefore, high performance and efficient hardware design are sought. However, the improvement of the softmax function is difficult because the exponent and the division units are complex. In this paper, we propose new approximate hardware architectures for both the exponent and the division units. Compared with the state-of-the-art designs, the proposed approximate softmax design consumes significantly less resources and also achieves high performance while maintaining a very high accuracy.