Neural Networks on an FPGA and the Exploration of Hardware Friendly Activation Functions
Document Type
Article
Publication Date
12-31-2020
Publication Title
Journal of Computer and Communications
Volume
8
Issue
12
First page number:
251
Last page number:
277
Abstract
This paper describes our implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset—the Modified National Institute of Standards and Technology (MNIST) database. We also propose a novel hardware-friendly activation function called the dynamic Rectifid Linear Unit (ReLU)—D-ReLU function that achieves higher performance than traditional activation functions at no cost to accuracy. We built a 2-layer online training multilayer perceptron (MLP) neural network on an FPGA with varying data width. Reducing the data width from 8 to 4 bits only reduces prediction accuracy by 11%, but the FPGA area decreases by 41%. Compared to networks that use the sigmoid functions, our proposed D-ReLU function uses 24% - 41% less area with no loss to prediction accuracy. Further reducing the data width of the 3-layer networks from 8 to 4 bits, the prediction accuracies only decrease by 3% - 5%, with area being reduced by 9% - 28%. Moreover, FPGA solutions have 29 times faster execution time, even despite running at a 60× lower clock rate. Thus, FPGA implementations of neural networks offer a high-performance, low power alternative to traditional software methods, and our novel D-ReLU activation function offers additional improvements to performance and power saving.
Keywords
Deep learning; D-ReLU; Dynamic ReLU; FPGA; Hardware acceleration; Machine learning; MLP; Activation function
Disciplines
Electrical and Computer Engineering | Electrical and Electronics | Engineering
Language
English
Repository Citation
Si, J.,
Harris, S. L.,
Yfantis, E.
(2020).
Neural Networks on an FPGA and the Exploration of Hardware Friendly Activation Functions.
Journal of Computer and Communications, 8(12),
251-277.
http://dx.doi.org/10.4236/jcc.2020.812021
