Neural Networks on an FPGA and the Exploration of Hardware Friendly Activation Functions

Document Type

Article

Publication Date

12-31-2020

Publication Title

Journal of Computer and Communications

Volume

8

Issue

12

First page number:

251

Last page number:

277

Abstract

This paper describes our implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset—the Modified National Institute of Standards and Technology (MNIST) database. We also propose a novel hardware-friendly activation function called the dynamic Rectifid Linear Unit (ReLU)—D-ReLU function that achieves higher performance than traditional activation functions at no cost to accuracy. We built a 2-layer online training multilayer perceptron (MLP) neural network on an FPGA with varying data width. Reducing the data width from 8 to 4 bits only reduces prediction accuracy by 11%, but the FPGA area decreases by 41%. Compared to networks that use the sigmoid functions, our proposed D-ReLU function uses 24% - 41% less area with no loss to prediction accuracy. Further reducing the data width of the 3-layer networks from 8 to 4 bits, the prediction accuracies only decrease by 3% - 5%, with area being reduced by 9% - 28%. Moreover, FPGA solutions have 29 times faster execution time, even despite running at a 60× lower clock rate. Thus, FPGA implementations of neural networks offer a high-performance, low power alternative to traditional software methods, and our novel D-ReLU activation function offers additional improvements to performance and power saving.

Keywords

Deep learning; D-ReLU; Dynamic ReLU; FPGA; Hardware acceleration; Machine learning; MLP; Activation function

Disciplines

Electrical and Computer Engineering | Electrical and Electronics | Engineering

Language

English

UNLV article access

Search your library

Share

COinS