A Novel Online and Non-Parametric Approach for Drift Detection in Big Data

Authors

Moinak Bhaduri, University of Nevada, Las VegasFollow
Justin Zhan, University of Nevada, Las VegasFollow
Carter Chiu, University of Nevada, Las Vegas
Felix Zhan, University of Nevada, Las Vegas

Document Type

Article

Publication Date

1-1-2017

Publication Title

IEEE Access

Volume

5

First page number:

15883

Last page number:

15892

Abstract

A sizable amount of current literature on online drift detection tools thrive on unrealistic parametric strictures such as normality or on non-parametric methods whose power performance is questionable. Using minimal realistic assumptions such as unimodality, we have strived to proffer an alternative, through a novel application of Bernstein's inequality. Simulations from such parametric densities as Beta and Logit-normal as well as real-data analyses demonstrate this new method's superiority over similar techniques relying on bounds, such as Hoeffding's. Improvements are apparent in terms of higher power, efficient sample sizes, and sensitivity to parameter values. © 2013 IEEE.

Keywords

Change point detection; Non-parametric methods; Hoeffding's inequality; Big data; Anomaly detection

Language

english

Repository Citation

Bhaduri, M., Zhan, J., Chiu, C., Zhan, F. (2017). A Novel Online and Non-Parametric Approach for Drift Detection in Big Data. IEEE Access, 5 15883-15892.
http://dx.doi.org/10.1109/ACCESS.2017.2735378