Fast and Accurate Machine Learning-based Malware Detection via RC4 Ciphertext Analysis

Document Type

Conference Proceeding

Publication Date


Publication Title

2019 International Conference on Computing, Networking and Communications (ICNC)


Springer, Cham

First page number:


Last page number:



Recent malware increases its viability by employing ciphers which help to hide malicious intention and/or behavior against detection schemes. So far, many efforts have been made to detect malware and to prevent it from damaging clients by monitoring network packets. However, these conventional detection schemes tend to treat an encrypted packet as legitimate due to the hardness of extracting information from ciphertexts. Cryptoanalysis of each packet flowing over a network might be one feasible solution to the problem. However, this approach is computationally expensive and lacks accuracy, and thus it is consequently not a practical solution. To address the problem, we firstly introduce a discovery that a fixed encryption key generates unique statistical patterns on RC4 ciphertexts. To the best of our knowledge, this unique signature has never been discussed in the literature. Then, we propose a machine learning-based detection scheme that can identify malware packets efficiently and accurately by leveraging the discovery. The proposed scheme directly analyze network packets without decrypting ciphertexts. Moreover, our analysis demonstrates the proposed scheme requires only a tiny subset of the network packet.


Detection of Encrypted Malware; Statistical Weakness of RC4; Machine Learning; Ciphertext Analysis; Malware Detection


Computer Sciences | Physical Sciences and Mathematics



UNLV article access

Find in your library