Detection of zero-day malware based on the analysis of opcode sequences

Today, rapid growth in the amount of malicious software is causing a serious global security threat. Unfortunately, widespread signature-based malware detection mechanisms are not able to deal with constantly appearing new types of malware and variants of existing ones, until an instance of this malware has damaged several computers or networks. In this research, we apply an anomaly detection approach which can cope with the problem of new malware detection. First, executable files are analyzed in order to extract operation code sequences and then n-gram models are employed to discover essential features from these sequences. A clustering algorithm based on the iterative usage of support vector machines and support vector data descriptions is applied to analyze feature vectors obtained and to build a benign software behavior model. Finally, this model is used to detect malicious executables within new files. The scheme proposed allows one to detect malware unseen previously. The simulation results presented show that the method results in a higher accuracy rate than that of the existing analogues.