A dynamic malware analyzer against virtual machine aware malicious software

ABSTRACT Nowadays, cyber‐world is being enriched by a large variety of digital information technology‐based services. An increasing rate of remote and mobile usage leads to a remarkable dependency on information security. Analysis and detection of malicious software or so‐called malware is a challenging task due to the introduction of advanced obfuscation techniques by malware authors. In this study, we mainly concentrate on anti‐virtual machine evasion techniques to provide secure and reproducible environments for malware analysis and its implementation issues. Malwares are identified on the basis of their behaviors by taking precautions related to the anti‐virtual machine detection techniques. The dynamic malware analyzer tool is deployed to execute anti‐virtual machine‐aware malware samples in VMware environment. Dynamic malware analyzer monitors system resources such as connections, processes, windows registry, and file operations. Success ratio of detection is tested by using public malware sets with an accuracy of 92%. The effectiveness and success of the behavior‐based malware analyzer tool is exploited and current state of the art of malware detection schemes is presented. Copyright © 2013 John Wiley & Sons, Ltd.