Topics in Low-Level Reverse Engineering, with Applications to Software Security

Abstract : The goal of the project was to investigate automated techniques for analyzing computer malware codes, so as to simplify and accelerate the process of penetrating the defenses mounted by such malware to prevent analysis and extracting the internal logic of the malware. The investigators focused on analysis techniques that did not require the execution of the malware code. The project resulted in the development of a number of techniques for the analysis of executable files, including: a theoretical model for reasoning about malware code that modifies itself as it runs; an approach to automatically identify anti-analysis defenses in malware codes; an approach to automatically identify and emulate the code that performs the actual decryption of the malware code, and thereby extract the malware code; and an approach to detect possible errors in the instruction sequence obtained from examining a malware executable file. These results formed substantial components of one PhD dissertation and one MS thesis in Computer Science.