A normality based method for detecting kernel rootkits

Rootkits are stealthy, malicious software that allow an attacker to gain and maintain control of a system, attack other systems, destroy evidence, and decrease the chance of detection. Existing detection methods typically rely on a priori knowledge and operate by either (a) saving the system state before infection and comparing this information post infection, or (b) installing a detection program before infection. This approach focuses on detection using reduced a priori knowledge in the form of general knowledge of the statistical properties of broad classes of operating system/architecture pairs. A modified normality based approach proved effective in detecting kernel rootkits infecting the kernel via the system call target modification attack. This approach capitalizes on the discovery that system calls are loaded into memory sequentially, with the higher level calls, which are more likely to be infected by kernel rootkits loaded first, and the lower level calls loaded later. In the single case evaluated, the enyelkm rootkit, neither false positives nor false positives were indicated. The enyelkm rootkit was selected for analysis since it infects the Linux kernel via the system call target modification attack, which is the subject of this research.