Towards a science of anomaly detection system evasion

A fundamental drawback of current anomaly detection systems (ADSs) is the ability of a skilled attacker to evade detection. This is due to the flawed assumption that an attacker does not have any information about an ADS. Advanced persistent threats that are capable of monitoring network behavior can always estimate some information about ADSs which makes these ADSs susceptible to evasion attacks. Hence in this paper, we first assume the role of an attacker to launch evasion attacks on anomaly detection systems. We show that the ADSs can be completely paralyzed by parameter estimation attacks. We then present a mathematical model to measure evasion margin with the aim to understand the science of evasion due to ADS design. Finally, to minimize the evasion margin, we propose a key-based randomization scheme for existing ADSs and discuss its robustness against evasion attacks. Case studies are presented to illustrate the design methodology and extensive experimentation is performed to corroborate the results.