A Brief History Of Machine Learning In Cybersecurity

How to connect all the dots in a complex threat landscape

As the volume of cyberattacks grows, security analysts have become overwhelmed. To address this issue, developers are showing more interest in using Machine Learning (ML) to automate threat-hunting. In fact, researchers have tried to implement ML in cybersecurity solutions since the late 1980s, but progress has been slow. Today, ML is showing increasing promise with the advent of Big Data because the quality of information from which ML can learn is improving. However, there is much more to be done.

Anomaly Detection – The Early Days

When we talk about security, we want a system that can separate good from bad, normal from abnormal. Therefore, it is quite natural to apply anomaly detection to security. We can trace the beginning of anomaly detection back to 1987₁ when researchers started building intrusion detection systems (IDS). Around 1998-1999, DARPA (the government agency that created the Internet), created benchmark sets and called for research on ML methods in security₂. Unfortunately, few of the results were practical enough and even fewer products got to the operational stage.

Anomaly detection is based on unsupervised learning, which is a type of self-organized learning that helps find previously unknown patterns in a data set without the use of pre-existing labels. In essence, a system based on unsupervised learning knows what is normal, and identifies anything abnormal as an anomaly. For example, an IDS might know what ‘normal’ traffic looks like, and it will alert on any traffic variants that don’t match that knowledge such as a vulnerability scanner. In short, anomaly detection systems based on unsupervised learning make a binary decision (normal/abnormal) and don’t make sophisticated evaluations. Some refer to unsupervised learning applications as ‘one-class problems.’