SnortML

SnortML is a new machine learning-based detection engine for Snort that uses neural network models instead of rules.

At a high level, there are two components to this new detection engine. The first component is the snort_ml_engine itself, which loads pre-trained machine learning models, instantiates classifiers based on these models and then makes the classifiers available for detection. The second is the snort_ml inspector, which subscribes to data provided by Snort service inspectors, passes the data to classifiers, and then acts on the output of the classifiers.

Currently, the snort_ml_engine module only has one model type, namely the http_param_model, but we plan on building other models in the future.

This http_param_model is used for classifying HTTP parameters as malicious or normal. Once the snort_ml_engine loads the http_param_model, it can be used in the snort_ml inspector to detect exploits.

The inspector subscribes to the HTTP request data provided by the HTTP inspector through the publish/subscribe interface. It then passes this data (HTTP URI query and optionally HTTP POST body) to a binary classifier based on the http_param_model. This classifier then returns the probability that it saw an exploit. Based on this probability, SnortML can generate an alert, similar to a Snort rule alert, which can be configured to block malicious traffic.

Now that you know how the machine learning engine works, let’s get into how the models work.

SnortML models are designed to be extremely flexible, much like their Snort rule counterparts. To that end, we based our models and our inference engine on TensorFlow. The TensorFlow project is a free and open-source library for machine learning and artificial intelligence.

Any TensorFlow model can be a SnortML binary classifier model so long as it satisfies three conditions, namely, the model must have a single input tensor and a single output tensor, the input and output tensor types must be 32-bit floating point, and finally, the output tensor must have only a single element. We plan on adding other model types in the future (including multiclass classifiers), but right now, this is the only model type currently supported. The SnortML engine uses TensorFlow through a support library we call LibML. The LibML library handles loading, configuring and running machine learning models for Snort. It also includes the XNNPACK accelerator needed to run CPU-bound models at line rate. The easiest way to build a SnortML model is to use the TensorFlow Keras API. If you are new to machine learning, don’t worry, Keras is a simple but powerful deep-learning framework that allows you to build neural networks and train them in a few lines of Python. To get started, import the following:

import os
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers
from urllib.parse import unquote_to_bytes

We are going to train our example model on just two samples, but a real production model would use far more:

# Example data

data = [
    { 'str':'foo=1', 'attack':0 },
    { 'str':'foo=1%27%20OR%201=1%2D%2D', 'attack':1 }
]

The next thing we need to do is prepare our data. SnortML models expect input data to be zero-padded which is what we are going to do here:

# Prepare Data
maxlen = 1024
X = []
Y = []

def decode_query(str):
    return unquote_to_bytes(str.replace('+',' '))

for item in data:
    arr = decode_query(item['str'])[:maxlen]
    arrlen = len(arr)
    seq = [0] * maxlen
    for i in range(arrlen):
        seq[maxlen - arrlen + i] = arr[i]
    X.append(seq)
    Y.append(item['attack'])

Now, we need to construct a neural network that can classify our data. This example uses a simple LSTM (Long Short-Term Memory) network, but other combinations of layers available in Keras work here as well. LSTM is a type of neural network that is keenly suited to identify patterns in sequences of data, such as the sequences of bytes in HTTP parameters. To translate the bytes on the wire to tensors that the LSTM can accept, we can place an embedding layer in front of it. Embedding layers are a kind of association layer, they can learn relationships between input data (bytes in our case) and output those relationships as tensors that the LSTM neurons can accept.

Finally, we will converge the output of our LSTM neurons to a single output neuron with a Dense layer. This will serve as the output of the neural network.

#
# Build Model (Simple LSTM)
#

model = tf.keras.Sequential([
    layers.Input(shape=(maxlen,), batch_size=1),
    layers.Embedding(256, 32),
    layers.LSTM(16),
    layers.Dense(1, activation='sigmoid')])

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.summary()

Now for the fun part — let’s train this neural network:

#
# Train Model
#

model.fit(np.asarray(X).astype(np.float32),
          np.asarray(Y).astype(np.float32),
          epochs=100, batch_size=1)

Training output:

Model: "sequential"
-----------------------------------------------------------------
Layer (type)                Output Shape              Param #
=================================================================
embedding (Embedding)       (1, 1024, 32)             8192

lstm (LSTM)                 (1, 16)                   3136

dense (Dense)               (1, 1)                    17

=================================================================

Total params: 11,345
Trainable params: 11,345
Non-trainable params: 0
-----------------------------------------------------------------
Epoch 1/100
2/2 [==============================] - 1s 129ms/step - loss: 0.6910 - accuracy: 0.5000
...
Epoch 100/100
2/2 [==============================] - 0s 134ms/step - loss: 0.0208 - accuracy: 1.0000

As you can see above, the accuracy of our network increased, and the loss dropped. These metrics show that the neural network learned to differentiate attack from normal in our example dataset.

Now, let’s save this model to a file so we can load it in Snort:

#
# Save Model
#

model.export('model')

converter = tf.lite.TFLiteConverter.from_saved_model('model')

snort_model = converter.convert()

with open('snort.model', 'wb') as f:
    f.write(snort_model)

Now that we have a model file, we can run it against PCAPs with Snort 3:

$ snort -q --talos \
 --lua 'snort_ml_engine = { http_param_model = "snort.model" };' \
 --lua 'snort_ml = {};' \
 -r test.pcap

##### test.pcap #####
    [411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1)
#####

If you have Snort 3 built with debug messages enabled, you can even trace the ML engine input and output.

Notice that even with variations in the SQL injection attack above, we still detected it.

$ snort -q --talos \
 --lua 'trace = { modules = { snort_ml = { all = 1 } } };' \
 --lua 'snort_ml_engine = { http_param_model = "snort.model" };' \
 --lua 'snort_ml = {};' \
 -r test.pcap

P0:snort_ml:classifier:1: input (query): foo=1' OR 2=2--
P0:snort_ml:classifier:1: output: 0.971977
P0:snort_ml:classifier:1: <ALERT>

##### test.pcap #####
    [411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1)
#####

Snort 3 Rule Writing Guide

SnortML