Overview

There are three tasks in this project: Task I aims to develop your skills in applying unsupervised machine learning techniques for anomaly detection. Task II helps you better understand how to use gradient descent-based methods to generate adversarial examples against supervised learning models beyond the computer vision domain. In Task III, you are asked to read and review a recent paper on adversarial machine learning.

Specifically, (1) for Tasks I and II, two network traffic (NetFlow) datasets are provided, one for each task. Both datasets contain botnet traffic and normal traffic. You need to identify botnet IP addresses from both two datasets. (2) For Task II you also need to choose a botnet IP address, and explain how to manipulate the corresponding raw network traffic records in order to bypass detection. (3) Each student has been assigned a paper for Task III, which will be sent individually via email.

Deliverables

1. Task I – Source code (Python) and SPL queries used to do the following:

a. Generate/Select features from the packet capture files (training and test datasets) using Splunk. You can use apps such as Splunk Machine Learning Toolkit, but all features have to be generated/selected within Splunk.

b. Use two alternative feature generation/selection methods (filter-based, wrapper- based, etc.) to select features from packet capture files (training and test datasets).

c. Use Python/Splunk to build six models: apply two different anomaly detection techniques on each of the three set of features generated/extracted from 1.a. and 1.b.

d. Score the test data such that cyberattacks are assigned the highest (or lowest)1

scores.

e. Return the IP addresses of attackers and the timestamps of their first and last attempt for attacking the network service (per attack scenario).

f. Compare and discuss the results from different feature extraction and different anomaly detection techniques.

g. Prepare a TXT file including all stream ID which your program classifies as attack traffic, separated by newlines (i.e., one stream ID in each line).

2. Task II

a. Source code in Python, including:

i. Building, training and testing the supervised learning model.

ii. Generating adversarial examples for a chosen botnet IP address, i.e., how to modify its feature values.

b. Explain how to change the raw traffic sent from/to the chosen botnet IP address, in order to reflect the modified feature values. For example, the following six features are extracted for each IP address: (1) mean outbound packet size, (2) variance of outbound packet size, (3) mean packet count per second, (4) max packet count per second, (5) mean of packet jitter, (6) variance of packet jitter. A supervised model  is trained on these features to decide whether an IP address is malicious. You find that by manipulating the values of the third and fourth features, a botnet IP address is labelled as “normal” by the model. Then how do you change the raw traffic records so that they are consistent with the modified feature values? For instance, if 1000 raw traffic records were related to the bot, do you change all 1000 records, or only a subset, e.g., 100/200 of them? How do you change each of the selected traffic records?

* Note that for Task II, (1) the model is trained to classify each IP address, NOT each traffic record, as demonstrated in the above example. (2) The focus is not to train an accurate detection model (i.e., do not spend too much effort on improving model performance), but to understand the difference of generating adversarial examples in domains other than computer vision: in the vision domain, raw pixels are often taken as input, and attackers can directly manipulate them. However, in other domains such as cyber security, raw data cannot be fed into a model directly, and instead features need to be extracted first. Therefore, although it would not be difficult to know how to manipulate the features to bypass detection, there will be different ways to change the raw traffic records, in order to be consistent with the modified feature values and without affecting the botnet functionality.

3. Task III. In this task, you will learn how to write a review for an academic paper. Typically, a review should include the following parts:

a. Summary. Your review starts with a brief summary of the main ideas of the paper. It helps meta-reviewers, program chairs and the authors to determine whether there are any misunderstandings.

b. Merits. List the main contributions of the paper in this section. Contributions can be theoretical, methodological, algorithmic, empirical, etc.

c. Main review. Provide a thorough review of the paper, including:

i. Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? Is it clear how this work differs from previous contributions? Is related work adequately cited?

ii. Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

iii. Clarity: Is the submission clearly written? Is it well organized? If not, please make constructive suggestions for improving its clarity.

iv. Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

*Note that (1) the questions listed in c.i -- c.iv are for explanation only. DO NOT write the main review in the form of Q&A. Write it like an essay instead. (2) Some papers include appendix, which may include proofs, additional experimental settings and results. The appendix helps you better understand the paper, but your review should focus on the main part of the paper.

For Tasks I & II:

4. A README that briefly details how your program(s)/queries work(s). You may use any external resources for your program(s) that you wish. You must indicate (cite) what these external resources are and where you obtained them, in the README file.

*Note: please submit a separate README file for Tasks I & II.

Technical Report

A technical report of around 2000 words comprising:

Task I:

1. An overview of the test dataset using Splunk and explaining feature generation/selection using SPL queries and Splunk native functionalities.

2. Description of your methodology for generating features. Briefly explain your method for the first project, and discuss your modifications and new findings in Project 2.

3. Review of at least two anomaly detection methods that you have used.

4. Description of the experimental setup and evaluation of the (two) methods in detecting anomalies on the test datasets using features generated in Splunk and also features generated using alternative methods. Description should also comprise IP addresses of attacker(s) and victim(s), the attacked service(s), the timestamp, and the type of the attack per attack scenario identified.

5. Description of your final CSV file, the scoring and thresholding technique you used for detecting the reported anomalies2.

6. Conclusion and discussion: describe anomaly detection method worked best given the attack scenario.

Task II:

7. Explanation of the generated features and your choice of supervised learning model. Note that supervised learning is used here, and the mode is the target against which adversarial examples will be generated.

8. Choosing one IP address classified as botnet by your model, and explaining:

a. How to perturb its features via gradient descent-based method to bypass the detection of your model;

b. How to change the raw network traffic sent from/to it, in order to be consistent with the modified feature values and without affecting the botnet functionality.

You should include a bibliography and citations to relevant research papers and external resources and code you have used.

Review

A review of 400 – 500 words of the assignment paper.

1. Summary. This part should contain no more than three sentences. Please be brief, but specific.

2. Merits. List the top three or more main contributions.

3. Main review. In order for your reviews to provide useful feedback to authors, write this section in a top-down manner and start from the most important aspects. Your arguments should be objective, specific, concise and polite.

Assessment Criteria

Code quality and README (2 marks) Technical report (17 marks)

1. Methodology: (4 marks)

You will describe your methodology in a manner that would make your work reproducible. You should describe in detail:

Tasks I and II

a. The features that were generated and/or selected.

Task I

a. The training data that was used to learn the anomaly detection models. You should explain how the parameter settings for your methods were performed (e.g., setting the ������ parameter in OCSVM3). You should not use the test data for setting the parameters.

b. The scoring that was performed in each model to rank the data instances.

c. The thresholding on the scores that was performed in each model to label the attacks.