CIV4100: Autonomous Vehicle Systems Assignment 2

Overview

This assignment includes the development and testing of the deep learning-based perception modules in autonomous vehicles (AV).  The assignment makes use of the  accumulated  knowledge and experiences learned in the course up to Week 11 of the semester.

Tasks:        There are three parts in this assignment:

Part 1: Developing a deep learning-based perception model for AV;

Part 2: Testing, adversarial attacks, and defence for perception system; and

Part 3: Reporting and recording.

Resources: There are two datasets in this assignment:

Dataset:     The dataset traffic_sign_final_dataset.zip will be loaded automatically in the

Jupyter code or can also be downloaded (see Section A1).

Kaggle evaluation data: The evaluation dataset Kaggle_dataset.zip will be loaded automatically in the Jupyter code or can also be downloaded (see Section A1).

Reference:Convolutional Neural Network, andGoogle Colab

Important Note

This assignment is an individual assignment.

Submissions

You must submit the followings:

.  Your complete Jupyter (Python) scripts and models’ performance (Sections A2, A3). .  Your complete report using the Microsoft Word template provided (Section A4).

.  The video recording showing your codes, main results and comments (Section A5)

Part 1. Develop a deep learning-based perception model for AV (40%)

In this Part, you are required to develop a Python script to analyse the given dataset used for the AV perception. You are then required to build a CNN model, improve its accuracy, submit results in Kaggle platform for ranking and discuss the finding in the report and recorded video (see Section 3).

Task  1.1:  Dataset  processing  and  analysis  (10%).  Process  the  data  and visualize the  data including the unique class images, numbers of counts for each class and distribution of image sizes. Read the inputs (i.e., images) into the format usable by the model, and then split them into the training, validation and testing data subsets with the ratio of 8:1:1.

Task 1.2: Deep learning model development (10%). Develop and train a simple CNN-based perception model (e.g. VGG variant) for detecting signboards using the training and validation data subsets. Then evaluate the obtained CNN model’s performance using the testing data subset by means of accuracy, precision and F1 scores.

Task 1.3: Model improvement (20%). Calibrate and finetune the CNN model obtained in Task 1.2 or develop a whole new model to improve accuracy. Use the improved model to predict the label of all the images in the Kaggle evaluation dataset and save them in a CSV file following the format below

The accuracy can then be checked by submitting your CSV file to theKaggle platform(see Section A3) which will rank your performance against your peers in the class.

Part 2. Testing, adversarial attacks and defence (40%)

In this Part, you are required to undertake the following tasks.

Task 2.1: Testing and validation (15%). You are required to write functions to generate new test cases and implement the testing of the model in Task 1.2 using both traditional testing (using ground- truth as test oracle) and metamorphic testing (having no test oracle). A set of at least 20 source (original) test cases should be used to test the model in Task 1.2. Discuss the findings.

Task 2.2. Adversarial attacks (10%). You are required to undertake the adversarial attack on the model in Task 1.2 using the training data subset in Task 1.1 and one of the adversarial schemes with its default parameters learned in Week 10. Evaluate the effectiveness of the attack and discuss the findings.

Task 2.3. Defence against adversarial attacks (15%). You are required to implement two different defence methods to improve the robustness of your perception system against the adversarial attack in Task 2.2 using the training data subset in Task 1.1, and evaluate their performance on the testing data subset in Task 1.1. Record the results in two tables (see the Report Template) and discuss the findings.

Part 3. Reporting and recording (20%)

You are required to prepare a report (using the template in Section A4) and a video recording (see Section A5).

In the report, you must present your approach to solve the tasks, outline the process, summarise the main findings and discuss their insights in each part (Part 1 and Part 2). The report should not exceed 10 pages excluding the references and appendices.

In the video, you should (a) select or put boxes around different parts of the codes to highlight and record a walkthrough to explain your approach for the codes  (e.g., the chosen algorithm, the functions used, the optimization applied, etc.) and (b) demonstrate their successful execution. The video should not exceed 3 minutes (otherwise, mark penalty might be applied).

Appendix & instructions

A1. Dataset

The  dataset  traffic_sign_final_dataset.zip  will  be  loaded  automatically  in  the  Jupyter  code. Alternatively, you can also download it directly from:

https://onedrive.live.com/download?cid=475DAB8C26138376&resid=475DAB8C26138376%21941 &authkey=AAwDwhuIU599rTg

Kaggle evaluation data: The dataset Kaggle_dataset.zip will be also loaded automatically, or you can download it directly from:

https://onedrive.live.com/download?cid=475DAB8C26138376&resid=475DAB8C26138376%21968 &authkey=APkizHowCwOku4E

A2. The Jupyter code template

Please use to thisJupyter code templateto complete

your Assignment 2.

You must save a copy of your own to use by selecting

File -> Save a copy in your Drive (see screenshot

below). This Jupyter script works well in Google Colab,

and   hence  you   may  want  to  create  an  account

https://colab.research.google.com/instead of setting up

a Jupyter-based system in your computer.

A3. The Kaggle

To complete Task 1.3, you are required to submit the model output in the Kaggle website below

https://www.kaggle.com/t/9c643b837d884651a08e4afd831119ef

You must use your Monash email to register an account and use it for submission. Using non- Monash email to create an account and/or to game the submission system is not allowed and will be regarded as a breach of the Monash Assessment and Academic Integrity Policy.

You must  provide this  Kaggle account  information  in the  Report and demonstrate the  model’s evaluation with this account in the video. You can submit the output more than one time; but note that only a maximum of 5 submissions are allowed per day, and hence you may want to use it wisely.

A4. The report templates

You are required to use the provided template for the completion of this assignment. Please refer to the file CIV4100-Assignment2-ReportTemplate.docx given in Moodle.

A5. The video submission

Please ensure that you submit the video in MP4 format, and keep it under 3 minutes in duration. You may use Zoom to record your screen or any other method of your choice. For optimal viewing experience, please aim for a high resolution and clear audio quality.