Machine Learning Practical 2022/23: Coursework 1

1   Introduction

The aim of this coursework is to study the classification of images of handwritten digits using neural networks. The first part of this coursework will concern the identification and discussion of a fundamental problem in machine learning, as shown in Figure 1. Following this preliminary discussion, you will further investigate this problem in wider and deeper neural networks, study it in terms of network width and depth. The second part involves implementing different methods to combat the problem identified in Task 1 and then comparing these methods empirically and theoretically. In the final part, you will briefly discuss one related work to the methods

examined in Task 2.

The coursework will use an extended version of the MNIST database, the EMNIST Balanced dataset, described in Section 2.  Section 3 describes the additional code provided for the coursework (in branch mlp2022-23/coursework1 of the MLP github), and Section 4describes how the coursework is structured into three tasks. The main deliverable of this coursework is a report, discussed in section 8, using a template that is available on the github. Section 9 discusses the details of carrying out and submitting the coursework, and the marking scheme is discussed in Section 10.

You will need to submit your completed report as a PDF file and your local version of the mlp code including any changes you made to the provided ( .py files). The detailed submission instructions are given in Section 9.2 – please follow these instructions carefully.

2    EMNIST dataset

In this coursework we shall use the EMNIST (Extended MNIST) Balanced dataset [Cohen et al., 2017], https://www.nist.gov/itl/iad/image-group/emnist-dataset. EMNIST extends the well-known MNIST by including images of handwritten letters (upper and lower case) as well as handwritten digits. Both EMNIST and MNIST are extracted from the same underlying dataset, referred to as NIST Special Database 19. Both use the same conversion process resulting in centred images of dimension 28×28.

There are 62 potential classes for EMNIST (10 digits, 26 lower case letters, and 26 upper case letters). However, we shall use a reduced label set of 47 different labels. This is because (following the data conversion process) there are 15 letters for which it is confusing to discriminate between upper-case and lower-case versions. In the

47 label set, upper- and lower-case labels are merged for the following letters:

C,  I,  J,  K,  L,  M,  O,  P,  S, U,  V, W,  X,  Y,  Z.

The training set for Balanced EMNIST has about twice the number of examples as the MNIST training set, thus you should expect the run-time of your experiments to be about twice as long. The expected accuracy rates are lower for EMNIST than for MNIST (as EMNIST has more classes, and more confusable examples), and differences in accuracy between different systems should be larger. Cohen et al. [2017] present some baseline

results for EMNIST.

You do not need to directly download the EMNIST database from the nist.gov website, as it is part of the coursework1 branch in the mlpractical Github repository, discussed in Section 3below.

3   Github branch mlp2022-23/coursework1

You should run all of the experiments for the coursework inside the (mini-)Conda environment you set up for the labs.  The code for the coursework is available on the course Github repository on a branch mlp2022-23/coursework1.  To create a local working copy of this branch in your local repository you need to do the following.

1. Make sure all modified files on the branch you are currently have been committed (see notes/getting- started-in-a-lab.mdif you are unsure how to do this).

2. Fetch changes to the upstream origin repository by running git  fetch  origin

3. Checkout a new local branch from the fetched branch using                     git  checkout  -b  coursework1  origin/mlp2022-23/coursework1

You will now have a new branch in your local repository with all the code necessary for the coursework in it. This branch includes the following additions to your setup:

• A new EMNISTDataProvider class in the mlp .data_providers module.  This class makes some changes to the MNISTDataProvider class, linking to the EMNIST  Balanced data, and setting the number of classes to 47.

• Training, validation, and test sets for the EMNIST  Balanced dataset that you will use in this coursework.

• In order to further improve performance and mitigate the problem identified in neural networks, you will also need to implement a new class in the mlp .layers module:

DropoutLayer

and also two weight penalty tecniques in the mlp .penalties module:

L1Penalty  and  L2Penalty.

• DropoutandPenalty_tests .ipynb Jupyter notebook

to be used for testing the implementations of DropoutLayer, L1Penalty  and  L2Penalty classes. The tests serve as a safeguard to prevent experimentation with faulty code which might lead to wrong conclusions. Tests in general are a vital ingredient for good software development, and especially important for building correct and efficient deep learning systems.

Please note that passing these preliminary tests does not necessarily mean your classes are absolutely bug-free. If you get unexpected curves during model training, re-check your implementation of the classes.

• A directory called report which contains the LaTeX template and style files for your report. You should copy all these files into the directory which will contain your report.

(a) Error curve on the training and validation set of EMNIST dataset.

(b) Accuracy curve on the training and validation set of EMNIST dataset.         Figure 1: Error and Accuracy curves for a baseline model on EMNIST Dataset.

4   Tasks

The coursework is structured into 3 tasks, the first two are supported by experiments on the EMNIST dataset.

1. Identification of a fundamental problem in machine learning as shown in Fig 1 and setting up a baseline system on EMNIST by a valid hyper-parameter search.

2. A research investigation and analysis into whether using Dropout and/or Weight Penalty (L1Penalty and L2Penalty) addresses the problem found in training machine learning models (Fig1). How do these two approaches improve/degrade the model’s performance?

3. A brief literature review of a specific paper as discussed in Section 7 and summarising the whole report.

5   Task 1: Problem identification

Figure 1 shows the training and validation error curves in Figure 1a and also training and validation accuracies in Figure 1bfor a model with 2 hidden layers1with ReLU trained on the EMNIST dataset by using cross-entropy error function. This curve can be re-produced by running the model settings defined in the Coursework1 .ipynb notebook in the github repository. We first identify and discuss the problem shown by the curves in Figure 1 as overfitting, and briefly discuss potential solutions in this section for overcoming this problem.

Varying number of hidden units. Initially you will train various 1-hidden layer networks by using either 32, 64 and 128 ReLU hidden units per layer on EMNIST. Note that 1-hidden layer network contains two layers, one mapping input units to hidden units and another one mapping hidden units to output units. 2 and 3-hidden layer networks would contain 3 and 4 layers respectively. Make sure you use Adam optimizer with the hyperparameters provided in the template and train each network for 100 epochs. Visualise and discuss how increasing number of hidden units affects the validation performance and whether it worsens or mitigates the overfitting problem.

Varying number of layers. Here you will train various neural networks by using either 1, 2, 3 hidden layers with 128 ReLU hidden units per layer on EMNIST. Make sure that you use Adam optimizer with the hyperparameters provided in the template and train each network for 100 epochs. Visualise and discuss how increasing number of layers affects the validation performance and whether it worsens or mitigates the overfitting problem.

The questions in (mlp-cw1-questions .tex) that you must answer and count for this task are:

• Question 1;

• Question 2;

• Question 5;

• Question 6;

• Question 7;

• Question Table 1;

• Question Figure 2;

• Question 8;

• Question 9;

• Question Table 2;

• Question Figure 3;

• Question 10;

• Question 11; and

• Question 12.

(20 Marks)

6   Task 2: Mitigating the problem with Dropout and Weight Penalty

Definition and Motivation.  We provide the analysis and explanation for Dropout, L1Penalty, and L2Penalty. You will have to, in your own words, explain how one could use a combination of L1 and L2 regularization, discussing any potential benefits of this approach.

The question in (mlp-cw1-questions .tex) that you must answer and counts for this part of the task is:

• Question 13.

(10 Marks)

Implementing Dropout and Weight Penalty.  Here you will implement DropoutLayer, L1Penalty and L2Penalty and test their correctness. Here are the steps to follow:

1. Implement the Dropout class in the DropoutLayer of the mlp .layers module. You need to implement fprop and bprop methods for this class. Please note that the solution uses the original dropout formulation (i.e. scale the hidden unit activations by inclusion probability p in the final network for compensating missing units).  The sample distribution to be used for Dropout implementation is numpy’s uniform distribution, U(0,1) to pass the unit tests.

2. Implement the L1Penalty and L2Penalty class in the L1Penalty and L2Penalty of the mlp .penalties module. You need to implement __call__ and grad methods for this class. After defining these functions, they can be provided as a parameter, weights_penalty,  biases_penalty in the AffineLayer class while creating the multi-layer neural network.

3. Verify the correctness of your implementation using the supplied unit tests in DropoutandPenalty_tests .ipynb

4. Automatically create test outputs xxxxxxx_regularization_test_pack .npy, by running the provided program scripts/generate_regularization_layer_test_outputs .py which uses your code for the previously mentioned layers to run your fprop, bprop, __call__ and grad methods where necessary for each layer on a unique test vector generated using your student ID number.

To do this part simply go to the scripts folder scripts/ and then run

python  generate_regularization_layer_test_outputs .py  --student_id  sxxxxxxx replac- ing the student id with yours. A file called xxxxxxx_regularization_test_pack .npy will be gener- ated under data which you need to submit with your report.

(20 Marks)

EMNIST  Experiments.    In this section you should modify your baseline network to one that uses DropoutLayer, L1Penalty, or L2Penalty and train a model for each case. For the experiments, your baseline network should contain 3 hidden layers and 128 hidden units with ReLU activation function. You should use the Adam optimizer with a learning rate of 10−4 as specified in the template.

Your main aim is to i) investigate whether/how each of these functions addresses the above mentioned problem, ii) study the generalization performance of your network when used with one of these functions, iii) discover the best possible network configuration, when the only available options to choose from are Dropout and Weight Penalty functions and the hyper-parameters (Dropout inclusion probability and penalty coefficient for the Weight Penalty functions). You should use weight penalty on both weights and biases of your layers. Otherwise, unless explicitly specified, you can leave classes arguments to their default values.

The Dropout inclusion probability is a float value in the range (0,1), e.g. 0.5, chosen manually. Penalty coefficient is also a manually selected float value, e.g. 0.001, usually in the range of 0. 1 − 0.00001 . For model selection, you should use validation performance to pick the best model and finally report test performance of the best model.

Ensure that you thoroughly describe how these functions affect performance when used with different hyperpa- rameters in your report, ideally both at the theoretical and empirical level. When running such experiments, the expected amount of work is not a brute-force exploration of all possible variations of network configurations and hyperparameters, but a carefully designed set of experiments that provides meaningful analysis and insights. We have prespecified for what hyperparameter values you should run each individual experiment for L1/L2 regularisation and Dropout on Table 3 of the template. You should not rerun the experiments for which we provide results, but you will have to run a new experiment to get test results for the best performing model. You will have to identify and argue for a set of 8 different hyperparameter combinations for which you would have ran the combined Dropout and L1/L2 experiments. (The number 8 was not picked because there are for example 8 obvious combinations to pick or because one could not arguably run more, but rather to constraint your options and limit the amount of time put into this. There are many valid combinations of experiments to try, but you should motivate your specific selection.)

The questions in (mlp-cw1-questions .tex) that you must answer and count for this task are:

• Question Table 3;

• Question Figure 4;

• Question 14;

• Question 15; and

• Question 3.

(35 Marks)

7   Task 3: Literature Review and Conclusions

In this section, you will explore one related work, Decoupled Weight Decay Regularization [Loshchilov and Hutter, 2019], discussing the relation between L2 weight penalty and weight decay in Adam optimizer, its relation to your weight penalty implementation, its experimental results. Note that this review must be in your own words.

The questions in (mlp-cw1-questions .tex) that you must answer and count for this task are:

• Question 16;

• Question 17; and

• Question 18 (for the review); and

• Question 4 (for the report summary).

(15 Marks)

8    Report

Your coursework will be primarily assessed based on your submitted report.

The report template is divided into sections, though questions for each task might spread to more than one such section, as described in the tasks above. Please read the template before starting to answer the questions to get a sense of how it all fits together. This understanding will provide context (and in some cases example structure) for your answers, while also preparing you for coursework 2, where there will be less structure prebuilt in the template.

The directory coursework1/report contains the file (mlp-cw1-questions .tex) where you will add the answers to the questions, and a template for your report (mlp-cw1-template .tex) which you should not edit; the generated pdf file (mlp-cw1-template .pdf) is also provided, and you should read this file carefully as it contains some useful information about the required structure and content. The template is written in LaTeX, and you should not edit it. Instead, you will input your solutions by editing the file (mlp-cw1-questions .tex).

You should copy the files in the report directory to the directory containing the LaTeX file of your report, as pdflatex will need to access these files when building the pdf document from the LaTeX source file.

While inputting your answers in (mlp-cw1-questions .tex), the first thing you should do is add your UUN in place of (sXXXXXXX) at the start of the file. Then, answer each question, being careful to only edit the text that appears in the brackets of the commands (\youranswer).

The questions ask you to replace the text in red, fill in the tables provided in the template, and replace the figures specified with ones you created from your experiments. There is no specific word-count limit for any question, and you are responsible for identifying the correct level of detail based on the question itself (e.g. "discussion" implies an extensive analysis) and context (document section and surrounding text).

There are 18 TEXT QUESTIONS (a few of the short first ones have their answers added to both the Introduction and the Abstract). Replace the text inside the brackets of the command (\youranswer) with your answer to the question.

There are also 3“questions”to replace some placeholder FIGURES with your own, and 3“questions”asking you to fill in the missing entries in the TABLES provided.

Note that questions are ordered by the order of appearance of their answers in the text, and not by the order you should tackle them. Specifically, you cannot answer Questions 2, 3, and 4 before concluding all of the relevant experiments and analysis. Similarly, you should fill in the TABLES and FIGURES before discussing the results presented there.

Also note that, if for some reason you do not manage to produce results for some FIGURES and TABLES, then you can get partial marks by discussing your expectations of the results in the relevant TEXT QUESTIONS (for example Question 8 makes use of Table 1 and Figure 2).

Ideally, all figures should be included in your report file as vector graphics filesrather than raster files as this will make sure all detail in the plot is visible. Matplotlib supports saving high quality figures in a wide range of common image formats using the savefigfunction. You should use savefig rather than copying the screen-resolution raster images outputted in the notebook. An example of using savefig to save a figure as a PDF file (which can be included as graphics in LaTeXcompiled with pdflatex is given below.

import matplotlib .pyplot  as  plt

import  numpy  as  np

# Generate  some  example  data  to plot

x  =  np .linspace(0 . ,  1 . ,  100)

y1  =  np .sin(2 .  *  np .pi  *  x)

y2  =  np .cos(2 .  *  np .pi  *  x)

fig_size  =  (6 ,  3)    #  Set  figure  size  in  inches  (width,  height)

fig  =  plt .figure(figsize=fig_size)    # Create  a  new  figure  object

ax  =  fig .add_subplot(1 ,  1 ,  1)    # Add  a  single  axes  to  the  figure

# Plot  lines  giving  each  a  label  for  the  legend  and  setting  line  width  to  2 ax .plot(x,  y1,  linewidth=2,  label=’$y  =  \sin(2\pi  x)$’)

ax .plot(x,  y2,  linewidth=2,  label=’$y  =  \cos(2\pi  x)$’)

#  Set  the  axes  labels .  Can  use  LaTeX in  labels  within  $ . . .$  delimiters . ax .set_xlabel( ’$x$’ ,  fontsize=12)

ax .set_ylabel( ’$y$’ ,  fontsize=12)

ax .grid(’on’)    #  Turn  axes  grid  on

ax .legend(loc=’best’,  fontsize=11)    # Add  a  legend

fig .tight_layout()    #  This minimises  whitespace  around  the  axes .

fig .savefig(’file-name .pdf’)  #  Save  figure  to  current  directory  in  PDF  format

If you make use of any any books, articles, web pages or other resources you should appropriately cite these in your report. You do not need to cite material from the course lecture slides or lab notebooks.

To create a pdf file mlp-cw1-template .pdf from a LaTeX source file (mlp-cw1-template .tex), you can run the following in a terminal:

pdflatex mlp-cw1-template

bibtex mlp-cw1-template

pdflatex mlp-cw1-template

pdflatex mlp-cw1-template

(Yes, you have to run pdflatex multiple times, in order for latex to construct the internal document references.) An alternative, simpler approach uses the latexmk program:

latexmk  -pdf mlp-cw1-template

Another alternative is to use an online LaTeX authoring environment such as https://overleaf.com – note that all staff and students have free access to Overleaf Pro - see https://www.ed.ac.uk/information-services/computing/ desktop-personal/software/main-software-deals/other-software/overleaf.

It is worth learning how to use LaTeX effectively, as it is particularly powerful for mathematical and academic writing. There are many tutorials on the web.

9   Mechanics

Marks: This assignment will be assessed out of 100 marks and forms 10% of your final grade for the course.

Academic conduct: Assessed work is subject to University regulations on academic conduct:

http://web.inf.ed.ac.uk/infweb/admin/policies/academic-misconduct

Submission: You can submit more than once up until the submission deadline. All submissions are timestamped automatically. We will mark the latest submission that comes in before the deadline.                                     If you submit anything before the deadline, you may not resubmit after the deadline. (This policy allows us to begin marking submissions immediately after the deadline, without having to worry that some may need to be re-marked).                                                                                                                                          If you do not submit anything before the deadline, you may submit exactly once after the deadline, and a late penalty will be applied to this submission unless you have received an approved extension. Please be aware that late submissions may receive lower priority for marking, and marks may not be returned within the same timeframe as for on-time submissions.

Warning: Unfortunately the submission system on Learn will technically allow you to submit late even if you submitted before the deadline (i.e. it does not enforce the above policy). Don’t do this! We will mark the version that we retrieve just after the deadline.

Extension requests: For additional information about late penalties and extension requests, see the School web page below. Do not email any course staff directly about extension requests; you must follow the instructions on the web page.

http://web.inf.ed.ac.uk/infweb/student-services/ito/admin/coursework-projects/late-coursework-extension-requests

Late submission penalty: Following the University guidelines, late coursework submitted without an authorised

extension will be recorded as late and the following penalties will apply: 5 percentage points will be deducted for every calendar day or part thereof it is late, up to a maximum of 7 calendar days. After this time a mark of zero will be recorded.                                                                                                                            Please note! If you have received an extension, and then submit late in relation to your extended deadline, you automatically receive a mark of 0. (This is standard policy, and not a course decision).

9.1    Backing up your work

It is strongly recommended you use some method for backing up your work. Those working in their AFS homespace on DICE will have their work automatically backed up as part of the routine backup of all user homespaces. If you are working on a personal computer you should have your own backup method in place (e.g. saving additional copies to an external drive, syncing to a cloud service or pushing commits to your local Git repository to a private repository on Github). Loss of work through failure to back up does not constitute a good reason for late submission.

You may additionally wish to keep your coursework under version control in your local Git repository on the coursework1 branch.

If you make regular commits of your work on the coursework this will allow you to better keep track of the changes you have made and if necessary revert to previous versions of files and/or restore accidentally deleted work. This is not however required and you should note that keeping your work under version control is a distinct issue from backing up to guard against hard drive failure. If you are working on a personal computer you should still keep an additional back up of your work as described above.

9.2   Submission

Your coursework submission should be done online on the Learncourse webpage.

Your submission should include one zip file sxxxxxxx .zip that should contain

• Your  test  outputs  xxxxxxx_regularization_test_pack .npy.     which  can  be  generated  by implementing  the  previously  mentioned  classes,  going  into  scripts/  and  running  python generate_regularization_layer_test_outputs.py  --student_id  sxxxxxxx          replacing the student id with yours. A file called xxxxxxx_regularization_test_pack .npy will be generated under data which you need to submit with your report and the code.

• your completed report as a PDF file renamed as sxxxxxxx_report .pdf, using the provided template

• your local version of the mlp code including any changes you made to the modules ( .py files) and the Coursework_ 1 .ipynb notebook.

Please do not submit anything else (e.g. log files).

You can use this command on Linux machines to zip all the files together –

zip  -r  sxxxxxxx .zip mlp/  Coursework_ 1 .ipynb  sxxxxxxx_report .pdf

xxxxxxx_regularization_test_pack .npy

Replace sxxxxxxx with your student id.

Please check whether these files are included in the zip file before moving to the next step:

unzip  -l  sxxxxxxx .zip

Note that this file must not have your model weights and its size should be few Mbs only.

Once you have successfully created the .zip file and checked its content, you need to login to your Learn Machine  Learning  Practical  (2022-2023)[YR] webpage and submit the file.

• Migrate to the section Assessment on the left column on the course page.

• Click on Coursework 1.

• A page will appear where you will need to browse and upload your .zip file that you created previously in Attach  Files and then click Submit.

You can amend an existing submission by attaching a different .zip file using the Attach Files option and then Submit again.

Note that we will only mark the last uploaded coursework in case you amend your files. Thus it is your responsibility to make sure that correct files are uploaded. Please check that your zip file is not empty or missing files.

10   Marking Guidelines

This document (Section 4in particular) and the template report (mlp-cw1-template .pdf) provide a description of what you are expected to do in this assignment, and how the report should be written and structured.

Assignments will be marked using the scale defined by the University Common Marking Scheme:

Numeric mark < 40

40-49

50-59

60-69

70-79

80-100

Equivalent letter

F

D

C

B

A3

A1, A2

grade   Approximate meaning

fail

poor

acceptable

good

very good/distinction

excellent/outstanding/high distinction