In this project, you will implement several discriminative models for pattern classification. You are encouraged to use Python and Google Colab for this project.  You are ONLY allowed to use libraries for linear algebra operations, such as matrix multiplication, matrix inversion, matrix factorization, and data visualization to plot your results. You are NOT allowed to use any existing machine learning or statistics toolkits or libraries or any open-source codes for this project and you will have to implement the most parts of the model learning and testing algorithms yourself as a practice of various algorithms learned in class.  This is the purpose of this project.  If you have any questions, please consult the instructor for what you can or can not use for this project.  You will use a popular data set, called MNIST, throughout the project. TheMNIST database of handwritten digits contains 60,000 training images and 10,000 test images, and the images are 28-by-28 in size.

Refer to

https://colab.research.google.com/drive/1FyahMGAE22716sUCrNXPvTrKTwS615Hd

for how to load it in Python. In this project, for simplicity, you just use pixels as raw features.

Question 1

Logistic Regression (10 mark): Revise the logistic regression implementation at Colab (https:  //colab.research.google.com/drive/1qUi2qgwBMmHJb7Bx7ca_braGtptmzFOQ?usp=sharing) to implement minimum classification error (MCE) method in Section 6.3 (page 113) to learn a  binary classifier to classify two digits (’7’ vs.  ’9’) in the MNIST data set.  Compare its perfor-  mance with that of logistic regress and report the best results in the heldout test data of these  two digits.

Question 2

Support Vector Machine (20 marks): Use all training data of 10 digits to learn two 10-class clas- sifiers (considering to use the one vs. one strategy) using linear support vector machine (SVM) and nonlinear SVM (with Gaussian RBF kernel), and report the best possible classification per- formance in the held-out test images.  Dont just call any off-the-shelf optimizer.  You need to use the sequential minimization optimization (SMO) method to implement the SVM optimizer yourself.  Compare your SMO optimizer with the PGD optimizer at Colab (https://colab. research.google.com/drive/1vlN46Xhfv4ES8jFWWI0iTj3WHKjb0KMg?usp=sharing) in terms of classification accuracy and running speed.

Question 3

Deep Neural Networks for nonlinear feature extraction (20 marks): Based on the implemen- tation of fully-connected deep neural networks at Colab (https://colab.research.google. com/drive/1TY3fAAS18lyGrlEsvLhaYm6fIBP7CDgD?usp=sharing), implement two nonlinear feature extraction methods using deep neural networks:

. Autoencoder (Section 4.4.1)

. Bottleneck features (Section 4.4.2)

for the MNIST data. Use all training data of 10 digits to train these two feature extractors and investigate various network structures (such as different number of layers and nodes per layer) for the best possible classification performance.  When you measure the classification perfor- mance, please use your nonlinear SVM (Gaussian kernel) code from Question 2 to implement a binary classifier on the extracted features of the training data of two digits (’7’ vs.  ’9’).  At last, report the best configuration and its corresponding classification accuracy in the heldout test images for two pairs of digits (’7’ vs.  ’9’ and ’0’ vs.  ’8’). Based on the results, discuss the differences between these two nonlinear extractors using autoencoder and bottleneck features.

What to submit?

You need to submit all of your codes written for this project. Please provide clear instruc- tions on how to repeat your experiments in a separate readme file or write them clearly in your Jupyter Notebooks. Moreover, you need to submit a project report (in pdf, maximum 6 pages) to summarize what you have done in terms of algorithm development and experimen- tal fine-tuning, also report the best results and their corresponding settings for each classifier and discuss your findings from this project.