Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

Computer Vision Assignment 2 (CLab2) S1-2024

0 Overview

This assignment has two tasks in total, is graded out of 20 marks, and is worth 15% of your final mark for this course.

0.1 Objectives

The goal of this assignment is to develop and assess proficiency at mid-level image processing, including corner detection, and deep learning techniques, including domain adaptation for deep neural networks.

0.2 Permitted Python libraries

The Python libraries that you may use in this assignment are

• OpenCV (cv2);

• NumPy;

• Matplotlib;

• scikit-image (skimage);

• scikit-learn (sklearn);

• SciPy;

• Pillow (PIL);

• PyTorch (torch); and

• torchvision.

Use of other Python libraries will result in a score of 0 for the task in which the library was used.

0.3 Advice

1.  Before writing your report, we recommend you watch the video “how to write a good lab report” on Wattle. The markers do not want to just see a collection of experimental results, but rather a coherent explanation and interpretation of the results, and key parts of your source code with detailed comments. Note that these are suggestions from a previous version of the course, not all of which apply to this assignment. In particular, they do not override any requirements in this document.

2.  The requirements for submission are at the end of this document. Please ensure your submission meets the requirements.

3.  The report is to be uploaded to the Wattle site before the due time. This course does not allow late submissions. That is, any submission after the deadline will receive a mark of 0.

4.  This is an individual assignment. All students must work individually when coding and writing the report.

0.4 Academic Integrity

You are expected to comply with the university policy on academic integrity and plagiarism. Please ensure you cite appropriately any resources that you use (lecture notes, papers, online documents, code), and complete all tasks independently. All academic integrity violations will be reported and can result insignificant penalties. More importantly, working through the problems yourself will help you learn the material and will set you up for success in the final exam.

Figure 1: Code listing for harris.py.

2 Task 2: Domain Adaptation (10 marks)

For this task, your objective is to implement domain adaptation techniques using the pretrained  ResNet-34 network available in the torchvision library. No custom network architecture implementa- tion is required for this task. Nevertheless, you are required to develop your own code to load the  dataset and to perform training and validation of your network.

In this task, you will be using a subset of the DomainNet dataset [1]. The specific domains selected for this task are Real and Sketch. We use the Real domain as the source domain and Sketch domain as  the target domain. To facilitate neural network training for students who do not have reliable access  to GPUs, we have selected only ten classes from each domain, which include backpack, book, car, pizza, sandwich, snake, sock, tiger, tree, and watermelon. Images in this dataset do not have a uniform  shape, so they will need to be reshaped to 224 × 224 before being fed into the network.

Download the dataset zip file from hereor execute the dataset_downloader.py script.

1.  Complete the following preparations steps.

(a)  Download the dataset directly from here or execute the data downloading script

dataset_downloader.py.

Note: There should be 3984 images in the real_train folder, 1712 in the real_test folder, 1956 in the sketch_train folder, and 841 in the sketch_test folder. (0 marks)

(b)  Implement a PyTorch Dataset class for the dataset and load the data with appropriate transformations.

Hint: Your transformation should at a minimum include resizing the images to 224 × 224, and normalising the images using the means (0.485, 0.456, 0.406) and standard deviations (0.229, 0.224, 0.225). You are allowed to add data augmentation techniques to the transformation. (0.5 marks)

(c)  Load the pretrained ResNet-34 model from torchvision and modify its final fully- connected layer to accommodate the specific number of classes required for this task. (0.5 marks)

2.  Implement the code for fine-tuning a model.

(a)  Implement the train and test functions for fine-tuning.

Hint: The two functions could, for example, have the following input parameters: (src_loader, tgt_loader, model, optimizer). This is not a restriction on your function interface, but some suggestions. You are free to include additional parameters as needed. (1 mark)

(b)  Train theResNet-34 model on the Sketch domain (use images in sketch_train), assuming all training labels are given. Then evaluate the model on the Sketch domain (use images in sketch_test). Include a screenshot of your training curve in your report. (1 mark)

(c)  Train the ResNet-34 model on the Real domain (use images in real_train), assuming all training labels are given. Then evaluate the model on the Sketch domain (use images in sketch_test). Include a screenshot of your training curve in your report. (1 mark)

(d)  Report the classification accuracy obtained from steps 2(b) and 2(c) on the test set (i.e., sketch_test). Compare and contrast these results and provide an explanation for any observations. (1 mark)

3.  Implement the code for domain adaptation.

(a) You have two options for the domain adaptation loss function: the mean discrepancy loss or CORAL loss. Choose one of the two options in your implementation. In your code, add line-by-line comments and include these in your report. (1 mark)

i.  The mean discrepancy

where µ is the mean across feature dimensions for each datum (image) and is

the squared Euclidean distance.

ii.  The CORAL loss

where d is the dimension of the feature, is the squared Frobenius matrix

norm, and CS  and CT  denote the feature covariance matrices

where DS  = {xi }, DT  = {uj }, and i ∈ [1, nS ] and j ∈ [1, nT ]. The number of source and target data are nS  and nT  respectively, and x ∈ R1×d  and u ∈ R1×d  are the d-dimensional deeplayer activations Φ(I) of input image I.

(b)  Based on the ResNet-34 backbone pretrained on ImageNet, perform unsupervised domain adaptation. Specifically, the Real domain (real_train) is the source, and the Sketch domain (sketch_train) is the target. To train your model, you should use images AND labels from the Real domain, and images ONLY from the Sketch domain. You  should use stochastic gradient descent to minimise the joint loss function composed of the cross-entropy loss and the selected domain adaptation loss. After this training pro- cedure, evaluate classifier performance (accuracy) on the Sketch domain (sketch_test). Compare the new accuracy value with Task 2.2(c), describe any observations and  provide explanations. Include screenshots of your training curves in your report.

Hint: To optimise the joint loss, it is necessary to specify the weight assigned to the domain adaptation loss. For the mean discrepancy loss, we recommend a weight of 0.01. For the CORAL loss, we recommend a weight of 10. (3 marks)

(c)  In the previous question, we used the recommended weights for the domain adaptation loss functions. Do these recommended weights always work in practice? Why or why not? In contrast, the textbook assumes that there is no domain gap between the training and testing datasets and recommends using a labelled validation set for hyperparameters like these weights. In practical scenarios, where we have a labelled source domain and an unlabelled target domain with distribution shifts, what is the issue with the textbook’s way of selecting hyperparameters? Provide all necessary explanations. (1 mark)

Resources:

1.  Implementing  a PyTorch Dataset class  and using  a PyTorch DataLoader: https:// pytorch.org/tutorials/beginner/basics/data_tutorial.html

2.  Fine-tuning pretrained models from the torchvision library: https://pytorch.org/ tutorials/intermediate/torchvision_tutorial.html

3.  The PyTorch documentation: https://pytorch.org/docs/stable/index.html

4.  Deep CORAL [2] paper on domain adaptation: https://arxiv.org/abs/1607.01719

Note:

1.  Although we have provided links to PyTorch tutorials with detailed and explained code, you MUST write your own code for this task.

2.  For every question involving network training, log your loss values and accuracy for every training epoch to a file. These text files must be included in your ZIP file to provide evidence that you trained the networks.

References

[1]  Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1406–1415, 2019.

[2]  Baochen Sun and Kate Saenko.   Deep coral:  Correlation alignment for deep domain adaptation.   In Proceedings of the European Conference on Computer Vision Workshops, pages 443–450. Springer, 2016.