Assignment 1

February 20, 2021

1 Computer Vision 2021 Assignment 1: Image filtering

In this prac you will research, implement and test some image filtering operations. Image filtering by convolution is a fundamental step in many computer vision tasks and you will find it useful to have a firm grasp of how it works. For example, later in the course we will come across Convolutional Neural Networks (CNNs) which are built from convolutional image filters.

The main aims of the prac are:

• to understand the basics of how images are stored and processed in memory;

• to gain exposure to several common image filters, and understand how they work;

• to get practical experience implementing convolutional image filters;

• to test your intuition about image filtering by running some experiments;

• to report your results in a clear and concise manner.

This assignment relates to the following ACS CBOK areas: abstraction, design, hardware and software, data and information, HCI and programming.

1.1 General instructions

Follow the instructions in this Python notebook and the accompanying file a1code.py to answer each question. It’s your responsibility to make sure your answer to each question is clearly labelled and easy to understand. Note that most questions require some combination of Python code, graphical output, and text analysing or describing your results. Although we will check your code as needed, marks will be assigned based on the quality of your write up rather than for code correctness! This is not a programming test - we are more interested in your understanding of the topic.

Only a small amount of code is required to answer each question. We will make extensive use of the Python libraries

• numpy for mathematical functions

• skimage for image loading and processing

• matplotlib for displaying graphical results

• jupyter for Jupyter Notebooks

You should get familiar with the documentation for these libraries so that you can use them effectively.

2 The Questions

To get started, below is some setup code to import the libraries we need. You should not need to edit it.

2.1 Question 0: Numpy warm up! (5%)

Before starting the assignment, make sure you have a working Python 3 installation, with up to date versions of the libraries mentioned above. If this is all new to you, I’d suggest downloading an all in one Python installation such as Anaconda. Alternatively you can use a Python package manager such as pip or conda, to get the libraries you need. If you’re struggling with this please ask a question on the MyUni discussion forum.

For this assignment, you need some familiarity with numpy syntax. The numpy QuickStart should be enough to get you started:

https://numpy.org/doc/stable/user/quickstart.html

Here are a few warm up exercises to make sure you understand the basics. Answer them in the space below. Be sure to print the output of each question so we can see it!

1. Create a 2D numpy array A with 2 rows and 3 columns. Fill with values 1 to 6.

2. Create a 2D numpy array B with 3 rows and 2 columns. Fill with values 6 to 1.

3. Calculate A’ + B’, where A’ is the top left 2x2 submatrix of A.

4. Calculate the matrix product of A and B.

5. Calculate the element wise product of A and (B transpose).

You need to be comfortable with numpy arrays because that is how we store images. Let’s do that next!

2.2 Question 1: Loading and displaying an image (10%)

Below is a function to display an image using the pyplot module in matplotlib. Implement the load() and print_stats() functions in a1code.py so that the following code loads the mandrill image, displays it and prints its height, width and depth.

Return to this question after reading through the rest of the assignment. Find 2 more images to use as test cases in this assignment and display them below. Use your print_stats() function to display their height, width and number of channels. Explain why you have chosen each image.

Your explanation of images here

2.3 Question 2: Image processing (20%)

Now that you have an image stored as a numpy array, let’s try some operations on it.

1. Implement the crop() function in a1code.py. Use array slicing to crop the image.

2. Implement the resize() function in a1code.py.

3. Implement the change_contrast() function in a1code.py.

4. Implement the greyscale() function in a1code.py.

Is the contrast function always reversible? Explain why or why not.

2.4 Question 3: Convolution (20%)

2.4.1 3.1 2D convolution

Using the definition of 2D convolution from week 1, implement the convolution operation in the function conv2D() in a1code.py.

2.4.2 3.2 RGB convolution

In the function conv in a1code.py, extend your function conv2D to work on RGB images, by applying the 2D convolution to each channel independently.

2.4.3 3.3 Gaussian filter convolution

Use the gauss2D function provided in a1code.py to create a Gaussian kernel, and apply it to your images with convolution. You will obtain marks for trying different tests and analysing the results, for example:

• try varying the image size, and the size and variance of the filter

• subtract the filtered image from the original - this gives you an idea of what information is lost when filtering

What do you observe and why?

2.4.4 3.4 Sobel filters

Define a horizontal and vertical Sobel edge filter kernel and test them on your images. You will obtain marks for testing them and displaying results in interesting ways, for example:

• apply them to an image at different scales

• considering how to display positive and negative gradients

• apply different combinations of horizontal and vertical filters

Your comments/analysis of your results here…

2.5 Question 4: Image sampling and pyramids (25%)

2.5.1 4.1 Image Sampling

Apply your resize() function to reduce an image to 0.125 height and width, and then to enlarge the image back to its original size. Display the result and compare to the original image. Apply a Sobel filter to the resulting image and compare to the original. What do you observe and why?

2.5.2 4.2 Image Pyramids

Repeat this procedure, but this time, apply a Gaussian filter to the image before resizing. What do observe, and why?

Repeat again, but this time apply the scaling in 3 steps of 0.5, creating a pyramid of images. What do you observe, and why?

Your comments/analysis of your results here…

2.6 Question 5: Auto correlation (20%)

Recall from week 2 that cross-correlation is a measure of similarity between a template and an image. It is defined similarly to convolution.

2.6.1 5.1 Correlation function

Implement the function corr() in a1code.py, based on your convolution function. Hint: numpy’s flip() function may be useful here.

2.6.2 5.2 Auto-correlation

We will experiment with auto-correlation, where the template patch is taken from the image it is being compared to. Use the cropped eye from Question 2 as your template. Calculate the correlation of that template with every location in the image, and display the output as an image. Where is the maximum similarity? (Hint: numpy functions argmax() and unravel() will be useful here). For simplicity, you can use a greyscale version of the image and template.

Is it what you expect? Why or why not?

2.6.3 5.3 Modified auto-correlation

Try modifying your correlation template or the base image in different ways, and analyse the effect on correlation results. For example:

• if you did not find the correct location in 5.2, try centering the template about its mean (i.e. subtracting the mean brightness from each pixel)

• if you did find the correct location in 5.2, try using resize() and change_contrast() on the image. Where does it fail?

As before you will obtain marks for coming up with interesting tests and analysis, and displaying your results clearly.

Your comments/analysis of your results here…

2.7 Question 6: Normalised cross correlation (postgraduate, 10%)

This question is required for postgraduate students only. PG marks for the other questions will be scaled by 0.9.

Search online for “normalized cross correlation” (NCC). Implement and test NCC, and compare to your previous correlation results.

Your comments/analysis of your results here…