Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

ELEC 230 Assignment 4 (2021-2022) 20% weighting

Background

Adapted fromhttps://docs.opencv.org/4.2.0/d6/d6d/tutorial_mat_the_basic_image_container.html .

Please visit and read the rest of the above web-page for more information about Mat objects.

We have lots of ways to acquire digital images from the real world: digital cameras, scanners, computed tomography and magnetic resonance imaging, to name a few. What we see are images. However, when transforming this to our digital devices, what we record are numerical values for each of the points of the image.

Figure 1 What the mobile robot camera sees (from OpenCV.org)

In Figure 1 you can see that the ‘mirror’ of the car is nothing more than a matrix containing all the intensity values of the pixel points. How we get and store the pixels values may vary according to our needs, but in the end all images inside a computer world are reduced to numerical matrices and  other  information  describing  the  matrix  itself.  OpenCV  is  a  computer  vision  library  for processing and manipulating this information. The primary interface of OpenCV is in C++; it still retains a less comprehensive older C interface, so you may encounter elements of both languages in online tutorials.

Aim of this assignment

OpenCV is like the image processing brain’ of a modern mobile robot (although it’s more than this.) This assignment aims to give you an understanding of basic image processing with OpenCV, laying some of the foundations for you to begin to understand computer vision for a mobile robot. This assignment does not ask you to interface OpenCV with ROS (connecting the robot’s vision system with the  rest  of  its  brain)  although this would  be the  logical  next  step  and you  are encouraged to explore this in your own time.

In  this  assignment  you  will  generate  OpenCV  programs  which  can  perform  image  /  video processing algorithms on an image and video provided on Canvas. In Part A, the ultimate aim is to extract a door feature and similar features. In Part B, you are performing background subtraction in order to detect moving people.

Hints & Tips

Your code must be executable in a Linux x86-64 environment without any modification. For more   information   on   installing   and   working  with   OpenCV  4.2.0  with   appropriate dependencies on  Ubuntu  20.04,  please  have a  look at the  Image  Processing  lab script available on Canvas1 .

Before attempting the Assignment, follow all the steps in the Image Processing lab script on Canvas including working through the 7 tutorial pages recommended near the end of the script. After this, you are free to locate anyOpenCV 4.2.0 Tutorials pages that are relevant to this assignment. You are expected and encouraged to copy and adapt your code from the code shared on the OpenCV 4.2.0 Tutorials pages! :) Also, go over the Image Processing lecture notes / videos, making sure you understand the image processing algorithms taught there, to help your discussion. You can also consult the relevant part of Chapter 4 of the recommended textfor additional information.2

Part A: Smooth, edge detect, feature extract (85%)

1.   For your first task you are going to build a number of basic standalone image- processing programs using OpenCV and the Text  Editor, where each program does the following (replace “YI” with your own ‘last-letter initials’3 ):

•    Loads & displays an image door.jpeg’ (see Canvas) from the working directory

•    Also displays an output image, as follows:

a)   Program YI_G performs a Gaussian smoothing filter (4).

b)  Program YI_S performs a Sobel edge detection filter (4).

c)   Program YI_C performs a Canny edge detection filter (4).

d)  Program YI_H performs a Standard Hough Line transform to extract straight edges (4).

•    The bracketed ( ) maximum marks above assume the following:

• YI_S, YI_C and YI_H should incorporate conversion to grayscale, and Gaussian smoothing, before edge detection.

• YI_H should use Canny edge detection before finding straight edges .


• In any Canny’ function arguments (YI_C and YI_H), the high threshold should be set to be automatically three times the lower threshold .

• Each  program, should show  both the original and final  image  until the  user presses any key while a display window is selected (which should terminate the program).

• All relevant filters should initially (and in the submitted .cpp files) use a 3x3 mask .

No other functionality which is unnecessary for this assignment is included.

2.   Importantly, the code in each program’s .cpp file should be commented extensively BY YOU (delete the default comments present on the OpenCV code, and replace with your own, explaining each significant segment of code. (8)

3.   Demonstrate that each program in 1 (a) to (d) works by running it and taking focused screenshots of the original and final images. Include these in your report, with a brief explanation about the theory of each filter/transform (18).

4.   Continue  your  report  for  Part  A,  considering  the  following  questions.  Discuss  each question with the help of reading and experimentation (i.e. playing with the code!) and support your discussion with focused screenshots of the associated outputs:

a)  What  effect,  and  why,  does  increasing  the  kernel  size  have  on  the  Gaussian smoothing filter (assume the kernel is a square)? (3)

b)  What effect does entering 0.2, instead of -1, for the standard deviations in x and y, have on the Gaussian smoothing filter when the kernel size is 9x9?

Explain the effect. (5)

c)   With a smoothing kernel size of 7x7, what difference does it make if you change the smoothing filter used in the Sobel edge detector from Gaussian smoothing to

a normalized box filter (simple average filter)? Why is this?

Which do you think is better and why? (4)

d)  At  the  moment,  YI_C  is  probably  detecting  far  too  many  edges.  Change  the smoothing filter kernel to 7x7, and adjust the hysteresis minimum threshold in the Canny algorithm from 0 to 100 in steps of 25. Document & explain the differences you see. (4)

e)  What difference does it make if you effectively remove the smoothing stage from Canny edge detection by setting the smoothing kernel size to 1x1?

Explain what you observe. (5)

f)   Choose an optimal’ Canny threshold from 4(d) and compare the Sobel and Canny

edge detection algorithms (both with 7x7 smoothing filter kernel). Explain what difference(s) you see and why they may occur. (4)

g)  For the Hough transform program, adjust the smoothing kernel size (assume it’s square) and the  Canny minimum threshold’ value, to try and obtain a  better extraction of the door-frame. Comment on the quality of the results, including the relevance of your efforts to mobile robotics. Discuss how this relates to (e.g.) the need for dynamic thresholding. (10)


Part B: Some fun video background subtraction (15%)

1.   Your second task is to copy, save and buildthis video background subtraction program on OpenCV.org, which uses the BackgroundSubtractor class. Initially save and build the  copied program as it is” without any alterations. You do not need to replace the tutorial  comments with your own, for this program only. Name the program YI_backsub (replace

YI with your own last-letter initials as described in Footnote 4 above on p.4). This program does the following:

Loads a video.

•    Plays it.

•    Performs a background subtraction filter in another window .

Edit and re-make the code so that the program uses the K-nearest neighbours” (KNN) algorithm and is applied to “walking2.mp4” found on Canvas. (2) .

2.   Verify that the program works, by running it and taking a full-screen screenshot of the functionality showing the original video side  by side with the subtracted  background video. Include this in your report (no explanation needed.) (3)

3.   Continue your report for Part B, considering the following questions:

•    Why  do you think the  background-subtracted  moving  people  sometimes  look semi-transparent? (3)

•    Suggest a robotics application that background subtraction might be useful for, and explain your answer carefully. (5)

The Rules

1.  Your code should be tidy and well-commented. ‘Tidy’ means indentations are correct, etc.2 Write your own comments so that the purpose of each distinct part of the code is explained (see the example on Canvas, mentioned at the end of this document). All coding comments must be in your own words.

2.  While you are expected and encouraged to adapt code from the OpenCV.org tutorial pages, it will harm your marks if you include unnecessary copied code that does not relate to the assignment instructions (e.g. code for trackbars). Similarly, include only headers that are needed to satisfy the objectives . (Do not include

, for example.) In Part A, you must not include any third- party coding comments, e.g. comments in example code found on the OpenCV.org tutorial pages; write your own comments in your own words.

3.  This goes without saying: you should do this work on your own. All the work you turn in should be yours, and not done in collaboration with anyone else (or copied from them, with or without their knowledge).

•     If you use any external sources of coding inspiration, other thanOpenCV4.2.0 Tutorials,then let us know in a README file.

•     All sources of inspiration for your  understanding and explanations of the theory and operations of the various algorithms should be credited with a reference (and placed within quotation marks if quoted verbatim.)

What to Hand In

Hand  in everything that someone else  needs to  build and  run your  programs on  Linux / OpenCV 4.2.0. For this assignment, this means:

•    Your source code (.cpp files), well-commented in your own words

CMakeLists.txt

You should also submit a Word document including a 1-page cover sheet (with title, Abstract and academic integrity declaration), a Contents page, section headings and page numbers etc. (see Marking Criteria.) This document should also incorporate:

•    Background (around one paragraph)

•    Discussion of no more than 7 pages (suggest 6 pages for Part A), Calibri font size 11, single-spaced, ‘normal’ margins, including:

o Part A q.3: Demonstration of the functionality of your programs supported by: ofocused screenshots of initial output (see Marking Criteria)3

osome  explanation  of  the  theory  of  the  image  processing  algorithms  used  (see lectures and the recommended text)

o Part B q.2: Full-screen side-by-side screenshot as described in B q.2 o Each of the above should be followed by a discussion of the questions in the respective Part (for Part A, this is q.4, and for Part B, q.3), backed up by more focused output screenshots.

•    Appendix with all the full-screen versions of your focused screenshots with full desktop showing date/time (see Marking Criteria). These do not have to be numbered. The Appendix is not included in the 7-page limit above.

•    Do NOT copy and paste your entire .cpp code in this document!

Therefore, if you have completed everything you should be submitting 1 Word document, 5 .cpp files and a CMakeLists.txt file. Submit 7 individual files, not a zip file. You will only be graded on the files you submit, and points will be deducted if you break the submission instructions.

Do not hand in executable files. Each program should run after copying your files to the working directory and typing the following commands in a terminal:

$ cmake .

$ make name

$ ./name

…where name” is replaced by the name of the program.

Helpful Further Resources

Please make sure to start by following the Image Processing lecture videos/PDFs and the Week 9 lab script, as this will give you the background you need to carry out this assignment. If you have any questions, where a requirement is unclear or there are some concepts for which you need additional information, you can use the Canvas “Assignment 4” discussion board (DB) to ask questions.

Helpful insights into how OpenCV works (including the tutorials and other resources too such as descriptions of OpenCV modules (functions)) can be found at OpenCV.org.

If you read more widely to develop your understanding or support your arguments, make sure to attribute such resources with references.

For further interest and development, see the tutorials on the wiki.ROS.org pages about interfacing OpenCV with ROS.

Space problems?

You can make your focused screenshots quite small if you wish to for reasons of space, and tile several    together in a single figure (e.g. Figure 1(a), (b), etc.) Just make sure that at normal size (100% zoom) we  can see enough detail to confirm the functionality and/or what you are discussing or demonstrating, etc.

Example of (overly) well-commented code

Please see Canvas for an example of very well-commented code; you should aim to emulate this in your .cpp files, if to a slightly less extreme extent.