COSC 2670/2738 Practical Data Science (with Python) Project Assignment 3, Semester 2, 2022

Objective

The key objectives of this assignment are to learn how to compare and contrast several recommendation system algorithms. There are three major components of the assignment – a completed Jupyter notebook used to run your experiments, a written report, and a short video presentation where you describe what you did and your key findings.

The dataset you will use will be a sample of the Netflix Prize data. The problem is movie recommendation, and the data is already split into a training and validation set that you can use to run all of your experiments.

Provided files

The following template files are provided:

• SXXXXX-A3.ipynb : The primer Jupyter notebook file you should use to stage and run all of your experiments.

• netflix-5k.movie-titles.feather : The movie title dataframe that can be used to map a movieID to a title, as well as a list of genres.

• netflix-5k.train.feather : The training tuples for 5,000 users, where each tuple is ⟨userID,movieID,rating⟩ .

• netflix-5k.validation.feather  :  A predefined set of validation tuples for the same users that can be used by you to benchmark the performance of various algorithms.

• A3.pdf : This specification file.

Creating Your Workspace

Once again, you should rename the file SXXXXX-A3.ipynb appropriately based on your student ID.

Creating Your Anaconda Environment

In order to create your anaconda environment for this project, you should run the following command in a terminal shell:

conda  create  -n  PDSA3  python=3 .8

conda  activate  PDSA3

pip  install  jupyterhub  notebook  numpy  pandas

pip  install matplotlib  scikit-learn  seaborn

pip  install  kneed  scikit-surprise

Note that both kneed and scikit-surprise can be finicky on some systems.  For example on my machine, an error was thrown during the compile of scikit-surprise (Macbook Pro M1) but the install still worked when it tried a fallback install method. So if you find that it really fails for you using pip, then and only then resort to conda. This would break requirements.txt, but it should work reliably for everyone albeit not very reproducible. The magic commands would be:

conda  install  -c  conda-forge  kneed

conda  install  -c  conda-forge  scikit-surprise

You can type“pip freeze”to see a list of the packages that are correctly installed in your environment. You can also install scikit-learn-intelex and/or psutil if you want to use the Intel-based optimisations or debug memory management as shown in the sample Jupyter notebook.

If you also wish the timing and other jupyter extensions to be enabled in your notebook (optional but may be useful depending on how you decide to present your results), you need to run the following additional commands:

pip  install  jupyter_contrib_nbextensions

pip  install  jupyter_nbextensions_configurator

jupyter  contrib  nbextension  install  --user

jupyter  nbextensions_configurator  enable  --user

Now you just need to type “jupyter notebook” to start up jupyter correctly with access to the libraries you just installed.

Also, recall that if you ever stop working in your environment and come back later. You must open a terminal, run“conda activate PDSA3” and then“jupyter notebook”, otherwise you will not be working in the virtual environment you created above, and most things you try to do will probably start failing.

You should not use any other libraries to complete your assignment beyond the ones shown above without written permission from the course coordinator (Shane).

1    The Jupyter Notebook Primer (5 marks)

I have included the jupyter notebook I walked through at the end of the Week 10 Lectorial. This notebook will provide you with everything you need to correctly load the dataframe files from feather, and also includes an example of how to do both a grid search and randomised search for parameter tuning on one of the recommendation system algorithms included in Surprise. You should spend some time reading the API documentation and tutorial for this library provided at https://surpriselib.com. This will be critical information you should use to stage your experiments.

2    The Report (15 marks)

The main component of your assignment will be to carefully write up your key findings. Your report should not be more than 5 pages using 11pt font. You may also have one additional Appendix page containing additional graphs or tables.  Your final report must be submitted as a PDF file. You can use Microsoft Word, but I would strongly encourage you to consider writing up your report using LATEX (https:\overleaf. com).  Writing in LATEX may seem daunting at first, but Overleaf provides plenty of tutorials and examples, and it is pretty easy once you get the hang of it.  This spec file was written using LATEX. Microsoft Word is not a good tool for writing technical documents, and in fact most Computer Science Conferences all require papers to be written in LATEX. The quality of of the presentation layer is easily discernible by most people in Computer Science.  Write a paragraph or two with a graph or diagram in overleaf and then in MSWord and compare the two – I bet you’ll see the difference immediately!

Regardless of which tool you use to generate your final PDF, the format of the report should be:

1. Your name and student number at the top of page 1.

2. Introduction (usually no more than 1/2 page).

3. Methodology (usually about 1 page) - This would contain a clear description of each recommendation system algorithm you are using and a rationale as to why it is being used.

4. Experiments ( 3 pages) – This is the main component of your report. Here you should document all of the parameters and algorithms used, Tables, Figures, or Graphs that you create in order to compare and contrast all of the algorithms you have benchmarked and a discussion about what you have discovered. There is no reason include images of code snippets as you are submitting your Jupyter notebook already – you should include images of graphs you create – assuming they are important to the story you are telling.

5. Conclusion (1/2 page) – A clear summary of your key findings.

6. References (Separate page) – You can include as many references as you see fit and need in your report, and this is not counted against the 5 page limit.

7. Appendix (Separate page) – Any additional graphs or tables that you think are important but that you could not include into the main document because of space constraints.

Summary – a report that has a 5 page body, 1+ pages of citations, and 1 optional page at the end as an Appendix.

You must compare at least four different algorithms from the surprise library in your report. Note that one algorithm (e.g. SVD) with two different parameter settings does not count as two different algorithms. That is one algorithm with two different parameter settings, i.e. one algorithm. You can certainly include results for multiple parameter settings for each of the four algorithms in your experiments, just make sure you are using 4 different algorithms in total in your shootout.

In week 8, I provided several evaluation classes you can use to compute a wide variety of evaluation measures, and we encourage you to explore as many as you can, as doing so will provide additional evidence that your “winning” algorithm is really a winner. The“official” metric will be RMSE as this is the metric used in the original competition, but you should compare the 4 algorithms using a minimum of three different evaluation metrics. You should aim to have at least one algorithm that  can achieve an RMSE score ≤ 0.800. This should be achievable with a little parameter  tuning, if you choose good algorithms from surprise, and you may even decide that the  algorithm that got the best RMSE score is not necessarily the“best”algorithm overall  based on the experiments you have ran.  Hint:  Think carefully about what “good” movie recommendations really mean to you. We all know what it is, so what do you  think is the best way to prove or disprove recommendations from Algorithm A are  clearly better than the ones you get from Algorithm B. We covered a wide variety of evaluation measures in the Week 8 Lectorial, so go look at what was in that notebook  and think about it.

Other key hints – (1) If you have a Figure, Table, or Diagram in the report, it must have a caption and you must reference it in your report and discuss it. By that I mean “Table 1 contains a table of RMSE results for Algorithms A-E. We can see that ...”(2)  If you use ideas or code from somewhere else, you must include it in your references. A typical “bibtex”citation for something taken from the web would look something like:

@misc{StackA,

title  =  {{Stackoverflow  Discussion  on  X},

howpublished  =  {\url{http://www.example.com}} ,

note  =  {Accessed:  2022-10-05}

The preferred referencing style for Computer Science is usually APA. See https: //libguides.murdoch.edu.au/APA/sample for an example.   There are lots of tutorials available online on APA referencing,  so if you have never seen it, just search for tutorials on APA referencing and you’ll find more than you could ever read/watch.  (3) If it isn’t clear, you must include graphs, tables, and/or diagrams in your experimental section, which are to be used as evidence to back any claims about algorithm performance that you make.

3    The Video Presentation (10 marks)

Your final task it to create a 5-10 minute video recording of you presenting your findings.  You must create slides, and I would strongly encourage you to use PIP (Picture in Picture Mode) in your video. If you have no idea what I am talking about, have a look at this video created by our guest speaker when he was a PhD student at RMIT – https://www.youtube.com/watch?v=jFhhL1mkziQ The video format should be MP4. No other video formats will be accepted.

A good presentation will have visually appealing slides, a clear story line, and the presenter will not just be reading what is on the slide or from a piece of paper – they will be describing what the current slide is showing. If you want to know more, check out the talk by S. P. Jones on“How to Give a Great Research Talk”– https: //www.youtube.com/watch?v=sT_-owjKIbA

You might say to yourself, I’m not giving a research talk – but you are. You are doing experimental research in Data Science on Recommendation Systems. So, treat it like one.

If you have no idea how to create a PIP video of a presentation quickly and eas- ily, check out this video – https://www.youtube.com/watch?v=rKDKniBmwe0& feature=youtu.be. If you don’t like this technique just search for how to create a picture in picture presentation using Zoom or Microsoft Teams. It is surprisingly easy to do using Microsoft Teams, Zoom, or the open source program OBS. I’m sure there are many other ways to do it, but there are definitely simple and quick ways to do it once you see how. The point is that I don’t want “technology”to be an impediment

in creating a great presentation. It is much more useful spending time on getting the content right and doing a few practice runs so that you can create an interesting talk.

Submission

All submissions must be made on the canvas shell for the course.  A link will be

provided within canvas, under the Assignments tab for submissions.  Assignments submitted through any other method will not be marked.

Pay close attention because the final submission must be done differently than the first two assignments.  The rationale for this is so that each of you have access to a Turnitin report of your submission when you submit.  Since you are writing a report, this is a valuable tool for you to ensure that you do not get dinged for academic misconduct.  Believe it or not, we do take this seriously, and every semester several students get reported to the university for failing to comply with the plagiarism rules. So make sure you submit early enough that you can action it, otherwise, you’ll find yourself in a really uncomfortable meeting with a university panel and you won’t get a mark for the course until it is resolved!

This is an individual project. Do your own work.

You should submit four separate files in canvas.

[Your  Student [Your  Student [Your  Student [Your  Student

Number]-A3-notebook .ipynb

Number]-A3-notebook .pdf

Number]-A3-report .pdf

Number]-A3-presentation.mp4

Do not put them in a single zip file or you will not get the plagiarism report as it cannot run on archived files. If you do submit a single zip file, you will lose a mark, and we will still run the Turnitin report, but you will not get a copy of it, unless you end up getting flagged for plagiarism, and none of us wants that to happen. So for your own sanity and ours please submit the four files individually, named as shown above. If you do, you should be able to see the Turnitin results for your report and for your notebook.

GETTING HELP. There is a discussion board available in the course canvas. Please

do not post code snippets in this forum. Do ask questions if you have them! We will do our best to answer them as quickly as we can.  This is the best medium to ask a question, as there is a very strong chance that if you are confused about something, your classmates are too. We will also be running office hours in Week 13 from 3-4PM on Thursday in Collaborate Ultra if you want to stop by and ask a question.

Plagiarism Warning

University Policy on Academic Honesty and Plagiarism: It is University policy that cheating by students in any form is not permitted,  and that work submitted for assessment purposes must be the independent work of the student concerned. Please see the RMIT policy for more details:  https://www.rmit.edu.au/students/ my-course/assessment-results/academic-integrity.

THIS IS NOT A GROUP PROJECT. Students are reminded that this assignment is to be attempted individually. Plagiarism of any form will result in zero marks being given for this assessment, and can result in disciplinary action.  We routinely use plagiarism software on projects!  Please, please don’t do it.  Be aware that paying someone on a coding site to do it for you is a form of plagiarism. If you are submitting work that is not your own, regardless of how you got it – you are in breach of this policy.