Data Analytics ECS648U/ ECS784U/ ECS784P Coursework 2 specification for 2022

Coursework 2 specification for 2022

Data Analytics ECS648U/ ECS784U/ ECS784P

1. Important Dates

•   Release date: Week 9, Monday 21st March 2022 at 10:00 AM.

•   Submission deadline: Week 13, Wednesday 20th April 2022 at 10:00 AM.

•   Late submission deadline (penalty applies): Within 7 days after deadline.

General guidelines:

i.      Occasionally someone will upload their submission and forget to hit the submit button. When this happens, the submission will be marked as ‘Draft’ . Make sure you fully complete the submission process.

ii.     A penalty will be applied automatically by the system for late submissions.

a.   Lecturers cannot grant extensions and cannot remove the penalty.

b.   Extensions   can   only   be   granted   through   approval   of  an   Extenuating Circumstances (EC) claim. Details can be found on your Student Support page, including how to submit an EC claim along with deadline dates.

c.   Penalties are automatically applied by the system can only be challenged via submission of an EC.

d.   If you submit an EC form, your case will be reviewed by the EC panel. When the  panel  reaches  a  decision,  they  will  inform  both  you  and  the  Module Organiser.

e.   If you miss both the submission deadline and the late submission deadline, you will automatically receive a score of 0.

iii.      Submissions via e-mail are not accepted.

iv.      For details about submission regulations, please refer to your student handbook.

2. Coursework overview

Coursework 2 involves applying causal structure learning to a data set of your choice. You will have to complete a series of tasks, and then answer a set of questions.

•   This coursework is based on the lecture material covered between Weeks 6 and 12, and on the lab material covered between Weeks 9 and 11.

•   The coursework must be completed individually.

•   Submission should be a single file (Word or PDF) containing your answers to each of the questions. Ensure you clearly indicate which answer corresponds to what question. Data sets and other relevant files are not needed, but do save them in case we ask to have a look at them.

•   To  complete the  coursework,  follow the  tasks below  and  answer ALL  questions enumerated in Section 3. It is recommended that you read this document in full before you start completing Task 1.

•   You can start going through Task 1 as early as you want. If you face difficulties, we recommend that you try again after you attend the Week 9 lab, scheduled for Friday 25th March.

•   You can start working on your answers as early as you want, but keep in mind that you need to attend up to Week’s 11 material to gain the knowledge needed to answer all the questions.

TASK 1: Set up and reading

a)  Visithttp://bayesian-ai.eecs.qmul.ac.uk/bayesys/

b)  Download the Bayesys user manual.

c)   Set up the NetBeans project by following the steps in Section 1 of the manual.

d)  Read Sections 2, 3 and 4 of the manual.

e) Skip Section 5.

f)   Read Section 6 and repeat the example.

i. Skip subsections 6.3 and 6.4.

g)  Read Section 7 and repeat the example.

h) Skip Sections 8, 9 and 10.

i)   Read Section 11.

i. Skip subsection 11.6.

TASK 2: Determine research area and prepare data set

You are free to choose or collate your own data set. As with Coursework 1, we recommend that you address a data-related problem in your professional field or a field you are interested in. If you are motivated in the subject matter, the project will be more fun for you, and you will likely perform better. For your convenience, we have copied the data sources from Coursework

1 into this document (refer to Section 4).

Data requirements:

• Number of variables: The data set must contain a minimum of 8 variables (penalty applies below 8 variables).

While there is no upper bound restriction on the number of the variables, we strongly recommend using considerably less than 100 variables for the purposes of the coursework (but there is no penalty for using more variables). For example, using around 30 or less variables will make it much easier for you to visualise the causal graph, but will also improve learning speed significantly (some algorithms might take hours to complete when given  100 variables!). You do not need to use a special technique for feature selection – it is up to you to decide which variables to keep. We will not be assessing feature selection decisions.

• Re-use data from CW1: You are allowed to reuse the data set you have prepared for Coursework 1, as long as: a) you consider that data set to be suitable for causal structure learning (refer to Q1 in Section 3), and b) it contains at least 8 variables.

• Sample size: There are no restrictions. If algorithms take a long time to complete, you are also free to use part of the samples to speed up the learning process (you do not need permission from the lecturer to do this).

• Bayesys repository: You are not allowed to use any of the data sets available in the Bayesys repository for this coursework.

TASK 3: Pre-process your data set for structure learning

•   Bayesys     assumes     the     input     data     are     categorical     or     discrete;     e.g., {"low", "medium", "high"},  {"yellow", "blue", "green"},  {" < 10", "10-20", "20 + "} etc, rather than a continuous range of numbers. If your data set contains continuous variables, Bayesys will consider each of the variable values as a different category. This will cause problems with model dimensionality, leading to poor accuracy and high runtime (if this is not clear why, refer to the Conditional Probability Tables (CPTs) covered in the lectures).

To address this issue, you should discretise all continuous variables to reduce the number of states to reasonable levels. For example, a variable with continuous values ranging from 1 to 100 (e.g., {"14.34", "78.56", "89.23"}) can be discretised into categories  such  as  {"1to20", "21to40", "41to60", "61to80", "81to100"}.  Because Coursework 2 is not concerned with data pre-processing, you are free to follow any approach you wish to discretise continuous variables. You could discretise the variables manually as discussed in the above example, or even use K-means, but you will not be rewarded extra marks for choosing a more sophisticated approach. We simply need the input data set to be categorical, and it is up to you how you accomplish this.

•   Your data set must not contain missing values/empty cells. If it does, replace ALL empty cells with a new category value called missing (or use a different relevant name). This will force the algorithms to consider missing values as an additional state. If missing data follow a pattern, this may or may not help the algorithm to produce a more accurate graph, but this is out of the scope of Coursework 2.

Once you ensure your data set is consistent with what has been stated above, rename your data set to trainingData.csv and place it in folder Input.

TASK 4: Draw out your knowledge-based graph

1.   Use your personal knowledge to produce a knowledge-based causal graph based on the variables you decide to keep in your data set. Remember that this graph is based on your knowledge, and it is not necessarily correct or incorrect.

You may find it easier ifyou start drawing the graph by hand, and then recording the directed relationships in the DAGtrue.csv file. In creating your DAGtrue.csv file, we recommend that you edit one of the sample files that come with Bayesys; e.g., you can create a copy of the DAGtrue_ASIA.csv file available in the directory Sample input files/Structure learning, then rename the file to DAGtrue.csv, and then replace the

directed relationships with those present in your knowledge graph.

2.   Once you  are happy with the  graph you have prepared,  ensure the  file  is  called DAGtrue.csv and placed in folder Input.

NOTE: If your OS is not showing the file extensions (e.g., “ .csv”, “ .pdf”), name your file DAGtrue and not DAGtrue.csv; otherwise, the file might end up being called DAGtrue.csv.csv unintentionally (when the file extension is not visible). Ifthis happens, Bayesys will be unable to locate the file.

3.   Make a copy of the DAGtrue.csv file, and rename this copy into DAGlearned.csv and place it in folder Output. You can discard the copied file once you complete Task 4.

4.   Ensure that your DAGtrue.csv and trainingData.csv files (as per Task 3) are in folder Input, and the DAGlearned.csv file is in folder Output. Run Bayesys in NetBeans. Under tab Main, select Evaluate graph and then click on the first subprocess as shown below. Then hit the Run button found at the bottom of tab Main.

The  above  process  will  generate  output  information  in  the  terminal  window  of NetBeans. Save the last three lines, as highlighted in the Fig below; you will need this information later when answering some of the questions in Section 3.

Additionally, the above process should have generated one PDF files in folder Input called DAGtrue.pdf. Save this file as you will need it for later.

This only concerns MAC/Linux users: The above process might return an error while creating the PDF file, due to compatibility issues. Even if the system completes the process without errors, the PDF files generated may be corrupted and won`t open on MAC/Linux. If this happens, you should use the online GraphViz editor to produce your graphs,  available here: https://edotor.net/ , which  converts text  into  a visual drawing. As an example, copy the code shown below in the web editor:

digraph {

Earthquake -> Alarm

Burglar -> Alarm

Alarm -> Call

}

You can then edit the above code to be consistent with your DAGtrue.csv. You can actually copy-and-paste the variable relationships (e.g., Earthquake → Alarm) directly from DAGtrue.csv into the code editor, since they share the same format.

TASK 5: Perform structure learning

1.   Run Bayesys. Under tab Main, select Structure learning and algorithm HC_CPDAG (default selection). Select Evaluate graph and then click on the last two (out of four) options so that you also generate the learned DAG and CPDAG in PDF files, in addition to the DAGlearned.csv file which is generated by default. Then, hit the Run button.

2.   Once the above process completes, you should see:

i.      Relevant text generated in the terminal window of NetBeans.

ii.      The files DAGlearned.csv, DAGlearned.pdf and CPDAGlearned.pdf generated in folder Output. As stated in Task 4, the PDF files may be corrupted on MAC/Linux, and you will have to use the online GraphViz editor to produce the graph corresponding to DAGlearned.csv (simply copy the relationships from the CSV file into the editor as discussed in Task 4).

3.   Repeat  the  above  process  for  the  other  five  algorithms;  i.e.,  HC,  TABU_DAG, TABU_CPDAG, SaiyanH, and MAHC. Save the same output information and files that each algorithm produces (ensure you first read the NOTE below).

NOTE: As stated in the manual, Bayesys overwrites the output files every time it runs. You need to remember to either rename or move the output files to another folder before running the next algorithm.

Similarly, if you happen to have one of the output files open – for example, viewing the DAGlearned.pdf in Adobe Reader while running  structure  learning  - Bayesys will fail to replace the PDF file, and the output file will not reflect the latest iteration. Therefore, ensure you close all output files before running structure learning.

3. Questions

This coursework involves applying six different structure learning algorithms to your data set. We do not expect you to have a detailed understanding of how the algorithms operate. None of the  Questions  focuses  on  the  algorithms  and  hence,  your  answers  should not focus  on discussing differences between algorithms.

•   You should answer ALL questions.

•   You should answer the questions in your own words.

•   Do not exceed the maximum number of words specified for each question. If a question restricts the answer to, say 100 words, only the first 100 words will be considered when marking the answer.

•   Marking is out of 100.

QUESTION 1: Discuss the research area and the data set you have prepared, along with pointers to your data sources. Screen-capture part of your final version of your data set and present it here as a Figure. For example, if your data set contains  15 variables and  1,000 samples, you could present the first 10 columns and a small part of the sample size. Explain why you considered this data set to be suitable for structure learning, and what questions you expect a structure learning algorithm to answer.

Maximum number of words: 150 Marks: 10

QUESTION 2: Present your knowledge-based DAG (i.e., DAGtrue.pdf or the corresponding DAGtrue.csv graph visualised through the web editor), and briefly describe the information you have considered to produce this graph. For example, did you refer to the literature to obtain the necessary knowledge, or did you consider your own knowledge to be sufficient for this problem? If you referred to the literature to obtain additional information, provide references and very briefly describe the knowledge gained from each paper. If you did not refer to the literature, justify why you considered your own knowledge to be sufficient in determining the knowledge-based graph.

NOTE: It is possible to obtain maximum marks without referring to the literature, as long as you clearly justify why you considered your personal knowledge alone to be sufficient. Any references provided will not be counted towards the word limit.

Maximum number of words: 200 Marks: 10

QUESTION 3: Complete Table Q3 below with the results you have obtained by applying each of the algorithms to your data set during Task 5. Compare your CPDAG scores produced by F1, SHD and BSF with the corresponding CPDAG scores shown in Table 2.1 (page 12) in the Bayesys manual.

Specifically, are your scores mostly lower, similar, or higher compared to those shown in Table 2.1 in the manual? Why do you think this is? Is this the result you expected? Explain why.

Table Q3. The scores of the six algorithms when applied to your data set.

Maximum number of words: 250 Marks: 15

QUESTION 4: Present the CPDAG generated by HC_CPDAG (i.e., CPDAGlearned.pdf or the corresponding CPDAGlearned.csv graph visualised through the web editor). Highlight the three causal classes in the CPDAG. You only need to highlight one example for each causal class. If a causal class is not present in the CPDAG, explain why this might be the case.

Maximum number of words: 200 Marks: 10