Call for Papers for the

1ST ANU BIANNUAL BIO-INSPIRED COMPUTING STUDENT CONFERENCE

also being used for

COMP4660/8420 Assignment 1: Neural Networks

Submission Due: Sunday 29th April (Week 8) at 11:55pm

Context

Neural networks research prior to the deep learning boom has a lot to teach us still. The neural networks part of the course focuses on the research in this area from 1986 (back-propagation algorithm published) into the 1990s and early 2000s. Later papers tend to be too esoteric to be useful for course based learning and I prefer you to read papers rather than just textbooks.

The assignment is to provide you some experience with picking a small research topic in neural networks, to implement a method from a paper, test it, and write it up. To be possible as an assignment, we stick to simple (by modern standards) techniques, which fits well with the chronological focus from 1986 to early 2000s.

I have uploaded a number of papers relating to the neural network lectures. You pick one of them and either focus directly on that paper or on a paper they directly cite or are directly cited by. For the latter, in Google scholar you find the paper and look at the citing papers. The idea is to keep the focus relatively close to the topics covered in the neural networks part of the course.

Perhaps it is worth noting that some of the techniques developed in that period may be of use today. My example is the use of an input feature selection technique of mine from 1997 which has recently been implemented in the H2O AI package. In Google type: “Gedeon method” H2O and you get a surprising number of recent hits. I should also mention that I have no connections with H2O and found out about this when a Korean researcher emailed me to ask me about the function in H2O based on my paper. J Similarly, please also note that the CNN architecture dates to the 1980s (Fukushima’s Neocognitron in 1980 to LeCun’s coining of the term in 1989) but was not competitive given the computing power and datasets generally available.

Deliverables

1.  A report (about 4-6 pages of text content with a MAXIMUM of 10 pages, including references, diagrams, graphs and tables). Remember to keep your report clear and concise. I say that the text content should be about 4-6 pages to signal that we will not be counting lines, but if it is 10 pages long and it is all diagrams then it is very clear that there is too little text. Conversely, if it is 10 pages long and there is just one diagram then there is too much text, but 6 pages with one diagram is fine, and so on.

2.  PyTorch source code file(s).

3.  The data set you used for your assignment, in original and preprocessed form.

4.  A copy of the paper which you picked, and cited within your report. You should include the paper even if its one of the ones I posted in Wattle. This is for consistency, and makes marking easier as it clearly identifies the paper chosen.

Submission Method

Please submit your assignment via the EasyChair conference management system.

You will submit a pdf for the report, the name must start with your uNumber, and a zip folder of the other deliverables.

Objectives

The purpose for this assignment is for you to:

•  Develop a good understanding of artificial neural networks and enhance your skills in implementing them in PyTorch

•  Enhance your approach to investigating a real-world data set and setting up a problem for an artificial neural network to solve a classification/regression problem

•  Develop a good understanding for reporting investigations in a conference style paper

•  A small amount of experience in using Google scholar to find citing/cited papers

Task Description

Your task is to:

1.  devise a classification or regression problem to investigate using a real-world data set;

2.  implement a neural network in PyTorch to solve the problem and implement a method to determine the performance of the neural network;

3.  implement a technique from the literature (paper selected as described above) and determine its benefit or lack of benefit;

4.  compare your results with results published in a research paper reporting results on the data set you chose (see below); and

5.  write a report on your work

Data set

Choose a data set from the UCI Machine Learning Repository located at http://archive.ics.uci.edu/ml/datasets.html or a data set of your choice. Note that you may not choose the Iris or any dataset that we have used in the labs or which comes preloaded into PyTorch. You should choose a relatively complex data set with (say) a minimum 300 instances and (say) at least 15 attributes (features). You should find it difficult to obtain near perfect results, as you will be marked on the steps taken to achieve better results and how they compare with previous results, not on the results themselves. Ensure that the data set you choose has a relevant academic paper associated with it to compare your results with. Relevant papers are often found on the UCI repository website. The paper related to the dataset can be of a similar age to the technique paper you chose, you do not need to find the best modern result to compare to.

Your report should indicate which dataset you used and how to get it, what modifications you made to the encodings if you needed to do so, and cite a research paper with results on the data set to compare to your results, and the technique paper you chose. Other academic papers can generally be found by using Google scholar. Google scholar will have links to one or more paper repositories. You can get access if you do this from ANU campus as the library subscribes to most of the large document repositories. Some papers are in multiple repositories so if you cannot find a free copy then ask your tutor for advice. From off campus, if you log into the ANU Virtual Proxy server then usually you can access the same electronic resources. You should also mention in your report any different topologies and analyses used in your experimentation.

Design of a Problem

Detail in your report what you want to model in the data set and explain what the inputs and outputs you will use to develop the neural network model.

Implementation

Choose an appropriate measure to report the results produced by your neural networks. You can use the measure used in the research papers that have the results of their predictions on the data set you have used and would like to compare your results with. Remember to cite the papers in your report.

Report

The report must be in the style of an academic paper and must conform to the Lecture Notes in Computer Science conference paper proceedings format, but with the margins changed to 2 cm and header/footer to 1 cm. The template for the report can be downloaded from Wattle by clicking the link named “AssignmentReportTemplate-LNCS-Office2007.zip” and needs the margins to be modified. If you prefer to use Latex , there is a link on the ABCs conference website which takes you to Springer’s current pages. Any template from there will also need the margins to be modified. Use the Springer citation style as found in the template file you use.

Your report should have a meaningful title, which indicates what you have done. "COMP8420 Assignment" is not a meaningful title. Your title and content should NOT mention the course: we are modelling the assignment so that you are making a conference paper submission. Your u number should be showing in your email address. Your affiliation would be "Research School of Computer Science, Australian National University" etc. Please do not forget to include your name as the single author.

A suggested structure for the report:

•  Abstract – A paragraph that summarises the work you did, the results you found and whether it was better, same or worse than a published research paper for the same dataset. An abstract is similar to an executive summary of the entire report.4

•  Introduction – A description of the motivation for the choice of the data set, the problem that you modelled using a neural network and an outline of the investigations that you carried out using the model. This section should also include a brief background to the problem and the methods used perform the analysis. Remember to use citations!

•  Method – A description of the technique(s) you implemented and details of the investigations or tests you conducted using the technique.

•  Results and Discussion – Presentation of results from the investigations and detailed analysis of the results including comparison of your results with the results published in a research paper on the data set. Remember to use citations!

•  Conclusion and Future Work – A statement on your findings and how your work can be extended or how might it be improved. Even if you have conducted a thorough investigation there is always work left to do. Outlining future work is VERY important as it shows that you have thought about the problem and have a deeper understanding then just stating a conclusion.

A comment: you should read papers in relevant areas of the literature (e.g. on similar topics) to get more of a feel for how to write and layout academic papers. This forms part of the learning you should get out of advanced courses such as COMP4xxx/COMP8xxx given that the ANU is a research intensive University.

Peer Marking (optional)

Conference papers need reviewers, and conferences need a Program Committee. The plan is that all of you are the program committee, the program chair(s) will assign you some papers to review and you will either do them or not (the peer marking is optional). You will get marks for how well you do the reviews. How well means whether you gave appropriate criticism and helpful suggestions or not (a too critical or too uncritical review can be equally bad). I hope many of you will chose to do the peer marking. Any low marks can be redeemed in the final exam. There will be more details about the peer marking when we get closer to it, the reports need to be submitted before any peer marking is possible.

Second assignment

We will model the second assignment as either an extension of this assignment (using deep learning / evolutionary algorithms) or as a separate assignment. Currently the plan is to continue with the conference idea.

Conference

Will we really hold it as a physical conference? I hope we do go beyond a virtual conference, but it will be up to you all. We will discuss it during the rest of the semester.

Assessment Guide ( /30 i.e. marked out of 30)      worth 20% of overall course marks

Abstract ( /2)

Clear and concise abstract summarising the work done

Method section / Data Set and Model Design ( /8)

Valid reasons for choosing the data set

Good level of problem complexity

Clear and valid investigation aims

Model design that clearly serves the purpose of the investigations

Appropriate measures used to determine the performance of the neural network and predictions

Good explanation of the model design

Appropriate choice for the inputs and outputs of the prediction model with valid reasons

Evidence of good understanding of the relevant literature

Results and Discussion ( /6)

Good methods used to evaluate the model including an appropriate split of the train, test / validation data

Good level of detail used to analyse results

Conclusion and Future Work ( /2)

Appropriate conclusion of the work

Appropriate work suggested to extend and/or improve the work

References and citations ( /2)

References used as appropriate

Presentation ( /6)

Structure of the code is legible and well organised

Good report structure

Report is legible, clear and concise

Clear presentation of results including appropriate use of figures, tables and graphs

Report conforms to required style (including the use of appropriate language) and length6

Consistent style used for citing references

Correct grammar and spelling

Implementation ( /4)

Evidence of good code design - appropriate level of modularity, encapsulation and reusability of the code

Code is comprehensible with appropriate names of coding items e.g. code files, functions, variables

Code executes without errors

Good comments in the code