Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit 

COMP814 Text Mining

Assignment

Assignment

This is assignment can be done either individually or in pairs. You are recommended to do it in a pair as you will also gain teamwork skills. If you are new to coding then it is recommended that you pair up with someone with some coding skills. Only one person from the pair needs to submit the assignment.

 

If you are doing the assignment as a pair you should write approximately 12 pages and if you are doing it individually you should write approximately 6 pages, both excluding the references and appendices.

 

Objective

1. To be able to carry out a typical text mining task based on an objective.

2. To document the methodology and the findings in an appropriately formatted scientific paper suitable for publication in a conference.

 

Task Resources

You will be using models and code snippets that you developed as part of the labs in the python environment. You will use the dataset provided on Canvas as a zipped file named AssignmentBlogData.7z.

 

Your dataset consists of a set of 19,320 xml formatted text files. These files contain blogs collected from an anonymous blogging site which have been annotated with various types of anonymised metadata. The metadata has been integrated into the filenames. The text in each of the files contains the blogs corresponding to a blogger (as described in the metadata) with blog dates ranging from approximately 2001 to 2004.

 

Task Brief

You are employed by an innovation company who has bought the blogs with the objective of innovating new products/services based on what people have been talking about on popular blog sites.

 

In particular your boss wants to know the two most popular topics that the bloggers have been talking about in the following demographics :

a) Males

b) Females

c) Age brackets <=20 and over 20.

d) Students

e) Everyone

 

The dataset contains 19,320 blogs. If this happens to be a size that your computer takes too long to process, the task is such that you can segment the dataset and process it in batches. Also, this a raw dataset so you might have to deal with noise in the data, such as taking care of non-ascii characters.

Task Requirements

In order to achieve the objectives of the project, you will firstly need to read in the data, extract the meta data and segment it into the required demographics.

You will then need to design strategies to extract and cluster topics.

 

To be consistent within the class, let us use the same definition for a topic. Let us define a topic to be the mention of an OBJECT or a THING. So, for instance, you could simply take the THING that is mentioned as the highest number of times as the popular topic and correspondingly the second most popular topic. Conversely, you could also consider a topic to be the most popular noun and its “synonyms”.

 

Once you get the two most dominant “things” mentioned, expand the topic to be 2 verb/noun before and 2 verb/noun after the topic. Output them as “what has been said about the dominant “thing” in terms of the 4 surrounding nouns/verbs.

 

Repeat what you have done using frequency above, but this time use TFIDF. For this, consider all the blogs from one person as a document.

 

Compare the results from the two modes of counting and comment on which one is more accurate in your opinion with justifications.

 

Note that you will need to use various techniques such as stemming, lemmatization, PCA, stop word removal, inter alia, in order to get as accurate results as possible. The results will need to be evaluated manually and the strategy for evaluation should be described in your writeup.  

 

Write up

 

1. You need to document the research project as a scientific paper using latex double column IEEE conference format. The latex template can be downloaded from Canvas.

 

2. You should also submit a well commented and formatted python code as part of the appendix.

 

3.  Your paper should describe:

 

a) The task you set out to solve.

b) A literature review of same or similar tasks attempted by other researchers.

c) The details of your strategy to solve the problem. In this part you should describe the details of how you processed the data from start to finish including the details of how the data got processed in any external library you have used (if you have used it). This should explicitly describe the strategy used to extract the 2 most popular topics.

d) How you ensured the accuracy of your results.

e) The conclusion and how you would do the task differently if you were to do it again.

 

Approximate marking scheme.

 

Part of Assignment	Mark
Research question and rationale description	10
Data description and analysis	15
Research Design	30
Implementation (code) submitted as appendix	15
Analysis and Evaluation	20
Conclusion, formatting and references	10
Total	100

 

Treat this as a learning experience rather than an assessment exercise. The assignment is constrained enough for you to do exactly as specified and pass it, and flexible enough for you to be creative and attempt to get more accurate results, hence a better grade.