Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

Assignment 4

Large-Scale Text Processing

MET CS777

Description

In this assignment you will implement k-nearest neighbor classifier (KNNclassifier) to classify text documents. For example, given a search text “ How many goals did

Vancouver score last year?”, the algorithm searches all the documents corpus (corpus: large and structural text) and returns the top K similar documents.

The TF-IDF (Term Frequency - Inverse Document Frequency) is used as the similarity/distance measure between two document/texts.

In the first step, the top 20k English words of the corpus will be selected, then the TF-IDF matrix of a text corpus get computed, which is used to find similarity between the texts.

Wikipedia dataset

In this assignment, the Wikipedia data set is used. The entire Wikipedia data set has been downloaded from (https://dumps.wikimedia.org) and stored in a large file.

Each Wikipedia Page is a document and have a unique document ID and a

specific URL. For example,

• docID 418348

• URLhttps://en.wikipedia.org/wiki?curid=418348

Data format

Each line is a single document in a pseudo XML format.

“Term Frequency” is an indication of the number of times a term occurs in a document. Numerator is number of occurrences of a word, and the denominator is the sum of all    the words of the document.

Next, calculate “Inverse Document Frequency” for all the documents and finally calculate TF-IDF(w) and create TF-IDF matrix of the corpus:

Note that the “size of corpus” is total number of documents (numerator). To learn more about TF-IDF see the Wikipedia page: https://en.wikipedia.org/wiki/Tf-idf

Task 3 - Implement the getPrediction function (30 Points)

Finally, implement the function getPrediction(textInput, k),which will predict the membership of the textInput to the top 20 closest documents, and the list of top categories.

You should use the cosine similarity to calculate the distances.

Task 4 (30 points): Implement the code using Dataframes

Implement the complete code in Dataframe and printout the results of the task 3 using dataframes in pyspark. From the beginning of your code to the end of your kNN implementation you are allowed to usespark dataframe and python (including python libraries like numpy). You are not allowed to use RDDs.

Task 5 (10 points) Removing Stop Words and Do Stemming

Task 5.1 - Remove Stop Words

Figure 1: Screenshot of Spark History

Evaluation Criteria for Coding Tasks

Academic Misconduct Regarding Programming

In a programming class like this, there is sometimes a very fine line between “cheating” and acceptable and beneficial interaction between peers. Thus, it is very important that  you fully understand what is and what is not allowed in terms of collaboration with your classmates. We want to be 100% precise,so that there can be no confusion.

The rule on collaboration and communication with your classmates is as follows: you cannot transmit or receive code from or to anyone in the class in anyway —visually (by  showing someone your code), electronically (by emailing, posting, or otherwise sending someone your code), verbally (by reading code to someone) or in any other way we have not yet imagined. Any other collaboration is acceptable.

It is not allowed to collaborate and communicate with people who are not your classmates (or your TAs or instructor). This means posting any questions of any nature to programming forums such as StackOverflow is strictly prohibited. As far as going to  the web and using Google, we will apply the “two-line rule”. Go to any web page you   like and do any search that you like. But you cannot take more than two lines of code   from an external resource and include it in your assignment in any form. Note that changing variable names or otherwise transforming or obfuscating code you found on  the web does not render the “two-line rule” inapplicable. It is still a violation to obtain more than two lines of code from an external resource and turn it in, whatever you do to those two lines after you first obtain them.

Furthermore, you must always cite your sources. Add a comment to your code that includes the URL(s) that you consulted when constructing your solution. This turns out to be very helpful when you’re looking at something you wrote a while ago and you need to remind yourself what you were thinking.