Description

In this assignment you will implement Batch Gradient Descent to fit a line into a two-dimensional data set. You will implement a set of Spark jobs that will learn parameters of a line model using the New York City Taxi trip in the Year 2013 dataset. The dataset was released under the FOIL (The Freedom of Information Law) and made public by

Chris Whong (https://chriswhong.com/open-data/foil_nyc_taxi/). See the Assignment 1 for details of the dataset.

In this assignment you will implement a linear model using travel distance in miles to predict the fare amount, the money that is paid to the taxis. 

Please note that it is NOT allowed to useSpark ML and MLlib libraries in this assignment.

Taxi Dataset

This is the same dataset as Assignment 1. Please refer to there for the data description.

The dataset is in Comma Separated Volume Format (CSV). When you read a line and

split by comma sign (”,”), you will get a string array of length 17. Index number startes  from zero. For the assignment, you need to get trip distance in miles (index 5) and fare amount in dollars (index 11) as stated in the following table.

Table 1: Taxi Data Set fields

Data Clean-up Step

•    Remove all taxi rides that are less than 2 mins or more than 1 hour.

•    Remove all taxi rides that have ”fare amount” less than 3 dollars or more than 200 dollars

•    Remove all taxi rides that have “trip distance” less than 1 mile or more than 50 miles.

•    Remove all taxi rides that have “tolls amount” less than 3 dollars.

Note: Preprocess the data and store the results in your cluster storage.

You can download or access the datasets using following internal URLs:

Table 2: Datasets on AWS

Small dataset (93 MB compressed, uncompressed 384 MB) is for code development, implementation, and testing with roughly 2 million taxi trips.

Tasks

Task 1 (20 points): Simple Linear Regression

Find a simple line regression model to predict “fare amount” from the travel distance.

Consider a Simple Linear Regression model given in equation (1). The solutions for slope of the line, m, andy-intercept, b, are calculated in the equations (2) and (3).

Implement a PySpark job that calculates the exact answers for the parameters m and b.

Run your implementation on the large data set and report the computation time for your Spark Job for this task. You can get the execution time for the cloud services that you use.

Note: Execution of this task on the large dataset depends on your implementation and can take a long time. For example, on a cluster with 12 cores in total, it takes more than 40 min computation time.

Task 2 (40 Points) Find the Parameters using Gradient Descent

In this task, you should implement the batch gradient descent to find the optimal parameters for the Simple Linear Regression model.

• Load the data into spark cluster memory as RDD or Dataframe.

• Start with all parameter’s initial values equal to 0.1.

• Run for gradient descent for 100 iterations.

Cost function will be:

Evaluation Criteria for Coding Tasks

Academic Misconduct Regarding Programming

In a programming class like this, there is sometimes a very fine line between “cheating” and acceptable and beneficial interaction between peers. Thus, it is very important that  you fully understand what is and what is not allowed in terms of collaboration with your classmates. We want to be 100% precise,so that there can be no confusion.

The rule on collaboration and communication with your classmates is as follows: you cannot transmit or receive code from or to anyone in the class in anyway —visually (by showing someone your code), electronically (by emailing, posting, or otherwise sending someone your code), verbally (by reading code to someone) or in any other way we have not yet imagined. Any other collaboration is acceptable.

It is not allowed to collaborate and communicate with people who are not your classmates (or your TAs or instructor). This means posting any questions of any nature to programming forums such as StackOverflow is strictly prohibited. As far as going to  the web and using Google, we will apply the “two-line rule”. Go to any web page you like and do any search that you like. But you cannot take more than two lines of code from an external resource and include it in your assignment in any form. Note that changing variable names or otherwise transforming or obfuscating code you found on the web does not render the “two-line rule” inapplicable. It is still a violation to obtain more than two lines of code from an external resource and turn it in, whatever you do to those two lines after you first obtain them.