ECS765P Big Data Processing 2020-2021

Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit


ECS765P (2020-2021)

ECS765P Big Data Processing

SOLUTIONS AND MARKING SCHEME


Question 1

This question is about reliability and scalability in Hadoop.

(a)  Consider the following scenario. While executing a MapReduce job two replicated

blocks of a HDFS block with a replication factor of 3 are marked as corrupt, a Node- Manager fails and the ApplicationManager also fails. Does this prevent the job from running successfully? For the job to be successful the results must be the same as they would be if no failures occurred. Justify your answer.

[10 marks — basic]


Solution:

The job executes successfully 2 . Even though a majority of the replicas of

the data block are corrupt, the integrity of blocks is determined via a checksum rather than a majority decision . The failure of a NodeManager results in the ResourceManager restarting containers on a different nodes after negotiation with the ApplicationManager (This assumes that the ApplicationManager is restarted successfully. Students can also make an argument that the job failed if they follow this line of reasoning which is acceptable) 2 . When the ApplicationManager fails a new ApplicationManger is started by the ResourceManager on a different node. It will recover the job progression from the JobHistoryServer and will proceed with the job execution 3 .

Marking scheme:

for determining whether the jobs succeeds or fails.       for correct reasoning about the HDFS blocks.       for correct reasoning about the NodeManager. for correct reasoning about the ApplicationManager

(b)  There are seven stages to a MapReduce job:

•  Job setup

•  Load Split

•  Map

•  Copy

•  Merge

•  Reduce

•  Write part

Consider a scenario where there are 10,000 mapping tasks each with a duration of 5 seconds which yield 20 intermediate keys to reducer tasks.  Each mapping task requires 1 second to access its load split. The reducer tasks have a duration of 30 seconds. Each reducer requires 2 seconds to write information to HDFS. The job setup time is 20 seconds and copy/merge stages have a duration of 90 seconds. Assume the duration of the copy/merge stages does not depend on the number of reducers. Consider a cluster of 1,000 workers. You can assume that the ApplicationManager is not on one of the workers. What is the minimum execution time of the workload on this cluster? Explain your reasoning for your figure.


[10 marks — medium]








2021-12-31