Hadoop SlowJob Network Issue Process

High Level Overview of Network Troubleshoot Scenario for Teragen/Sort

* Job is 50% slower compared to twin cluster
* Found only specific map tasks are slow
* correlated that in re-runs only 4 nodes are slow
* check cpu metrics, all is fine
* dmesg on host is fine, messages file is fine
* mapper log shows a dfsClient warn for a 2 minute lookup for HDFS files
* checked host network utilization vs other nodes and found it multiple factors below
* checked network configs to find some nodes have bonded nics and some do not.