Hi There, I'm test driving Hadoop with CephFS as the storage layer. I was running the Terasort benchmark and I noticed a lot of network IO activity when compared to a HDFS storage layer setup. (Its a half-a-terabyte sort workload over two data nodes.)
Digging into the job tracker logs a little, I noticed that all the map tasks were being assigned to process a split (block) on non-local nodes (which explains all the network activity during the map phase) With Ceph: 2013-07-08 11:19:53,535 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201307081115_0001 = 500000000000. Number of splits = 7452 2013-07-08 11:19:53,538 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201307081115_0001 initialized successfully with 7452 map tasks and 32 reduce tasks. 2013-07-08 11:19:54,836 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task task_201307081115_0001_m_000000 2013-07-08 11:19:54,836 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307081115_0001_m_000000_0' to tip task_201307081115_0001_m_000000, for tracker 'tracker_vega7250:localhost/ 127.0.0.1:35422' 2013-07-08 11:19:54,990 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task task_201307081115_0001_m_000001 2013-07-08 11:19:54,990 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307081115_0001_m_000001_0' to tip task_201307081115_0001_m_000001, for tracker 'tracker_vega7249:localhost/ 127.0.0.1:36725' ... and so on. In comparison with HDFS, the job tracker logs looked something like this. The maps tasks were being assigned to process data blocks on the local nodes. 2013-07-08 03:55:32,656 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201307080351_0001 = 500000000000. Number of splits = 7452 2013-07-08 03:55:32,657 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307080351_0001_m_000000 has split on node:/default-rack/vega7247 2013-07-08 03:55:32,657 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307080351_0001_m_000001 has split on node:/default-rack/vega7247 2013-07-08 03:55:34,474 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307080351_0001_m_000000_0' to tip task_201307080351_0001_m_000000, for tracker 'tracker_vega7247:localhost/ 127.0.0.1:43320' 2013-07-08 03:55:34,475 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307080351_0001_m_000000 2013-07-08 03:55:34,475 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307080351_0001_m_000001_0' to tip task_201307080351_0001_m_000001, for tracker 'tracker_vega7247:localhost/ 127.0.0.1:43320' 2013-07-08 03:55:34,475 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307080351_0001_m_000001 Version Info: ceph version 0.61.4 hadoop 1.1.2 Has anyone else run into this ? Thanks KC
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
