Hi There,

I'm test driving Hadoop with CephFS as the storage layer. I was running the
Terasort benchmark and  I noticed a lot of network IO activity when
compared to a HDFS storage layer setup. (Its a half-a-terabyte sort
workload over two data nodes.)

Digging into the job tracker logs a little, I noticed that all the map
tasks were being assigned to process a split (block)  on non-local nodes
(which explains all the network activity during the map phase)

With Ceph:


2013-07-08 11:19:53,535 INFO org.apache.hadoop.mapred.JobInProgress: Input
size for job job_201307081115_0001 = 500000000000. Number of splits = 7452
2013-07-08 11:19:53,538 INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201307081115_0001 initialized successfully with 7452 map tasks and 32
reduce tasks.

2013-07-08 11:19:54,836 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing a non-local task task_201307081115_0001_m_000000
2013-07-08 11:19:54,836 INFO org.apache.hadoop.mapred.JobTracker: Adding
task (MAP) 'attempt_201307081115_0001_m_000000_0' to tip
task_201307081115_0001_m_000000, for tracker 'tracker_vega7250:localhost/
127.0.0.1:35422'

2013-07-08 11:19:54,990 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing a non-local task task_201307081115_0001_m_000001
2013-07-08 11:19:54,990 INFO org.apache.hadoop.mapred.JobTracker: Adding
task (MAP) 'attempt_201307081115_0001_m_000001_0' to tip
task_201307081115_0001_m_000001, for tracker 'tracker_vega7249:localhost/
127.0.0.1:36725'

... and so on.

In comparison with HDFS, the job tracker logs looked something like this.
The maps tasks were being assigned to process data blocks on the local
nodes.

2013-07-08 03:55:32,656 INFO org.apache.hadoop.mapred.JobInProgress: Input
size for job job_201307080351_0001 = 500000000000. Number of splits = 7452
2013-07-08 03:55:32,657 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201307080351_0001_m_000000 has split on node:/default-rack/vega7247
2013-07-08 03:55:32,657 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201307080351_0001_m_000001 has split on node:/default-rack/vega7247
2013-07-08 03:55:34,474 INFO org.apache.hadoop.mapred.JobTracker: Adding
task (MAP) 'attempt_201307080351_0001_m_000000_0' to tip
task_201307080351_0001_m_000000, for tracker 'tracker_vega7247:localhost/
127.0.0.1:43320'
2013-07-08 03:55:34,475 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201307080351_0001_m_000000
2013-07-08 03:55:34,475 INFO org.apache.hadoop.mapred.JobTracker: Adding
task (MAP) 'attempt_201307080351_0001_m_000001_0' to tip
task_201307080351_0001_m_000001, for tracker 'tracker_vega7247:localhost/
127.0.0.1:43320'
2013-07-08 03:55:34,475 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201307080351_0001_m_000001

Version Info:
ceph version 0.61.4
hadoop 1.1.2

Has anyone else run into this ?

Thanks
KC
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to