Hi Wei Here there are two operations that takes place for your query insert OVERWRITE LOCAL DIRECTORY '/disk2/iis1' select * where impressionid<'1239572996000'
1 - A map reduce job that performs the operation select * where impressionid<'1239572996000 2 - A file system operation that copies the output of Step 1 from hdfs to lfs (hadoop fs -copyToLocal). Step 2 would be executed only after completion of Step 1. Regards Bejoy.K.S ________________________________ From: "Lu, Wei" <[email protected]> To: "[email protected]" <[email protected]> Sent: Wednesday, March 7, 2012 5:12 PM Subject: Why Move Operations after MapReduce are in sequential? Hi, For the query below, I find the five Move Operations (after MapReduce job) are not operated in parallel. from impressions2 insert OVERWRITE LOCAL DIRECTORY '/disk2/iis1' select * where impressionid<'1239572996000' insert OVERWRITE LOCAL DIRECTORY '/disk2/iis2' select * where impressionid<'1239592780000' AND impressionid>='1239572996000' insert OVERWRITE LOCAL DIRECTORY '/disk2/iis3' select * where impressionid<'1239648597000' AND impressionid>='1239592780000' insert OVERWRITE LOCAL DIRECTORY '/disk2/iis4' select * where impressionid<'1239714028000' AND impressionid>='1239648597000' insert OVERWRITE LOCAL DIRECTORY '/disk2/iis5' select * where impressionid>='1239714028000'; ------ Ended Job = job_201203060735_0008 Copying data to local directory /disk2/iis1 Copying data to local directory /disk2/iis1 Copying data to local directory /disk2/iis2 Copying data to local directory /disk2/iis2 Copying data to local directory /disk2/iis3 Copying data to local directory /disk2/iis3 Copying data to local directory /disk2/iis4 Copying data to local directory /disk2/iis4 Copying data to local directory /disk2/iis5 Copying data to local directory /disk2/iis5 ------ I thought the Move Operations could be done in parallel, and the performance will be improved is the MapReduce temp result is pretty large. Regards, Wei
