Hi, Cosmin, Have you tried using DISTRIBUTE BY to distribute the query's data by the partitioning columns? That way all the data for each partition should be sent to the same reducer and should be written to a single file in each partition, I think.
If your data is being distributed by a different criteria, you will potentially have multiple reducers writing to the same partitions. Andre On 29 January 2014 10:51, Cosmin Cătălin Sanda <[email protected]>wrote: > Hi, > > I have a number of Hive jobs that run during a day. Each individual job > is outputting data to Amazon S3. The Hive jobs use dynamic partitioning. > > The problem is that when different jobs need to write to the same dynamic > partition, they will each generate one file. > > What I would like is for the subsequent jobs to load the existing data and > merge it with the new data. Can this be achieved somehow? Is there an > option that needs to be enabled? I already set: > > SET hive.merge.mapredfiles = true; > SET hive.exec.dynamic.partition = true; > SET hive.exec.dynamic.partition.mode = nonstrict; > > I should mention that the query that actually outputs to S3 is an INSERT > INTO TABLE query. The Hive version is 0.8.1 > > > Thank you, > Cosmin > -- André Araújo Big Data Consultant/Solutions Architect The Pythian Group - Australia - www.pythian.com Office (calls from within Australia): 1300 366 021 x1270 Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270 Mobile: +61 410 323 559 Fax: +61 2 9805 0544 IM: pythianaraujo @ AIM/MSN/Y! or [email protected] @ GTalk “Success is not about standing at the top, it's the steps you leave behind.” — Iker Pou (rock climber) -- --
