RE: Hadoop streaming with insert dynamic partition generate many small files

2014-02-26 Thread Bogala, Chandra Reddy
quot;sum_packets\":61,\"sum_flows\":35,\"rank\":1}, {\"protocol\":\"tcp\",\"sum_bytes\":20469,\"sum_packets\":229,\"sum_flows\":10,\"rank\":2}, {\"protocol\":\"icmp\",\"sum_bytes\&qu

RE: Hadoop streaming with insert dynamic partition generate many small files

2014-02-17 Thread Bogala, Chandra Reddy
g?. What's the reason absolute path not accepted in below stream reduce. From: Bogala, Chandra Reddy [Tech] Sent: Thursday, February 13, 2014 10:42 PM To: 'user@hive.apache.org' Subject: RE: Hadoop streaming with insert dynamic partition generate many small files Thanks Wang.

RE: Hadoop streaming with insert dynamic partition generate many small files

2014-02-13 Thread Bogala, Chandra Reddy
Wang [mailto:chen.apache.s...@gmail.com] Sent: Tuesday, February 04, 2014 3:00 AM To: user@hive.apache.org Subject: Re: Hadoop streaming with insert dynamic partition generate many small files Chandra, You don't necessary need java to implement the mapper/reducer. Checkout the answer in

Re: Hadoop streaming with insert dynamic partition generate many small files

2014-02-03 Thread Chen Wang
2014 12:26 PM > *To:* user@hive.apache.org > *Subject:* Re: Hadoop streaming with insert dynamic partition generate > many small files > > > > it seems that hive.exec.reducers.bytes.per.reducer is still not big > enough: I added another 0, and now i only gets one file under ea

RE: Hadoop streaming with insert dynamic partition generate many small files

2014-02-03 Thread Bogala, Chandra Reddy
From: Chen Wang [mailto:chen.apache.s...@gmail.com] Sent: Monday, February 03, 2014 12:26 PM To: user@hive.apache.org Subject: Re: Hadoop streaming with insert dynamic partition generate many small files it seems that hive.exec.reducers.bytes.per.reducer is still not big enough: I added another 0,

Re: Hadoop streaming with insert dynamic partition generate many small files

2014-02-02 Thread Chen Wang
it seems that hive.exec.reducers.bytes.per.reducer is still not big enough: I added another 0, and now i only gets one file under each partition. On Sun, Feb 2, 2014 at 10:14 PM, Chen Wang wrote: > Hi, > I am using java reducer reading from a table, and then write to another > one: > > FROM (

Hadoop streaming with insert dynamic partition generate many small files

2014-02-02 Thread Chen Wang
Hi, I am using java reducer reading from a table, and then write to another one: FROM ( FROM ( SELECT column1,... FROM table1 WHERE ( partition > 6 and partition < 12 ) ) A MAP A.co