Re: Hive query taking too much time

2011-12-08 Thread Wojciech Langiewicz
e- From: Wojciech Langiewicz [mailto:wlangiew...@gmail.com] Sent: Wednesday, December 07, 2011 8:15 PM To: user@hive.apache.org Subject: Re: Hive query taking too much time Hi, In this case it's much easier and faster to merge all files using this command: cat *.csv> output.csv hive

Re: Hive query taking too much time

2011-12-08 Thread Aniket Mokashi
ame bytes. What do you suggest? > > Kind Regards, > Keshav C Savant > > > -Original Message- > From: Wojciech Langiewicz [mailto:wlangiew...@gmail.com] > Sent: Wednesday, December 07, 2011 8:15 PM > To: user@hive.apache.org > Subject: Re: Hive query taking too much ti

RE: Hive query taking too much time

2011-12-07 Thread Savant, Keshav
e the 2624 files >>> into a smaller number like 24 or 48. That should speed up the mapper >>> phase significantly. >>> >>> ** ** >>> >>> *From:* Savant, Keshav [mailto:keshav.c.sav...@fisglobal.com] >>> *Sent:* Tuesday, December 06

Re: Hive query taking too much time

2011-12-07 Thread Wojciech Langiewicz
011 6:01 AM *To:* user@hive.apache.org *Subject:* Hive query taking too much time ** ** Hi All, ** ** My setup is hadoop-0.20.203.0 hive-0.7.1 ** ** I am having a total of 5 node cluster: 4 data nodes, 1 namenode (it is also acting as secondary name node). On namenode

RE: Hive query taking too much time

2011-12-07 Thread Savant, Keshav
14,271,688 Thanks a lot for your help. Kind Regards, Keshav C Savant From: Paul Mackles [mailto:pmack...@adobe.com] Sent: Tuesday, December 06, 2011 8:14 PM To: user@hive.apache.org Subject: RE: Hive query taking too much time How much time is it spending in the map/reduce phases

Re: Hive query taking too much time

2011-12-06 Thread Ayon Sinha
t my Blog for answers to commonly asked questions. From: Vikas Srivastava To: user@hive.apache.org Sent: Tuesday, December 6, 2011 10:00 PM Subject: Re: Hive query taking too much time hey if u having the same col of  all the files then you can easily merg

Re: Hive query taking too much time

2011-12-06 Thread Vikas Srivastava
the mapper phase significantly. >> >> ** ** >> >> *From:* Savant, Keshav [mailto:keshav.c.sav...@fisglobal.com] >> *Sent:* Tuesday, December 06, 2011 6:01 AM >> *To:* user@hive.apache.org >> *Subject:* Hive query taking too much time >> >> ** ** &g

Re: Hive query taking too much time

2011-12-06 Thread Mohit Gupta
, 2011 6:01 AM > *To:* user@hive.apache.org > *Subject:* Hive query taking too much time > > ** ** > > Hi All, > > ** ** > > My setup is > > hadoop-0.20.203.0 > > hive-0.7.1 > > ** ** > > I am having a total of 5 node cluster: 4 d

RE: Hive query taking too much time

2011-12-06 Thread Paul Mackles
. From: Savant, Keshav [mailto:keshav.c.sav...@fisglobal.com] Sent: Tuesday, December 06, 2011 6:01 AM To: user@hive.apache.org Subject: Hive query taking too much time Hi All, My setup is hadoop-0.20.203.0 hive-0.7.1 I am having a total of 5 node cluster: 4 data nodes, 1 namenode (it is also

Re: Hive query taking too much time

2011-12-06 Thread Wojciech Langiewicz
Hi, In your case total file size isn't main factor that reduces performance, number of files is. To test this try merging those over 2000 files into one (or few) big, then upload it to HDFS and test hive performance (it should be definitely higher). It this works you should think about mergin

Hive query taking too much time

2011-12-06 Thread Savant, Keshav
Hi All, My setup is hadoop-0.20.203.0 hive-0.7.1 I am having a total of 5 node cluster: 4 data nodes, 1 namenode (it is also acting as secondary name node). On namenode I have setup hive with HiveDerbyServerMode to support multiple hive server connection. I have inserted plain text C