Re: Hive Error on medium sized dataset

2011-01-26 Thread hadoop n00b
We typically get this error while running complex queries on our 4-node setup when the child JVM runs out of heap size. Would be interested in what the experts have to say about this error. On Thu, Jan 27, 2011 at 7:27 AM, Ajo Fod wrote: > Any chance you can convert the data to a tab separated t

Re: Hive Error on medium sized dataset

2011-01-26 Thread Ajo Fod
Any chance you can convert the data to a tab separated text file and try the same query? It may not be the SerDe, but it may be good to isolate that away as a potential source of the problem. -Ajo. On Wed, Jan 26, 2011 at 5:47 PM, Christopher, Pat < patrick.christop...@hp.com> wrote: > Hi, > >

Hive Error on medium sized dataset

2011-01-26 Thread Christopher, Pat
Hi, I'm attempting to load a small to medium sized log file, ~250MB, and produce some basic reports from it, counts etc. Nothing fancy. However, whenever I try and read the entire dataset, ~330k rows, I get the following error: FAILED: Execution Error, return code 2 from org.apache.hadoop.h

java.lang.IllegalStateException: Shutdown in progress

2011-01-26 Thread Tali K
Hi All, My hive jobs all finished successfully, output file created, hadoop job -list, displays no running jobs. I run hive -e ''> myfile. All Map/reduces are done and than I see 'OK' forever, until I kill this job with Ctrl C. Than I get an Exception:java.lang.IllegalStateExcept

Re: Hive Concurrency Model - does it work?

2011-01-26 Thread John Sichi
On Jan 26, 2011, at 10:52 AM, Jay Ramadorai wrote: > - Create views on temporary tables named by day. Have jobs go against the > views. When we are ready to rename, basically replace the view, pointing it > now to the new table of today. The key question here is: is the View metadata > consulted

Re: Hive Concurrency Model - does it work?

2011-01-26 Thread Namit Jain
The patch below has been committed. https://issues.apache.org/jira/browse/HIVE-1865 was a follow-up patch which should help concurrency. I have not tried backporting the patch on hive 0.5 or hive0.6, but I don’t think it will work, since the code has changed significantly, and a number of bug f

Hive Concurrency Model - does it work?

2011-01-26 Thread Jay Ramadorai
https://issues.apache.org/jira/browse/HIVE-1293 : Is this JIRA truly fixed and included in 0.7.0? If so, can the patch be applied separately on top of 0.5.0 or 0.6.0? Are there instructions somewhere for how to enable/integrate Zookeeper with Hive for this patch to work? The JIRA comments indica

RE: Distinct in hive

2011-01-26 Thread Guy Doulberg
Thanks That was it From: Namit Jain [mailto:nj...@fb.com] Sent: Tuesday, January 25, 2011 7:04 PM To: user@hive.apache.org Subject: Re: Distinct in hive Is there skew in data ? You may want to set the parameter: hive.groupby.skewindata: to true. Thanks, -namit From: Guy Doulberg mailto:guy.d

Re: Is there any method can not move array the raw data.

2011-01-26 Thread Ajo Fod
Have you tried using external tables? BTW, hive tables can be defined as text tables, so you can run mapreduce on them too. Just locate the tables under directory: /user/hive/warehouse/ Cheers, -Ajo. 2011/1/26 母延年YNM > When I use load data into table like this , > > LOAD DATA INPATH '/user/myn

Is there any method can not move array the raw data.

2011-01-26 Thread 母延年YNM
When I use load data into table like this , LOAD DATA INPATH '/user/myname/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2008-08-15'); The raw data /user/myname/kv2.txt was rename into hive`s store direcrory Is there any method can not move array the raw data. Why not make a soft link?