are you running out of open file handles ? you should look into that because
you are running everything on one node. you should look at your
namenode/datanode logs to make sure that's not the case.
the error sounds like a BlockException on HDFS and the hive copyTask is just
moving the file from yo
yes, i have a lot of small files. this is because i wanted to process
hourly instead of daily.
i will be checking into whether this is the case, i now am re-running
the process, and I see
332 files and directories, 231 blocks = 563 total. Heap Size is 119.88
MB / 910.25 MB (13%)
Configured Capaci
If you're running with the defaults I think its around 20gb. If you're
processing a couple hundred MBs you could easily hit this limit between desired
outputs and any intermediate files created. HDFS allocates the available space
in blocks so if you have a lot of small files, you'll run out of
Hello,
what is the metastore_db for and why it does not stay just under $HIVE_HOME.
any directory i run, this metastore_db directory gets created,
best,
c.b.
On Sat, Feb 12, 2011 at 2:51 AM, Ajo Fod wrote:
> I'd be surprised if this were not enough.
>
> -Ajo
>
> On Fri, Feb 11, 2011 at 2:51 P
What else would be left? There's still any site configs but otherwise it's a
cleaned system, right?
c.b.: you can specify your metastore_db location (so you only have one db) for
your hive instance as a whole in hive-site.xml with this:
javax.jdo.option.ConnectionURL
jdbc:derby:PATH_FOR_
but is there a ridiculously low default for hdfs space limits? I
looked everywhere in the configuration files, but could not find
anything that limits the size of hdfs
i think i am running on a 150GB hard drive, and the data I am
processing is in amounts of couple of hundred of megabytes at max.
I'd be surprised if this were not enough.
-Ajo
On Fri, Feb 11, 2011 at 2:51 PM, Cam Bazz wrote:
> Hello,
>
> I sometimes need to delete everything in hdfs and recreate the tables.
>
> The question is: how do I clear everything in the hdfs and hive?
>
> I delete everything in /tmp, hadoop/logs
Is your hdfs hitting its space limits?
Pat
-Original Message-
From: Cam Bazz [mailto:camb...@gmail.com]
Sent: Friday, February 11, 2011 4:38 PM
To: user@hive.apache.org
Subject: error out of all sudden
Hello,
I set up my one node pseudo distributed system, left with a cronjob,
copying
Hello,
I set up my one node pseudo distributed system, left with a cronjob,
copying data from a remote server and loading them to hadoop, and
doing some calculations per hour.
It stopped working today, giving me this error. I deleted everything,
and made it reprocess from beginning, and i still g
We use hive through the thrift api too. I have seen the above error when
someone accidentally connects to the thrift server port through multiple
instances of our hive-wrapper code. We open a bunch of hive thrift ports to
achieve the automation of running multiple hive queries in parallel i.e.
cont
Hello,
I sometimes need to delete everything in hdfs and recreate the tables.
The question is: how do I clear everything in the hdfs and hive?
I delete everything in /tmp, hadoop/logs and any metastore_db i can find.
then hadoop namenode -format
-
is this enough?
best regards,
c.b.
Are you using hive 0.6? ... may be fixed in the latest version.
Also I wonder why these thrift libraries are being used ... is this normal
hive operation, or can you do something to avoid using thrift?
-Ajo
On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi wrote:
>
> Hi all,
>
> I am loading data
Hello,
The query below produces a blank file, when no results are found:
insert overwrite table selection_hourly_clicks partition (date_hour = PARTNAME)
select sel_sid, count(*) cc from (
select split(parse_url(iv.referrer_url,'PATH'), '_')[1] sel_sid from
item_raw iv where iv.date_hour='PARTNAME
13 matches
Mail list logo
