alley
Reply-To: "user@hive.apache.org"
Date: Monday, September 17, 2018 at 2:00 PM
To: "user@hive.apache.org"
Subject: Re: Hive Compaction OOM
Ok, if you are against the wall, I'd suggest looking at the CompactorMR class,
which is the class that the Metastore uses to lau
of memory.
>
>
> Thanks
>
> Shawn Weeks
>
>
> --
> *From:* Owen O'Malley
> *Sent:* Monday, September 17, 2018 3:37:09 PM
> *To:* user@hive.apache.org
> *Subject:* Re: Hive Compaction OOM
>
> How many files is it trying to merge at once? By far the ea
Hive Compaction OOM
I've already tried giving the compactor 256+ gigabytes of memory. All that
changes is how long for it run out of memory.
Thanks
Shawn Weeks
From: Owen O'Malley
Sent: Monday, September 17, 2018 3:37:09 PM
To: user@hive.apache.org
Su
ng to work. See HIVE-18613.
Thanks
Shawn Weeks
From: Shawn Weeks mailto:swe...@weeksconsulting.us>>
Sent: Monday, September 17, 2018 12:28:25 PM
To: user@hive.apache.org<mailto:user@hive.apache.org>
Subject: Re: Hive Comp
t;
>
> Thanks
>
> Shawn Weeks
> --
> *From:* Shawn Weeks
> *Sent:* Monday, September 17, 2018 12:28:25 PM
> *To:* user@hive.apache.org
> *Subject:* Re: Hive Compaction OOM
>
>
> 2018-09-17 11:20:26,404 FATAL [main] org.apache.hadoop.mapred.YarnChild:
&g
ubject: Re: Hive Compaction OOM
2018-09-17 11:20:26,404 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error
running child : java.lang.OutOfMemoryError: Java heap space
at
com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:864)
: Owen O'Malley
Sent: Monday, September 17, 2018 11:28:43 AM
To: user@hive.apache.org
Subject: Re: Hive Compaction OOM
Shawn,
Can you provide the stack trace that you get with the OOM?
Thanks,
Owen
On Mon, Sep 17, 2018 at 9:27 AM Prasanth Jayachandran
mailto:pjayachand...@hortonwork
Shawn,
Can you provide the stack trace that you get with the OOM?
Thanks,
Owen
On Mon, Sep 17, 2018 at 9:27 AM Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:
> Hi Shawn
>
> You might be running into issues related to huge protobuf objects from
> huge string columns. Without
Hi Shawn
You might be running into issues related to huge protobuf objects from huge
string columns. Without
https://issues.apache.org/jira/plugins/servlet/mobile#issue/ORC-203 there isn’t
an option other than providing sufficiently large memory. If you can reload the
data with binary type tha
Let me start off by saying I've backed myself into a corner and would rather
not reprocess the data if possible. I have a Hive Transactional table in Hive
1.2.1 H that was loaded via NiFi Hive Streaming with a fairly large String
column containing XML Documents. Awful I know and I'm working on c
10 matches
Mail list logo
