Thanx for the response guys. I tried a few different compression sizes and
all of them did not work.
I guess our use-case is not a good candidate for orc or parquet (which I
tried too and it failed)
We will use some other file type.
Thanx again.
On Fri, May 16, 2014 at 2:26 PM, Prasanth Jayachan
With Hive 0.13 the ORC memory issue is mitigated because of this optimization
https://issues.apache.org/jira/browse/HIVE-6455. This optimization is enabled
by default.
But having 3283 columns is still huge. So I would still recommend reducing the
default compression (256KB) buffer size to a lowe
When I created the table, I had to reduce the orc.compress.size quite a bit
to make my table with many columns work. This was on Hive 0.12 (I thought
it was supposed to be fixed on Hive 0.13, but 3k+ columns is huge) The
default of orc.compress size is quite a bit larger ( think in the 268k
range)
Sorry for the double post. I did not show up for a while and then I could
not get to the archives page, so I thought I'd needed to resend.
On Fri, May 16, 2014 at 12:54 AM, Premal Shah wrote:
> I have a table in hive stored as text file with 3283 columns. All columns
> are of string data type.
>
I have a table in hive stored as text file with 3283 columns. All columns
are of string data type.
I'm trying to convert that table into an orc file table using this command
*create table orc_table stored as orc as select * from text_table;*
This is the setting under mapred-site.xml
mapred.
I have a table in hive stored as text file with 3283 columns. All columns
are of string data type.
I'm trying to convert that table into an orc file table using this command
*create table orc_table stored as orc as select * from text_table;*
This is the setting under mapred-site.xml
mapred.
