Re: Schema evolution for parquet table

2015-10-09 Thread Sergio Pena
Thanks Mohammad. I reviewed the HIVE-6784 patch and it has performance penalties because the way is implemented. I think we can do the promotion in a different way to avoid such penalties. If you take a look at ParquetStringInspector.java, this class gets a String value from different string writ

Re: Schema evolution for parquet table

2015-10-07 Thread Mohammad Islam
Hi Sergio, Thanks for your reply. I found one such effort : https://issues.apache.org/jira/browse/HIVE-6784 I consider to try it differently. Similar stuffs in ORC: https://issues.apache.org/jira/browse/HIVE-10591 Regards, Mohammad On Wednesday, October 7, 2015 8:58 AM, Sergio Pena wrote

Re: Schema evolution for parquet table

2015-10-07 Thread Sergio Pena
Hi Mohammad, Currently, Hive + Parquet does not support auto casting for wider types. That is be a very good idea to implement in Hive. I'll investigate the hive + parquet code, and see if it is something we can add in a future release. - Sergio On Tue, Oct 6, 2015 at 7:23 PM, Mohammad Islam w

Re: Schema evolution for parquet table

2015-10-06 Thread Mohammad Islam
Any hive+parquet user/dev to address this? Regards, Mohammad On Monday, October 5, 2015 3:41 PM, Mohammad Islam wrote: Hi, Does the parquet table support auto casting to wider data types? For example, If I have a parquet table where some parquet data files which have "int" as data type a

Schema evolution for parquet table

2015-10-05 Thread Mohammad Islam
Hi,Does the parquet table support auto casting to wider data types? For example, If I have a parquet table where some parquet data files which have "int"  as data type and other files have "long" data type for the same field. The table schema has type "bigint" for the same field.Does hive can rea