Re: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Abhay Bansal
I was able to find the property with some digging around and experimentation. Never knew that ppd had something to do with this property. On Thu, Apr 3, 2014 at 7:23 PM, Stephen Sprague wrote: > wow. good find. i hope these config settings are well documented and that > you didn't have to spend

Re: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Abhay Bansal
It is not essentially the client side. The file can be generated by a running MR job or any process for that matter which can then feed data to the Hadoop cluster to run hive queries. -Abhay On Thu, Apr 3, 2014 at 7:40 PM, Bogala, Chandra Reddy wrote: > I thought ORC file can be generated only

RE: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Bogala, Chandra Reddy
I thought ORC file can be generated only by running hive query on staging table and inserting into ORC table. If there is option to generate ORC file at client side by using java code then can you share that code or links related to that? Thanks, Chandra From: Abhay Bansal [mailto:abhaybansal.1.

Re: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Stephen Sprague
wow. good find. i hope these config settings are well documented and that you didn't have to spend alot time searching for that. Interesting that the default isn't true for this one. On Wed, Apr 2, 2014 at 11:00 PM, Abhay Bansal wrote: > I was able to resolve the issue by setting "hive.optimize

Re: Predicate pushdown optimisation not working for ORC

2014-04-02 Thread Abhay Bansal
I was able to resolve the issue by setting "hive.optimize.index.filter" to true. In the hadoop logs syslog:2014-04-03 05:44:51,204 INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included column ids = 3,8,13 syslog:2014-04-03 05:44:51,204 INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat:

Re: Predicate pushdown optimisation not working for ORC

2014-04-02 Thread Stephen Boesch
HI Abhay, What is the DDL for your "test" table? 2014-04-02 22:36 GMT-07:00 Abhay Bansal : > I am new to Hive, apologise for asking such a basic question. > > Following exercise was done with hive .12 and hadoop 0.20.203 > > I created a ORC file form java, and pushed it into a table with the s