Re: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Abhay Bansal
I was able to find the property with some digging around and experimentation. Never knew that ppd had something to do with this property. On Thu, Apr 3, 2014 at 7:23 PM, Stephen Sprague wrote: > wow. good find. i hope these config settings are well documented and that > you didn't have to spend

Re: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Abhay Bansal
l [mailto:abhaybansal.1...@gmail.com] > *Sent:* Thursday, April 03, 2014 11:06 AM > *To:* user@hive.apache.org > *Subject:* Predicate pushdown optimisation not working for ORC > > > > I am new to Hive, apologise for asking such a basic question. > > > > Following exerci

RE: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Bogala, Chandra Reddy
...@gmail.com] Sent: Thursday, April 03, 2014 11:06 AM To: user@hive.apache.org Subject: Predicate pushdown optimisation not working for ORC I am new to Hive, apologise for asking such a basic question. Following exercise was done with hive .12 and hadoop 0.20.203 I created a ORC file form java

Re: Predicate pushdown optimisation not working for ORC

2014-04-03 Thread Stephen Sprague
wow. good find. i hope these config settings are well documented and that you didn't have to spend alot time searching for that. Interesting that the default isn't true for this one. On Wed, Apr 2, 2014 at 11:00 PM, Abhay Bansal wrote: > I was able to resolve the issue by setting "hive.optimize

Re: Predicate pushdown optimisation not working for ORC

2014-04-02 Thread Abhay Bansal
I was able to resolve the issue by setting "hive.optimize.index.filter" to true. In the hadoop logs syslog:2014-04-03 05:44:51,204 INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included column ids = 3,8,13 syslog:2014-04-03 05:44:51,204 INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat:

Re: Predicate pushdown optimisation not working for ORC

2014-04-02 Thread Stephen Boesch
HI Abhay, What is the DDL for your "test" table? 2014-04-02 22:36 GMT-07:00 Abhay Bansal : > I am new to Hive, apologise for asking such a basic question. > > Following exercise was done with hive .12 and hadoop 0.20.203 > > I created a ORC file form java, and pushed it into a table with the s

Predicate pushdown optimisation not working for ORC

2014-04-02 Thread Abhay Bansal
I am new to Hive, apologise for asking such a basic question. Following exercise was done with hive .12 and hadoop 0.20.203 I created a ORC file form java, and pushed it into a table with the same schema. I checked the conf property hive.optimize.ppdtrue which should ideally use the ppd optimisat