Indexing in Hive

2014-04-18 Thread saquib khan
Hi, For large tables, its takes a lot of time to load the indexes in the index table. Is there any way we can reduce the index load time? CREATE TABLE SE_TX_SUMMARY (COUNTY string, BLOCKGROUPID string, GROUPING_ID int) PARTITIONED BY (EXPOSED_TIME int) row format delimited fields terminated by '

Indexing in Hive 0.12 on a partitioned and bucketed table

2014-03-20 Thread Sagar Mehta
Hi Guys, We have a Hive 0.12 ORC table that is partitioned on year, month, day, hour and is bucketed by one column. So far so good - We are seeing good speed up improvements as compared to non-ORC format. - Now we want to add an index on another commonly used column. My question was - Give

Problem with indexing in Hive

2012-07-26 Thread Ablimit Aji
I have written a custom index handler and wanted to test it. However hive is not using it. So I test with simple table (pokes (int foo, string bar)) which comes with hive distribution for testing purpose. Then I created a compact index and set the set hive.optimize.index.filter=true; However, upon

Re: Indexing in hive

2012-05-16 Thread Ranjith
.ql.index.compact.HiveCompactIndexInputFormat; > SELECT a, count(*) from t where j='and' group by a; > > Since the semantics of this usage make you specify the compact file, I have > not been able to figure out a way to use multiple indexes in the same query. > In this ca

Re: Indexing in hive

2012-05-16 Thread Mark Grover
In this case we are using the index on j, the column in the where clause. I hope you now understand why indexing in Hive is a work in progress:-) Good luck! Mark Mark Grover, Business Intelligence Analyst OANDA Corporation www: oanda.com www: fxtrade.com - Original Message ---

Re: Indexing in hive

2012-05-16 Thread Raghunath, Ranjith
: Zhaojun (Terry) Subject: Re: Indexing in hive Ransom, From this JIRA (https://issues.apache.org/jira/browse/HIVE-1644), it looks like automatic use of indexes using hive.optimize.index.filter was introduced in Hive 0.8. However, Ranjith seems to be using Hive 0.7.1 which doesn't support

Re: Indexing in hive

2012-05-16 Thread Mark Grover
che.org Cc: "Zhaojun (Terry)" Sent: Wednesday, May 16, 2012 8:32:55 PM Subject: RE: Indexing in hive “ hive.optimize.index.filter ” is the conf automatically use indexes If u set hive.optimize.index.groupby = true. It will set hive.optimize.index.filter =false. See your configurat

Re: Indexing in hive

2012-05-16 Thread Carl Steinbach
Hi Ranjith, Hive 0.7 supports the ability to build indexes, but the query compiler in 0.7 doesn't know how to optimize queries with these indexes. Hive 0.8 was the first release to include some support for optimizing query plans with indexes, and that only applies to GROUP BY and WHERE clauses und

RE: Indexing in hive

2012-05-16 Thread Hezhiqiang (Ransom)
“hive.optimize.index.filter” is the conf automatically use indexes If u set hive.optimize.index.groupby = true. It will set hive.optimize.index.filter=false. See your configurations. And you need to build index after create index. Best regards Ransom.

Indexing in hive

2012-05-16 Thread Raghunath, Ranjith
I am currently using hive 0.7.1 and creating indexes based on columns in the where clause. However, when I run the explain plan I do not see the index being leveraged. The syntax that I am using to build the index is as follows: CREATE INDEX x ON TABLE t(j) AS 'org.apache.hadoop.hive.ql.index.c