Vishesh Garg created HIVE-12989:
-----------------------------------
Summary: Hive with ORC - Data read from disk for 2 queries same
despite very different selectivities
Key: HIVE-12989
URL: https://issues.apache.org/jira/browse/HIVE-12989
Project: Hive
Issue Type: Bug
Components: ORC
Affects Versions: 1.2.1
Environment: TPCDS 100 GB
Reporter: Vishesh Garg
Priority: Minor
I'm using Hive 1.2.1 with ORC file format and doing some profiling for the data
read from the disk (after clearing file system caches) for 2 queries, the data
being loaded from the TPCDS benchmark. The queries are:
{code:sql}
Q1) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
2450816 and 2450817;
Q2) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
2450816 and 2452000;
{code}
I'm using flags *hive.vectorized.execution.enabled* = true and
*hive.optimize.index.filter* = true.
The output counts are about .1M and about 117M for Q1 and Q2 resp. On looking
at the hive logs, I see that about 11000 row-groups are included for Q1 and
33000 row-groups are included for Q2 after filtering using index. Hence, I had
expected vastly different sizes of data read. However, I see about 1GB of data
read for both the queries. I suspect this has something to do with the function
_addRgFilteredStreamToRanges()_
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
