[
https://issues.apache.org/jira/browse/HIVE-12989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishesh Garg updated HIVE-12989:
--------------------------------
Description:
I'm using Hive 1.2.1 with ORC file format and doing some profiling for the data
read from the disk (after clearing file system caches) for 2 queries, the data
being loaded from the TPCDS benchmark. The queries are:
{code:sql}
Q1) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
2450816 and 2450817;
Q2) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
2450816 and 2452000;
{code}
I'm using flags *hive.vectorized.execution.enabled* = true and
*hive.optimize.index.filter* = true.
The output counts are about .1M and about 117M for Q1 and Q2 resp. On looking
at the hive logs, I see that about 11000 row-groups are included for Q1 and
33000 row-groups are included for Q2 after filtering using ORC file index.
Hence, I had expected vastly different sizes of data read. However, I see about
1GB of data read for both the queries. I suspect this has something to do with
the function _addRgFilteredStreamToRanges()_
was:
I'm using Hive 1.2.1 with ORC file format and doing some profiling for the data
read from the disk (after clearing file system caches) for 2 queries, the data
being loaded from the TPCDS benchmark. The queries are:
{code:sql}
Q1) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
2450816 and 2450817;
Q2) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
2450816 and 2452000;
{code}
I'm using flags *hive.vectorized.execution.enabled* = true and
*hive.optimize.index.filter* = true.
The output counts are about .1M and about 117M for Q1 and Q2 resp. On looking
at the hive logs, I see that about 11000 row-groups are included for Q1 and
33000 row-groups are included for Q2 after filtering using index. Hence, I had
expected vastly different sizes of data read. However, I see about 1GB of data
read for both the queries. I suspect this has something to do with the function
_addRgFilteredStreamToRanges()_
> Hive with ORC - Data read from disk for 2 queries same despite very different
> selectivities
> -------------------------------------------------------------------------------------------
>
> Key: HIVE-12989
> URL: https://issues.apache.org/jira/browse/HIVE-12989
> Project: Hive
> Issue Type: Bug
> Components: ORC
> Affects Versions: 1.2.1
> Environment: TPCDS 100 GB
> Reporter: Vishesh Garg
> Priority: Minor
> Attachments: HIVE LOGS.zip
>
>
> I'm using Hive 1.2.1 with ORC file format and doing some profiling for the
> data read from the disk (after clearing file system caches) for 2 queries,
> the data being loaded from the TPCDS benchmark. The queries are:
> {code:sql}
> Q1) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
> 2450816 and 2450817;
> Q2) select count(ss_item_sk) from store_sales where ss_sold_date_sk between
> 2450816 and 2452000;
> {code}
> I'm using flags *hive.vectorized.execution.enabled* = true and
> *hive.optimize.index.filter* = true.
> The output counts are about .1M and about 117M for Q1 and Q2 resp. On looking
> at the hive logs, I see that about 11000 row-groups are included for Q1 and
> 33000 row-groups are included for Q2 after filtering using ORC file index.
> Hence, I had expected vastly different sizes of data read. However, I see
> about 1GB of data read for both the queries. I suspect this has something to
> do with the function _addRgFilteredStreamToRanges()_
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
