[
https://issues.apache.org/jira/browse/SPARK-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Allman updated SPARK-20331:
-----------------------------------
Description:
Spark 2.1 introduced scalable support for Hive tables with huge numbers of
partitions. Key to leveraging this support is the ability to prune unnecessary
table partitions to answer queries. Spark supports a subset of the class of
partition pruning predicates that the Hive metastore supports. If a user writes
a query with a partition pruning predicate that is *not* supported by Spark,
Spark falls back to loading all partitions and pruning client-side. We want to
broaden Spark's current partition pruning predicate pushdown capabilities.
One of the key missing capabilities is support for disjunctions. For example,
for a table partitioned by date, specifying with a predicate like
{code}date = 20161011 or date = 20161014{code}
will result in Spark fetching all partitions. For a table partitioned by date
and hour, querying a range of hours across dates can be quite difficult to
accomplish without fetching all partition metadata.
The current partition pruning support supports only comparisons against
literals. We can expand that to foldable expressions by evaluating them at
planning time.
We can also implement support for the "IN" comparison by expanding it to a
sequence of "OR"s.
This ticket covers those enhancements.
was:
Spark 2.1 introduced scalable support for Hive tables with huge numbers of
partitions. Key to leveraging this support is the ability to prune unnecessary
table partitions to answer queries. Spark supports a subset of the class of
partition pruning predicates that the Hive metastore supports. If a user writes
a query with a partition pruning predicate that is *not* supported by Spark,
Spark falls back to loading all partitions and pruning client-side. We want to
broaden Spark's current partition pruning predicate pushdown capabilities.
One of the key missing capabilities is support for disjunctions. For example,
for a table partitioned by date, specifying with a predicate like
{code}date = 20161011 or date = 20161014{code}
will result in Spark fetching all partitions. For a table partitioned by date
and hour, querying a range of hours across dates can be quite difficult to
accomplish without fetching all partition metadata.
The current partition pruning support supports only comparisons against
literals. We can expand that to foldable expressions by evaluating them at
planning time.
We can also implement support for the "IN" comparison by expanding it to a
sequence of "OR"s.
This ticket covers those enhancements.
A PR will follow.
> Broaden support for Hive partition pruning predicate pushdown
> -------------------------------------------------------------
>
> Key: SPARK-20331
> URL: https://issues.apache.org/jira/browse/SPARK-20331
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Michael Allman
>
> Spark 2.1 introduced scalable support for Hive tables with huge numbers of
> partitions. Key to leveraging this support is the ability to prune
> unnecessary table partitions to answer queries. Spark supports a subset of
> the class of partition pruning predicates that the Hive metastore supports.
> If a user writes a query with a partition pruning predicate that is *not*
> supported by Spark, Spark falls back to loading all partitions and pruning
> client-side. We want to broaden Spark's current partition pruning predicate
> pushdown capabilities.
> One of the key missing capabilities is support for disjunctions. For example,
> for a table partitioned by date, specifying with a predicate like
> {code}date = 20161011 or date = 20161014{code}
> will result in Spark fetching all partitions. For a table partitioned by date
> and hour, querying a range of hours across dates can be quite difficult to
> accomplish without fetching all partition metadata.
> The current partition pruning support supports only comparisons against
> literals. We can expand that to foldable expressions by evaluating them at
> planning time.
> We can also implement support for the "IN" comparison by expanding it to a
> sequence of "OR"s.
> This ticket covers those enhancements.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
