[
https://issues.apache.org/jira/browse/SPARK-57743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092366#comment-18092366
]
Nikita Awasthi commented on SPARK-57743:
----------------------------------------
User '0xAJX' has created a pull request for this issue:
https://github.com/apache/spark/pull/56834
> Addition of a partitions function in datalake V1 tables
> -------------------------------------------------------
>
> Key: SPARK-57743
> URL: https://issues.apache.org/jira/browse/SPARK-57743
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Abhishek Jadhav
> Priority: Minor
>
> *Why this is needed?*
> In my work I've had requirements to create SQLs that run against AWS Glue
> Data Catalog Tables where they need to get data from the latest partition
> only.
> Something like :-
> *SELECT * FROM db.table WHERE partition_col = (SELECT max(partition_col) FROM
> db.table)*
> But the issue with above approach is I've seen that pushdown predicate
> doesn't always read the metadata only and reads the data itself which
> shouldn't happen.
> In Athena, to solve this the query can be written as :-
> *SELECT * FROM db.table WHERE partition_col = (SELECT max(partition_col) FROM
> "db"."table$partitions")*
> But there isn't a similar approach in Spark for V1 tables and the only way is
> using SHOW PARTITIONS but it doesn't work as a subquery.
>
> *Solution:*
> The addition of a partitions function which returns the output of SHOW
> PARTITIONS but can be used within a larger query.
> {*}Suggested PR{*}: https://github.com/apache/spark/pull/56834
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
