Abhishek Jadhav created SPARK-57743:
---------------------------------------
Summary: Addition of a partitions function in datalake V1 tables
Key: SPARK-57743
URL: https://issues.apache.org/jira/browse/SPARK-57743
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 4.2.0
Reporter: Abhishek Jadhav
*Why this is needed?*
In my work I've had requirements to create SQLs that run against AWS Glue Data
Catalog Tables where they need to get data from the latest partition only.
Something like :-
*SELECT * FROM db.table WHERE partition_col = (SELECT max(partition_col) FROM
db.table)*
But the issue with above approach is I've seen that pushdown predicate doesn't
always read the metadata only and reads the data itself which shouldn't happen.
In Athena, to solve this the query can be written as :-
*SELECT * FROM db.table WHERE partition_col = (SELECT max(partition_col) FROM
"db"."table$partitions")*
But there isn't a similar approach in Spark for V1 tables and the only way is
using SHOW PARTITIONS but it doesn't work as a subquery.
*Solution:*
The addition of a partitions function which returns the output of SHOW
PARTITIONS but can be used within a larger query.
{*}Suggested PR{*}: https://github.com/apache/spark/pull/56834
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
