lastbus opened a new pull request, #46073: URL: https://github.com/apache/spark/pull/46073
### What changes were proposed in this pull request? During the PruneFileSourcePartitions process, we can optimize by casting the dataType of the constant to match the dataType of the corresponding partition key. ### Why are the changes needed? Suppose we have a partitioned table `table_pt` with partition colum `dt` which is StringType and the table metadata is managed by Hive Metastore, if we filter partition by dt = '123', this filter can be pushed down to data source directly, but if the filter condition is number, e.g. dt = 123, Spark will not known which partition should be pushed down. Thus in the process of physical plan optimization, Spark will pull all of that table's partition meta data to client side, to decide which partition filter should be push down to the data source. This is poor of performance if the table has thousands of partitions and increasing the risk of hive metastore oom. In our production env, we encounter this problem. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? in our production env, this fix is OK ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
