Re: [Spark 3.0] DataSourceV2 FileScan - Hive style partition pruning

Thanks Gengliang. Please let me know if I can help. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Spark 3.0] DataSourceV2 FileScan - Hive style partition pruning

Hi Guy, Thanks for reporting the issue. I am working on it and there will be a PR this week. Gengliang On Mon, Dec 30, 2019 at 6:41 AM Guy Khazma wrote: > Hi, > > It seems that hive style partition pruning is not working for file based > data sources such as Parquet and ORC. > This causes seri

[Spark 3.0] DataSourceV2 FileScan - Hive style partition pruning

Hi, It seems that hive style partition pruning is not working for file based data sources such as Parquet and ORC. This causes serious performance degradation for non hive tables. The reason for that is that the FileScan