garyli1019 opened a new pull request #1643: URL: https://github.com/apache/incubator-hudi/pull/1643
## What is the purpose of the pull request JIRA: https://issues.apache.org/jira/browse/HUDI-110 Add an example to support the Spark auto partition extractor to improve read performance. With the existing way how we query the ReadOptimized table through Spark, we can take advantage of https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery directly. This feature only support folder structure like `/partitionKey=partitionValue` and `partitionValue` could be `String` or `Int` This feature only applies to the Spark internally supported file format. If we use a custom relation, we will need a custom partition parser to extract partition from the path. This was handled by [DataSource](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L147) internally. We need to add this example to the doc as well. ## Brief change log - Add test to demo Spark Partition Discovery. ## Verify this pull request This change added tests and can be verified as follows: - *Added unit test testSparkAutoPartition* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
