yihua commented on code in PR #20029:
URL: https://github.com/apache/hudi/pull/20029#discussion_r4076537718
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestNestedSchemaPruningOptimization.scala:
##########
@@ -299,6 +299,39 @@ class TestNestedSchemaPruningOptimization extends
HoodieSparkSqlTestBase {
}
}
+ test("Nested struct predicate pushdown") {
+ withTempDir { tmp =>
+ Seq("cow", "mor").foreach { tableType =>
+ val tableName = generateTableName
+ val tablePath = s"${tmp.getCanonicalPath}/$tableName"
+
+ createTableWithNestedStructSchema(tableType, tableName, tablePath)
+
+ val selectDF = spark.sql(s"SELECT id FROM $tableName WHERE item.name =
'a1' AND item.price > 5")
+
+ val fileScan = findFileScan(selectDF.queryExecution.executedPlan)
+ val planString = fileScan.toString()
+
+ assertTrue(fileScan.dataFilters.nonEmpty,
+ s"[$tableType] dataFilters should contain nested predicates but was
empty")
+ assertFalse(planString.contains("PushedFilters: []"),
+ s"[$tableType] PushedFilters should not be empty — nested predicate
pushdown is not working.\n" +
+ s"Verify that shortName() returns a lowercase value present in " +
+
s"spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources.\nPlan:\n$planString")
+ assertTrue(planString.contains("PushedFilters: ["),
Review Comment:
I think this test passes on master without the fix: Spark infers
`IsNotNull(item)` from `item.name = 'a1'` (GetStructField is null-intolerant),
and that top-level filter is pushed regardless of the allowlist, so
`PushedFilters` is never `[]` here. Could you assert on the nested predicates
themselves instead, e.g. `fileScan.metadata("PushedFilters")` contains
`EqualTo(item.name,a1)` and `GreaterThan(item.price,5)` (and, ideally, a
negative leg with the extension's conf change undone showing they are absent)?
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieSparkSessionExtension.scala:
##########
@@ -45,6 +49,8 @@ class HoodieSparkSessionExtension extends
(SparkSessionExtensions => Unit)
extensions.injectOptimizerRule(ruleBuilder(_))
}
+ extensions.injectOptimizerRule(HoodieNestedPredicatePushdownConfig(_))
Review Comment:
This only helps sessions that load HoodieSparkSessionExtension; a plain
`spark.read.format("hudi")` or Spark Connect/Thrift session without
`spark.sql.extensions` still ends up with `hoodiefilegroup` against the default
`parquet,orc` allowlist. Have you considered having `shortName()` return the
base file format (`parquet` / `orc` via `tableConfig.getBaseFileFormat`)
instead? That restores the 0.x / 1.0.x behaviour on every read path without
touching session state, and ORC base files get the same treatment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
