hudi-agent commented on code in PR #20201:
URL: https://github.com/apache/hudi/pull/20201#discussion_r4235329911
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantShreddingMixedLayouts.scala:
##########
@@ -607,6 +607,17 @@ class TestVariantShreddingMixedLayouts extends
HoodieSparkSqlTestBase with Varia
assert(scanReadSchema(sql) == s"struct<id:int,v:$readV>", s"[$leg]
ReadSchema of the scan of: $sql")
}
+ // The scan reports the base file format as its short name (#20029), so
Spark pushes nested
+ // predicates: with the rule on, a variant_get predicate arrives as a
filter on the projection
+ // struct's ordinal-named member, a name no file carries as a parquet
leaf. ParquetFilters drops
+ // it per file - it neither skips a row group nor lands on another column
- and the checkAnswer
+ // counts in each leg, identical on both arms, are what pin that. This
pins that the filter
+ // really was pushed, so those counts cannot go vacuous if the pushdown is
lost again.
+ def assertPushedFilters(sql: String, pushIntoScan: String, on: String,
off: String, leg: String): Unit = {
Review Comment:
🤖 nit: could `assertPushedFilters` take a `Boolean` (or reuse the
already-parsed flag) instead of the `pushIntoScan` String plus `.toBoolean`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
