hudi-agent commented on code in PR #20029:
URL: https://github.com/apache/hudi/pull/20029#discussion_r4076089508
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieSparkSessionExtension.scala:
##########
@@ -57,3 +63,26 @@ class HoodieSparkSessionExtension extends
(SparkSessionExtensions => Unit)
sparkAdapter.injectPlannerStrategies(extensions)
}
}
+
+/**
+ * Ensures that Spark's nested predicate pushdown allowlist includes
"hoodiefilegroup"
+ * so that nested struct field predicates (e.g., msg.field = 'value') are
pushed down
+ * to the Parquet reader when using HoodieFileGroupReaderBasedFileFormat.
+ */
+case class HoodieNestedPredicatePushdownConfig(spark: SparkSession) extends
Rule[LogicalPlan] with Logging {
+ @volatile private var applied = false
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ if (!applied) {
+ val key =
"spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources"
+ val current = spark.conf.get(key, "parquet,orc")
+ if (!current.toLowerCase.contains("hoodiefilegroup")) {
Review Comment:
🤖 If a user has explicitly set this config to empty to turn nested pushdown
off (a known escape hatch for Parquet nested-filter issues), this appends and
silently re-enables it for Hudi — is that intended? And once `applied` is set,
a later `RESET`/`SET` by the user is never reconciled. Minor:
`spark.conf.get(key)` returns the registered entry's default, so hard-coding
`"parquet,orc"` isn't needed.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieSparkSessionExtension.scala:
##########
@@ -57,3 +63,26 @@ class HoodieSparkSessionExtension extends
(SparkSessionExtensions => Unit)
sparkAdapter.injectPlannerStrategies(extensions)
}
}
+
+/**
+ * Ensures that Spark's nested predicate pushdown allowlist includes
"hoodiefilegroup"
+ * so that nested struct field predicates (e.g., msg.field = 'value') are
pushed down
+ * to the Parquet reader when using HoodieFileGroupReaderBasedFileFormat.
+ */
+case class HoodieNestedPredicatePushdownConfig(spark: SparkSession) extends
Rule[LogicalPlan] with Logging {
+ @volatile private var applied = false
+
+ override def apply(plan: LogicalPlan): LogicalPlan = {
Review Comment:
🤖 nit: this mutates a session-wide, comma-separated SQLConf string from
inside an optimizer rule, using an `@volatile applied` flag to only run once —
that's a fairly surprising place to do config setup. Might be worth moving this
to `HoodieSparkSessionExtension.apply` itself (setting the conf once at
extension-injection time) rather than as a `Rule[LogicalPlan]` that runs on
every plan until it flips a flag.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieSparkSessionExtension.scala:
##########
@@ -57,3 +63,26 @@ class HoodieSparkSessionExtension extends
(SparkSessionExtensions => Unit)
sparkAdapter.injectPlannerStrategies(extensions)
}
}
+
+/**
+ * Ensures that Spark's nested predicate pushdown allowlist includes
"hoodiefilegroup"
+ * so that nested struct field predicates (e.g., msg.field = 'value') are
pushed down
+ * to the Parquet reader when using HoodieFileGroupReaderBasedFileFormat.
+ */
+case class HoodieNestedPredicatePushdownConfig(spark: SparkSession) extends
Rule[LogicalPlan] with Logging {
Review Comment:
🤖 This only takes effect when `HoodieSparkSessionExtension` is configured,
so sessions using just `spark.read.format("hudi")` (no `spark.sql.extensions`)
still get `PushedFilters: []`. Since the class extends `ParquetFileFormat`
(inherited `shortName` = `parquet`, already on Spark's default allowlist) and
nothing in the repo references the name, would simply removing the
`shortName()` override work — that was the pre-#13632 behavior and needs no
config mutation? Alternatively, `DefaultSource`'s existing
`SparkSession.getActiveSession.foreach` block runs for both SQL and DataFrame
entry points if you'd rather keep a distinct name.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
