ajsquared commented on code in PR #7343:
URL: https://github.com/apache/datafusion-comet/pull/7343#discussion_r4433621960
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala:
##########
@@ -327,6 +333,56 @@ object CometNativeScan extends
CometOperatorSerde[CometScanExec] with CometTypeS
}
}
+ /**
+ * Apply Spark's Parquet pushdown settings before serializing scan
predicates. Unsupported
+ * conjuncts may be dropped, including inside OR, but never below NOT:
weakening a negated
+ * predicate could prune rows Spark would read and hide an error in the
enclosing filter.
+ *
+ * Public because execution-time scalar-subquery filters are serialized from
Spark's package.
+ */
+ def filterForParquetPushdown(filter: Expression, conf: SQLConf):
Option[Expression] = {
+ val pushDownDate = conf.parquetFilterPushDownDate
+ val pushDownTimestamp = conf.parquetFilterPushDownTimestamp
+ val pushDownDecimal = conf.parquetFilterPushDownDecimal
+ val pushDownStringPredicate = conf.parquetFilterPushDownStringPredicate
+ val inFilterThreshold = conf.parquetFilterPushDownInFilterThreshold
+
+ def usesDisabledSetting(expression: Expression): Boolean =
expression.exists {
+ case _: StartsWith | _: EndsWith | _: Contains =>
!pushDownStringPredicate
+ case e =>
+ e.dataType match {
+ case DateType => !pushDownDate
+ case TimestampType | TimestampNTZType => !pushDownTimestamp
+ case _: DecimalType => !pushDownDecimal
+ case _ => false
+ }
+ }
+
+ def select(expression: Expression, canPartialPushDown: Boolean):
Option[Expression] =
+ expression match {
+ case And(left, right) =>
+ (select(left, canPartialPushDown), select(right,
canPartialPushDown)) match {
+ case (Some(l), Some(r)) => Some(And(l, r))
+ case (l, r) if canPartialPushDown => l.orElse(r)
Review Comment:
[P2] With global Parquet pushdown enabled and
`spark.sql.parquet.filterPushdown.date=false`, `SELECT count(*) FROM t WHERE
(unix_date(make_date(y,1,1)) > 0 AND a = 1) OR b = 1` can return `0` where
Spark raises `ARITHMETIC_OVERFLOW`. Use one row group where `a` and `b` are
always zero and one row has `y = 6000000`. Spark evaluates the date first. The
`l.orElse(r)` branch drops the date comparison, leaving `a = 1 OR b = 1` for
Parquet statistics to evaluate. That predicate eliminates the entire group even
with row filtering off. Exclude each original scan predicate containing
`MakeDate` before partially rewriting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
