Shekharrajak commented on code in PR #2948:
URL: https://github.com/apache/datafusion-comet/pull/2948#discussion_r2740040695
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -478,29 +478,35 @@ case class CometScanRule(session: SparkSession) extends
Rule[SparkPlan] with Com
false
}
- // Check for unsupported transform functions in residual expressions
- // iceberg-rust can only handle identity transforms in residuals; all
other transforms
- // (truncate, bucket, year, month, day, hour) must fall back to Spark
+ // Check for transform functions in residual expressions
+ // Non-identity transforms (truncate, bucket, year, month, day, hour)
in residuals
+ // are now supported - they skip row-group filtering and are handled
+ // post-scan by CometFilter.
+ // This is less optimal than row-group filtering but still allows
native execution.
val transformFunctionsSupported =
try {
IcebergReflection.findNonIdentityTransformInResiduals(metadata.tasks) match {
case Some(transformType) =>
- // Found unsupported transform
- fallbackReasons +=
- s"Iceberg transform function '$transformType' in residual
expression " +
- "is not yet supported by iceberg-rust. " +
- "Only identity transforms are supported."
- false
+ // Found non-identity transform - log info and continue with
native scan
+ // Row-group filtering will skip these predicates, but
post-scan
+ // filtering will apply
+ logInfo(
+ s"Iceberg residual contains transform '$transformType' - " +
+ "row-group filtering will skip this predicate, " +
+ "post-scan filtering by CometFilter will apply instead.")
+ true // Allow native execution
case None =>
- // No unsupported transforms found - safe to use native
execution
+ // No non-identity transforms - optimal row-group filtering
will apply
true
}
} catch {
case e: Exception =>
- // Reflection failure - cannot verify safety, must fall back
- fallbackReasons += "Iceberg reflection failure: Could not check
for " +
- s"transform functions in residuals: ${e.getMessage}"
- false
+ // Reflection failure - log warning but allow native execution
+ // The predicate conversion will handle unsupported cases
gracefully
+ logWarning(
+ s"Could not check for transform functions in residuals:
${e.getMessage}. " +
+ "Continuing with native scan.")
+ true
Review Comment:
if we could not find transform using
IcebergReflection.findNonIdentityTransformInResiduals then we will do with
native scan and get the correct result :
Native Scan -> CometFilter -> User Query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
