hudi-agent commented on code in PR #20226:
URL: https://github.com/apache/hudi/pull/20226#discussion_r4235327920
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/DataSkippingUtils.scala:
##########
@@ -321,16 +321,17 @@ object DataSkippingUtils extends Logging {
})
// Filter "colA like 'xxx%'"
- // Translates to "colA_minValue <= xxx AND xxx <= colA_maxValue" for
index lookup
+ // Translates to "(colA_minValue <= xxx AND xxx <= colA_maxValue) OR
colA_minValue like 'xxx%'" for index lookup
//
- // NOTE: Since a) this operator matches strings by prefix and b) given
that this column is going to be ordered
- // lexicographically, we essentially need to check that provided
literal falls w/in min/max bounds of the
- // given column
+ // NOTE: Strings starting w/ the prefix form a contiguous range of the
binary (byte-wise) ordering, which starts
+ // at the prefix itself but also holds strings sorting after it
(for ex, "xxx1" > "xxx"). Hence the file may
+ // hold such strings if either the prefix falls w/in its min/max
bounds, or its min value starts w/ it
case StartsWith(sourceExpr @ AllowedTransformationExpression(attrRef), v
@ Literal(_: UTF8String, _)) =>
getTargetIndexedColumnName(attrRef, indexedCols)
.map { colName =>
val targetExprBuilder: Expression => Expression =
swapAttributeRefInExpr(sourceExpr, attrRef, _)
- genColumnValuesEqualToExpression(colName, v, targetExprBuilder)
+ Or(genColumnValuesEqualToExpression(colName, v, targetExprBuilder),
Review Comment:
🤖 nit: could you split the two disjuncts into named vals (e.g.
`prefixWithinBounds` and `minStartsWithPrefix`) before combining them with
`Or`? It would make the intent easier to read than the nested call.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
