cshuo commented on code in PR #20292:
URL: https://github.com/apache/hudi/pull/20292#discussion_r4266417145
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PrimaryKeyPruners.java:
##########
@@ -55,6 +58,18 @@ public static Function<Integer, Integer>
getBucketIdFunc(List<ResolvedExpression
// the hash keys must be evaluated in the record key field sequence.
.sorted(java.util.Map.Entry.comparingByKey())
.map(Pair::getValue)
+ .map(value -> {
+ // Compound record keys are decoded by
KeyGenUtils.extractRecordKeysByFields before hashing.
+ // Match its placeholder handling even when the bucket index uses
only one of the key fields.
+ if (pkFields.size() > 1) {
+ if (NULL_RECORDKEY_PLACEHOLDER.equals(value)) {
Review Comment:
This is a corner case, but user-provided keys can contain arbitrary strings,
including the non-empty literals `'__empty__'` or `'__null__'`. The writer
decodes these as placeholders before hashing, while pruning hashes them
literally and selects the wrong bucket.
The ambiguity you pointed out does exist: the current encoding cannot
distinguish placeholder literals from actual null/empty values. That is a
pre-existing issue requiring a separate fix. This change only aligns pruning
with the existing writer bucket assignment; it does not change key equality or
sorting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
