the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300863078
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public HoodieRecord.HoodieRecordType getRecordType() {
* Basic handling of deletes that is used by many of the spark mergers
* returns null if merger specific logic should be used
*/
- protected Option<Pair<HoodieRecord, Schema>> handleDeletes(HoodieRecord
older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties
props) {
+ protected Pair<HoodieRecord, Schema> handleDeletes(HoodieRecord older,
Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
ValidationUtils.checkArgument(older.getRecordType() ==
HoodieRecord.HoodieRecordType.SPARK);
ValidationUtils.checkArgument(newer.getRecordType() ==
HoodieRecord.HoodieRecordType.SPARK);
if (newer instanceof HoodieSparkRecord) {
HoodieSparkRecord newSparkRecord = (HoodieSparkRecord) newer;
if (newSparkRecord.isDelete(newSchema, props)) {
// Delete record
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(),
HoodieOperation.DELETE, OrderingValues.getDefault(),
HoodieRecord.HoodieRecordType.SPARK), newSchema);
Review Comment:
There are definitely gaps with respect to event time ordering. Even the
payload flow was not allowing ordering value on some deletes so it is quite
inconsistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
