danny0405 commented on a change in pull request #4880: URL: https://github.com/apache/hudi/pull/4880#discussion_r818311010
########## File path: hudi-common/src/main/java/org/apache/hudi/common/model/DeleteKey.java ########## @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hudi.common.model; + +import java.util.Objects; + +/** + * Delete key is a combination of HoodieKey and ordering value. + * The key is used for {@link org.apache.hudi.common.table.log.block.HoodieDeleteBlock} + * to support per-record deletions. The deletion block is always appended after the data block, + * we need to keep the ordering val to combine with the data records when merging, or the data may + * be dropped if there are intermediate deletions for the inputs + * (a new INSERT comes after a DELETE in one input batch). + */ +public class DeleteKey extends HoodieKey { Review comment: > +1 on providing concrete suggestions to move forward. > > @danny0405 The main contention here is that `DeleteKey` is not really a key. we need some cleanup where we pass the orderingVal from the engine API (flink level), all the way down the actual places where I/O happens. Also mandate orderingVal across all the payloads. I was hoping we would do that in the new merge API design. But that's getting delayed now. Thanks, here is the case, i'm trying to explain the details again When consuming the CDC source, flink would tag each record with a `RowKind` corresponding to the `HoodieOperation` field of our format. When user opens the changelog mode, we do not use the DELETE block encode and treat each delete records as normal data payload in order to keep the deleted val to use for retraction in streaming. So the current code can work correctly. But in the default upsert mode, say user do not want to keep the hoodie operations flags for streaming read, the bug/problem comes in: We use the DELETE block encoding and if there is disorder deletes in the merge handle batch, the data lost happens. That's why i try to fix that in this PR then. Hope this can give the help for the ideas of designing new merging APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
