vinothchandar commented on a change in pull request #4067:
URL: https://github.com/apache/hudi/pull/4067#discussion_r765264407
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java
##########
@@ -137,6 +136,12 @@
.withDocumentation("There are cases when extra files are requested to be
deleted from metadata table which was never added before. This config"
+ "determines how to handle such spurious deletes");
+ public static final ConfigProperty<Boolean> RECORDKEY_DE_DUPLICATE =
ConfigProperty
+ .key(METADATA_PREFIX + ".recordkey.deduplicate")
+ .defaultValue(true)
+ .sinceVersion("0.10.0")
Review comment:
this is 0.11.0
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java
##########
@@ -137,6 +136,12 @@
.withDocumentation("There are cases when extra files are requested to be
deleted from metadata table which was never added before. This config"
+ "determines how to handle such spurious deletes");
+ public static final ConfigProperty<Boolean> RECORDKEY_DE_DUPLICATE =
ConfigProperty
+ .key(METADATA_PREFIX + ".recordkey.deduplicate")
+ .defaultValue(true)
+ .sinceVersion("0.10.0")
+ .withDocumentation("When enabled, Metadata table records will have key
de-duplication.");
Review comment:
can we make this doc clearer? IMO this has to be an internal config. i.e
`"_." + METADATA_PREFIX + ".recordkey.deduplicate"` that the user should not
normally bother with
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
##########
@@ -98,16 +101,13 @@ public HoodieLogFileReader(FileSystem fs, HoodieLogFile
logFile, Schema readerSc
this.reverseReader = reverseReader;
this.enableInlineReading = enableInlineReading;
this.keyField = keyField;
+ this.keyDeDuplication = false;
Review comment:
can we fix the naming here? we are just not storing the `key ` again in
the payload right? Can we refer to this as `excludeKeyInPayload` or sth like
that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
