kbuci commented on code in PR #18295:
URL: https://github.com/apache/hudi/pull/18295#discussion_r2976402112
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:
##########
@@ -350,6 +388,26 @@ public static HoodieWriteConfig createMetadataWriteConfig(
}
HoodieWriteConfig metadataWriteConfig = builder.build();
+ if (deriveMetadataLockConfigsFromDataTableConfigs) {
+ // We need to update the MDT write config to have the same lock related
configs as the data table.
+ // All data table props with the lock prefix are always copied (to
override MDT defaults with
+ // user-configured values). Other data table props not present in MDT
config are also copied to
+ // support custom lock providers that may use non-standard config keys.
+ Properties lockProps = new Properties();
+ TypedProperties dataTableProps = writeConfig.getProps();
+ TypedProperties mdtProps = metadataWriteConfig.getProps();
+ for (String key : dataTableProps.stringPropertyNames()) {
Review Comment:
Yeah because of the possible edge cases I noticed and that you called out in
the above comment, I am leaning towards instead changing this PR to back to my
first attempt - creating a new lock config for MDT write config to use by
> iterating through lock provider types the same way we already handle all
metrics reporter type here. But then the issue is that this would have to be
updated anytime we add a new lock config or lock provider
Since even though that increases the amount of code by a lot (to handle
every lock provider case for lock providers implemented in HUDI), at least it
will be easier to follow and mainly update.
And for lock provider classes that are not in packages accessible to this
class, we resort to the heuristic/hack of putting in all props (from data table
write config) with
`key.startsWith(LockConfiguration.LOCK_PREFIX)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
