nsivabalan commented on code in PR #13292:
URL: https://github.com/apache/hudi/pull/13292#discussion_r2103908607
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:
##########
@@ -84,8 +88,19 @@ public class HoodieMetadataWriteUtils {
*/
@VisibleForTesting
public static HoodieWriteConfig createMetadataWriteConfig(
- HoodieWriteConfig writeConfig, HoodieFailedWritesCleaningPolicy
failedWritesCleaningPolicy) {
+ HoodieWriteConfig writeConfig, HoodieFailedWritesCleaningPolicy
failedWritesCleaningPolicy,
+ HoodieTableVersion datatableVersion) {
String tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+ boolean isStreamingWritesToMetadataEnabled =
writeConfig.isStreamingWritesToMetadataEnabled(datatableVersion);
+ WriteConcurrencyMode concurrencyMode = isStreamingWritesToMetadataEnabled
+ ? WriteConcurrencyMode.NON_BLOCKING_CONCURRENCY_CONTROL :
WriteConcurrencyMode.SINGLE_WRITER;
+ HoodieLockConfig lockConfig = isStreamingWritesToMetadataEnabled
+ ?
HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class).build()
: HoodieLockConfig.newBuilder().build();
Review Comment:
this is little intricate.
here is the rational:
We are not looking to make MDT writes completely independent of data table
locks yet. Even w/ streaming writes, the final write where we write FILES
partition to MDT still happens within data table lock. So, we don't really need
a separate lock for MDT.
for instance:
```
BaseHoodieWriteClient.commit() {
acquire data table lock
conflict resolution
Marker reconciliation.
write to metadata table.
complete the commit data table.
}
```
So, we are not looking to dislodge this w/ streaming writes. This is going
to stay the same.
Only change will be, for the streaming writes (all partitions in MDT except
FILES partition), we will be writing to MDT w/o any locks. This will be
synonymous to how data table write handles execute w/o any locks even if user
have configured a lock provider for data table in general.
So, given this, we do not really need a completely diff lock to MDT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
