prashantwason commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2910863621
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -114,14 +110,9 @@ protected Option<HoodieTableMetadataWriter>
getMetadataWriter(
HoodieTableMetadataWriter metadataWriter = streamingWrites
?
SparkMetadataWriterFactory.createWithStreamingWrites(getContext().getStorageConf(),
config, failedWritesCleaningPolicy, getContext(),
Option.of(triggeringInstantTimestamp))
: SparkMetadataWriterFactory.create(getContext().getStorageConf(),
config, failedWritesCleaningPolicy, getContext(),
Option.of(triggeringInstantTimestamp), metaClient.getTableConfig());
- try {
- if (isMetadataTableExists || metaClient.getStorage().exists(
-
HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath()))) {
- isMetadataTableExists = true;
- return Option.of(metadataWriter);
- }
- } catch (IOException e) {
- throw new HoodieMetadataException("Checking existence of metadata
table failed", e);
+ if (isMetadataTableExists || metadataWriter.isInitialized()) {
Review Comment:
Good question. The MDT partitions will NOT go stale in this scenario. Here's
why:
The metadata writer's `getMetadataPartitionsToUpdate()` method (in
`HoodieBackedTableMetadataWriter.java:1171`) uses
`dataMetaClient.getTableConfig().getMetadataPartitions()` as the source of
truth for which partitions to update — not the write config's enabled partition
settings. So even if the write config doesn't explicitly enable certain MDT
partitions, as long as the table config marks them as available, the writer
will continue to update them.
With this PR's change to add the `isMetadataTableAvailable()` check, the
writer is always created when MDT exists on disk, ensuring the partitions stay
in sync regardless of the write config's `metadata.enable` setting.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -509,6 +509,18 @@ public final class HoodieMetadataConfig extends
HoodieConfig {
+ "The index name either starts with or matches exactly can be one
of the following: "
+
StringUtils.join(Arrays.stream(MetadataPartitionType.values()).map(MetadataPartitionType::getPartitionPath).collect(Collectors.toList()),
", "));
+ // Config to enable/disable automatic deletion of MDT partitions
Review Comment:
Removed the redundant comment in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
