prashantwason commented on code in PR #18181:
URL: https://github.com/apache/hudi/pull/18181#discussion_r2910865241
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -509,6 +509,18 @@ public final class HoodieMetadataConfig extends
HoodieConfig {
+ "The index name either starts with or matches exactly can be one
of the following: "
+
StringUtils.join(Arrays.stream(MetadataPartitionType.values()).map(MetadataPartitionType::getPartitionPath).collect(Collectors.toList()),
", "));
+ // Config to enable/disable automatic deletion of MDT partitions
+ public static final ConfigProperty<Boolean> AUTO_DELETE_PARTITIONS =
ConfigProperty
+ .key(METADATA_PREFIX + ".auto.delete.partitions")
+ .defaultValue(true)
Review Comment:
Added two tests in `TestHoodieTable.java` in the latest commit:
1. `testMaybeDeleteMetadataTableSkipsWhenAutoDeleteDisabled` - verifies MDT
is not deleted when `auto.delete.partitions=false`
2. `testDeleteMetadataIndexIfNecessarySkipsWhenAutoDeleteDisabled` -
verifies index deletion is skipped when `auto.delete.partitions=false`
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -114,14 +110,9 @@ protected Option<HoodieTableMetadataWriter>
getMetadataWriter(
HoodieTableMetadataWriter metadataWriter = streamingWrites
?
SparkMetadataWriterFactory.createWithStreamingWrites(getContext().getStorageConf(),
config, failedWritesCleaningPolicy, getContext(),
Option.of(triggeringInstantTimestamp))
: SparkMetadataWriterFactory.create(getContext().getStorageConf(),
config, failedWritesCleaningPolicy, getContext(),
Option.of(triggeringInstantTimestamp), metaClient.getTableConfig());
- try {
- if (isMetadataTableExists || metaClient.getStorage().exists(
-
HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath()))) {
- isMetadataTableExists = true;
- return Option.of(metadataWriter);
- }
- } catch (IOException e) {
- throw new HoodieMetadataException("Checking existence of metadata
table failed", e);
+ if (isMetadataTableExists || metadataWriter.isInitialized()) {
Review Comment:
The writer creation path in `HoodieSparkTable.getMetadataWriter()` requires
Spark context for a proper integration test. The two unit tests added in
`TestHoodieTable.java` cover the auto-delete guard behavior at the
`HoodieTable` level.
For the outdated partitions concern — as explained above, the metadata
writer uses `dataMetaClient.getTableConfig().getMetadataPartitions()` as the
source of truth for updates, so partitions won't go stale as long as the writer
is created. The `isMetadataTableAvailable()` condition ensures the writer is
created whenever MDT exists on disk.
If you'd like a Spark integration test specifically for the writer creation
path, I can add one — let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
