yihua commented on code in PR #13711:
URL: https://github.com/apache/hudi/pull/13711#discussion_r2350093739
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/table/upgrade/TestNineToEightDowngradeHandler.java:
##########
@@ -265,7 +265,7 @@ void testDowngradeDropsOnlyV2OrAboveIndexes() {
)).thenAnswer(invocation -> null); // Do nothing
// Mock the dropNonV1SecondaryIndexPartitions to simulate dropping V2
indexes
- mockedUtils.when(() ->
UpgradeDowngradeUtils.dropNonV1SecondaryIndexPartitions(
+ mockedUtils.when(() -> UpgradeDowngradeUtils.dropNonV1IndexPartitions(
Review Comment:
Do we have functional tests (without mocks) on dropping v2 indexes?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -575,10 +576,11 @@ private Pair<List<String>, Pair<Integer,
HoodieData<HoodieRecord>>> initializeCo
if (partitionIdToAllFilesMap.isEmpty()) {
return Pair.of(Collections.emptyList(), Pair.of(fileGroupCount,
engineContext.emptyHoodieData()));
}
+ HoodieIndexVersion columnStatsIndexVersion =
existingIndexVersionOrDefault(PARTITION_NAME_COLUMN_STATS, dataMetaClient);
// Find the columns to index
final List<String> columnsToIndex = new
ArrayList<>(HoodieTableMetadataUtil.getColumnsToIndex(dataMetaClient.getTableConfig(),
dataWriteConfig.getMetadataConfig(), tableSchema, true,
-
Option.of(dataWriteConfig.getRecordMerger().getRecordType())).keySet());
+ Option.of(dataWriteConfig.getRecordMerger().getRecordType()),
columnStatsIndexVersion).keySet());
Review Comment:
Let's fix this one.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToNineUpgradeHandler.java:
##########
@@ -117,6 +124,18 @@ public UpgradeDowngrade.TableConfigChangeSet
upgrade(HoodieWriteConfig config,
metaClient.getIndexDefinitionPath(),
indexMetadataOpt.get(),
metaClient.getTableConfig().getTableVersion());
+ //Validate index columns
+ TableSchemaResolver tableSchemaResolver = new
TableSchemaResolver(metaClient);
+ try {
+ List<String> partitionsToDrop = mdtPartitionsToDrop(indexMetadataOpt,
tableSchemaResolver.getTableAvroSchema());
+ if (!partitionsToDrop.isEmpty()) {
+ try (BaseHoodieWriteClient writeClient =
upgradeDowngradeHelper.getWriteClient(config, context)) {
+ writeClient.dropIndex(partitionsToDrop);
+ }
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
Review Comment:
I think the column stats index can follow the same upgrade and downgrade
flow as secondary index where V1 column stats index can be updated after table
upgrade from v8 to v9; during downgrade, V2 column stats index is dropped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
