nsivabalan commented on code in PR #13741:
URL: https://github.com/apache/hudi/pull/13741#discussion_r2299289211
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -44,6 +46,28 @@ import scala.util.Using
class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val sqlTempTable = "tbl"
+ @ParameterizedTest
+ @ValueSource(booleans = Array(true, false))
+ def testRLICreationUsingSQL(isPartitioned: Boolean): Unit = {
Review Comment:
why do we need a new test. why can't we add validations to one of the
existing test only?
`TestPartitionedRecordLevelIndex`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -738,8 +742,8 @@ void initializeFilegroupsAndCommit(MetadataPartitionType
partitionType,
}
private void initializeFilegroupsAndCommitToRecordIndexPartition(String
commitTimeForPartition,
-
Lazy<List<Pair<String, FileSlice>>> lazyLatestMergedPartitionFileSliceList)
throws IOException {
- if (dataWriteConfig.isPartitionedRecordIndexEnabled()) {
+
Lazy<List<Pair<String, FileSlice>>> lazyLatestMergedPartitionFileSliceList,
boolean isPartitioned) throws IOException {
Review Comment:
`isPartitionedRli`. if not, might get confused with data table partitions
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -259,6 +260,13 @@ public boolean buildIndexDefinition(HoodieIndexDefinition
indexDefinition) {
}
if (updateIndexDefn) {
writeIndexMetadataToStorage();
+ String indexMetaPath = getIndexDefinitionPath();
+ // update table config if necessary
Review Comment:
He moved this code from HoodieIndexUtils
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -476,7 +478,9 @@ private boolean initializeFromFilesystem(String
dataTableInstantTime, List<Metad
initializeFilegroupsAndCommit(partitionType,
COLUMN_STATS.getPartitionPath(), fileGroupCountAndRecordsPair,
instantTimeForPartition, colStatsColumnsAndRecord.getKey());
break;
case RECORD_INDEX:
-
initializeFilegroupsAndCommitToRecordIndexPartition(instantTimeForPartition,
lazyLatestMergedPartitionFileSliceList);
+ boolean isPartitioned =
dataWriteConfig.isPartitionedRecordIndexEnabled();
+ createRecordIndexDefinition(dataMetaClient,
Collections.singletonMap(HoodieRecordIndex.IS_PARTITIONED_OPTION,
String.valueOf(isPartitioned)));
Review Comment:
+1
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -656,16 +656,7 @@ public static <R> HoodieRecord<R>
createNewTaggedHoodieRecord(HoodieRecord<R> ol
public static void register(HoodieTableMetaClient metaClient,
HoodieIndexDefinition indexDefinition) {
LOG.info("Registering index {} of using {}",
indexDefinition.getIndexName(), indexDefinition.getIndexType());
// build HoodieIndexMetadata and then add to index definition file
- boolean indexDefnUpdated =
metaClient.buildIndexDefinition(indexDefinition);
- if (indexDefnUpdated) {
- String indexMetaPath = metaClient.getIndexDefinitionPath();
- // update table config if necessary
- if
(!metaClient.getTableConfig().getProps().containsKey(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH.key())
- ||
!metaClient.getTableConfig().getRelativeIndexDefinitionPath().isPresent()) {
-
metaClient.getTableConfig().setValue(HoodieTableConfig.RELATIVE_INDEX_DEFINITION_PATH,
FSUtils.getRelativePartitionPath(metaClient.getBasePath(), new
StoragePath(indexMetaPath)));
- HoodieTableConfig.update(metaClient.getStorage(),
metaClient.getMetaPath(), metaClient.getTableConfig().getProps());
- }
- }
+ metaClient.buildIndexDefinition(indexDefinition);
Review Comment:
Also, shoudln't we make `metaclient.buildIndexDefinition` return void?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
