yihua commented on code in PR #13208:
URL: https://github.com/apache/hudi/pull/13208#discussion_r2055542014
##########
hudi-common/src/main/java/org/apache/hudi/keygen/constant/KeyGeneratorType.java:
##########
@@ -150,4 +150,12 @@ public static String getKeyGeneratorClassName(Map<String,
String> config) {
}
return null;
}
+
+ public KeyGeneratorType getAvroImplementation() {
+ if (name().endsWith("AVRO")) {
+ return this;
+ }
+ String avroName = name() + "_AVRO";
+ return KeyGeneratorType.valueOf(avroName);
Review Comment:
There are certain key generator types that do not have "AVRO" counterpart,
e.g., `KeyGeneratorType.HOODIE_TABLE_METADATA` (which is the only important one
to take care of). Should that be accounted for since MDT read will use
`HoodieAvroReaderContext`?
##########
hudi-common/src/test/java/org/apache/hudi/avro/TestHoodieAvroReaderContext.java:
##########
@@ -182,6 +185,20 @@ void getRecordKeyFromMetadataFields() {
assertEquals(recordKey, avroReaderContext.getRecordKey(indexedRecord,
schemaWithMetaFields));
}
+ @ParameterizedTest
+ @ValueSource(booleans = {false, true})
+ void getKeyGeneratorClassDefaults(boolean isPartitioned) {
Review Comment:
Test MDT keygen too.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -136,7 +136,7 @@ public void
testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
initialRecords, initialRecords);
// Two commits; reading one file group containing a base file and a log
file
- List<HoodieRecord> updates = dataGen.generateUniqueUpdates("002", 50);
+ List<HoodieRecord> updates = dataGen.generateUniqueUpdates("002", 90);
Review Comment:
Any reason to change this?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -154,10 +154,17 @@ public void
testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ private static Stream<Arguments> logFileOnlyCases() {
+ return Stream.of(
+ arguments(RecordMergeMode.COMMIT_TIME_ORDERING, "avro"),
+ arguments(RecordMergeMode.EVENT_TIME_ORDERING, "parquet"),
+ arguments(RecordMergeMode.CUSTOM, "avro"));
+ }
+
@ParameterizedTest
- @MethodSource("testArguments")
- public void testReadLogFilesOnlyInMergeOnReadTable(RecordMergeMode
recordMergeMode, String logDataBlockFormat, boolean populateMetaFields) throws
Exception {
- Map<String, String> writeConfigs = new
HashMap<>(getCommonConfigs(recordMergeMode, populateMetaFields));
+ @MethodSource("logFileOnlyCases")
+ public void testReadLogFilesOnlyInMergeOnReadTable(RecordMergeMode
recordMergeMode, String logDataBlockFormat) throws Exception {
Review Comment:
@nsivabalan @danny0405 is virtual key not supported with in-memory index?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
