nsivabalan commented on code in PR #19205:
URL: https://github.com/apache/hudi/pull/19205#discussion_r3573053381
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowCreateHandle.java:
##########
@@ -160,11 +170,45 @@ public HoodieRowCreateHandle(HoodieTable table,
public void write(InternalRow row) throws IOException {
if (populateMetaFields) {
writeRow(row);
+ } else if (populateCommitTime || populateFileName) {
+ writeRowSelectiveMetaFields(row);
} else {
writeRowNoMetaFields(row);
}
}
+ /**
+ * Selective meta-field write path: populate only the meta columns opted in
via
+ * {@code hoodie.meta.fields.mode} — {@code _hoodie_commit_time} (with
derived seq id) and/or
+ * {@code _hoodie_file_name}. The other meta columns stay null on disk.
Record key is never
+ * populated in this path, so the record key is not registered with the
write support (bloom
+ * filter / RLI hooks are meaningless without the record-key column).
+ */
+ private void writeRowSelectiveMetaFields(InternalRow row) {
+ try {
+ UTF8String[] metaFields = new UTF8String[5];
+ if (populateCommitTime) {
+ metaFields[HoodieRecord.COMMIT_TIME_METADATA_FIELD_ORD] =
shouldPreserveHoodieMetadata
+ ? row.getUTF8String(HoodieRecord.COMMIT_TIME_METADATA_FIELD_ORD) :
commitTime;
+ metaFields[HoodieRecord.COMMIT_SEQNO_METADATA_FIELD_ORD] =
shouldPreserveHoodieMetadata
Review Comment:
Addressed in
[cfd1fea706ed](https://github.com/apache/hudi/pull/19205/commits/cfd1fea706edee5d245201f057cd8755e724d9af):
dropped `_hoodie_commit_seqno` population from the selective path. Only
`_hoodie_commit_time` is populated. Downstream consumers that assume seqno is
populated whenever commit_time is (if any exist) are out of scope for this PR —
the seqno column simply stays null in `COMMIT_TIME_ONLY` /
`COMMIT_TIME_AND_FILE_NAME`. Rebased on latest upstream/master.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkParquetWriter.java:
##########
@@ -74,6 +94,16 @@ public void writeRowWithMetadata(HoodieKey key, InternalRow
row) throws IOExcept
super.write(row);
writeSupport.add(recordKey);
+ } else if (populateCommitTime || populateFileName) {
+ if (populateCommitTime) {
+ row.update(COMMIT_TIME_METADATA_FIELD.ordinal(), instantTime);
+ row.update(COMMIT_SEQNO_METADATA_FIELD.ordinal(),
Review Comment:
Addressed in
[cfd1fea706ed](https://github.com/apache/hudi/pull/19205/commits/cfd1fea706edee5d245201f057cd8755e724d9af)
— same as above. Seqno population dropped from the selective path in
`HoodieSparkParquetWriter#writeRowWithMetadata`.
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroParquetWriter.java:
##########
@@ -72,6 +94,20 @@ public void writeAvroWithMetadata(HoodieKey key,
IndexedRecord avroRecord) throw
taskContextSupplier.getPartitionIdSupplier().get(),
getWrittenRecordCount(), fileName);
super.write(avroRecord);
writeSupport.add(key.getRecordKey());
+ } else if (populateCommitTime || populateFileName) {
+ // Selective meta-field population. The other meta columns stay null on
disk, which Parquet
+ // stores as definition-level flags (zero data bytes). Bloom filter /
record-key index
+ // population is intentionally skipped — that requires the record-key
column.
+ GenericRecord genericRecord = (GenericRecord) avroRecord;
+ if (populateCommitTime) {
+ String seqId = HoodieRecord.generateSequenceId(instantTime,
Review Comment:
Addressed in
[cfd1fea706ed](https://github.com/apache/hudi/pull/19205/commits/cfd1fea706edee5d245201f057cd8755e724d9af)
— same as above. Seqno population dropped from the selective path in
`HoodieAvroParquetWriter#writeAvroWithMetadata`.
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroFileWriterFactory.java:
##########
@@ -89,7 +89,7 @@ protected HoodieFileWriter newParquetFileWriter(
hoodieConfig.getLongOrDefault(HoodieStorageConfig.PARQUET_MAX_FILE_SIZE),
storageConfiguration,
hoodieConfig.getDoubleOrDefault(HoodieStorageConfig.PARQUET_COMPRESSION_RATIO_FRACTION),
hoodieConfig.getBooleanOrDefault(HoodieStorageConfig.PARQUET_DICTIONARY_ENABLED));
- return new HoodieAvroParquetWriter(path, parquetConfig, instantTime,
taskContextSupplier, populateMetaFields);
+ return new HoodieAvroParquetWriter(path, parquetConfig, instantTime,
taskContextSupplier, populateMetaFields, metaFieldsMode);
Review Comment:
Addressed in
[cfd1fea706ed](https://github.com/apache/hudi/pull/19205/commits/cfd1fea706edee5d245201f057cd8755e724d9af):
folded `populateMetaFields` + `metaFieldsMode` into a single `MetaFieldsMode`
enum (`ALL` / `NONE` / `COMMIT_TIME_ONLY` / `FILE_NAME_ONLY` /
`COMMIT_TIME_AND_FILE_NAME`). The enum is now the sole meta-field argument
passed through every writer factory and writer constructor.
`populate.meta.fields` remains on the config surface for backward compat and is
auto-derived from the enum. The predicate helpers `isCommitTimePopulated()` /
`isFileNamePopulated()` on the enum are used in write branches. Also note: the
follow-up CLI PR ([#19206](https://github.com/apache/hudi/pull/19206)) uses the
same enum for the `table set-meta-fields-mode` command surface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
