[jira] [Commented] (HUDI-4330) NPE when trying to upsert into a dataset with no Meta Fields

[xi chaomin (Jira)]

Title: Message Title

xi chaomin commented on HUDI-4330   Re: NPE when trying to upsert into a dataset with no Meta Fields Currently, bloom filter depend on "hoodie.populate.meta.fields", If "hoodie.populate.meta.fields" is false, we won't write bloom filter to the footer.  Some code: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieAvroParquetWriter.java   @Override   public void writeAvroWithMetadata(HoodieKey key, R avroRecord) throws IOException {     if (populateMetaFields) {       prepRecordWithMetadata(key, avroRecord, instantTime,           taskContextSupplier.getPartitionIdSupplier().get(), getWrittenRecordCount(), fileName);       super.write(avroRecord);       writeSupport.add(key.getRecordKey());     } else {       super.write(avroRecord);     }   }  @Override   public void writeAvro(String key, IndexedRecord object) throws IOException {     super.write(object);     if (populateMetaFields) {       writeSupport.add(key);     }   }   hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieFileWriterFactory.java private static <T extends HoodieRecordPayload, R extends IndexedRecord> HoodieFileWriter<R> newParquetFileWriter( String instantTime, Path path, HoodieWriteConfig config, Schema schema, HoodieTable hoodieTable, TaskContextSupplier taskContextSupplier, boolean populateMetaFields) throws IOException { boolean enableBloomFilter = config.getIndexType().name().equals(BLOOM.name()) || config.getIndexType().name().equals(GLOBAL_BLOOM.name()); return newParquetFileWriter(instantTime, path, config, schema, hoodieTable.getHadoopConf(), taskContextSupplier, populateMetaFields, enableBloomFilter); } private static <T extends HoodieRecordPayload, R extends IndexedRecord> HoodieFileWriter<R> newParquetFileWriter( String instantTime, Path path, HoodieWriteConfig config, Schema schema, Configuration conf, TaskContextSupplier taskContextSupplier, boolean populateMetaFields, boolean enableBloomFilter) throws IOException { Option<BloomFilter> filter = enableBloomFilter ? Option.of(createBloomFilter(config)) : Option.empty(); HoodieAvroWriteSupport writeSupport = new HoodieAvroWriteSupport(new AvroSchemaConverter(conf).convert(schema), schema, filter); HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig = new HoodieParquetConfig<>(writeSupport, config.getParquetCompressionCodec(), config.getParquetBlockSize(), config.getParquetPageSize(), config.getParquetMaxFileSize(), conf, config.getParquetCompressionRatio(), config.parquetDictionaryEnabled()); return new HoodieAvroParquetWriter<>(path, parquetConfig, instantTime, taskContextSupplier, populateMetaFields); } Add Comment   This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)