yihua opened a new issue, #20253: URL: https://github.com/apache/hudi/issues/20253
Prepped Spark writes on the default AVRO record type (Spark SQL `UPDATE`, pk-less `MERGE INTO`, datasource prepped upserts) call `HoodieSchemaUtils.removeMetadataFields(dataFileSchema)` inside the per-record loop of `HoodieCreateRecordUtils.createHoodieRecordRdd`, building a new Avro schema for every record; each field copy also re-parses the Avro version string (`HoodieAvroUtils.gteqAvro1_12`). On a 6M-row, 30-column table, an `UPDATE` touching 612k rows allocated 332 GB, two thirds of it in this schema construction. Proposal: build the schema once per partition and evaluate the Avro version checks once. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
