yihua opened a new issue, #20268: URL: https://github.com/apache/hudi/issues/20268
Reading or compacting MOR file groups that carry partial-update log records (from a partial `MERGE INTO`) spends most of the per-record merge cost comparing Avro schemas: 16% of executor CPU and 47% of allocation in a 6M-row Spark SQL run. `SparkRecordMergingUtils` builds the merged schema of a full-plus-partial merge by converting the Spark struct type back to a `HoodieSchema`, which drops field defaults, so the result is unequal to the reader schema but hashes the same. Every schema-keyed lookup holding both then compares the two field by field, once per merged record. Proposal: reuse the reader schema as the merged schema when the merge covers every reader field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
