voonhous commented on PR #20036: URL: https://github.com/apache/hudi/pull/20036#issuecomment-5807907946
Okay, the squashed commit message for this PR is bugged. For anyone that is trying to figure out what the commit message is: ``` PushVariantIntoScan rewrites struct paths only, so a variant that is an array element or a map value reaches the reader as native VariantType and the reader context leaves it alone (#19783). Pin that "left alone" means read correctly: extractions, casts, filters, explodes and whole collection reads over parquet base files, parquet and avro log blocks and the merged MOR row, with a null element, a null map value and a null struct member arriving through the log, beside a top-level variant that is rewritten on the conf-on arm in the same scan. The plan check asserts the collections stay native on both arms. Every MOR leg then compacts and repeats the same reads over the compacted base file, so the collections and their nulls are also pinned through the compaction merge path. The Avro write path forces parquet-avro's old list structure, which cannot write a null array element of any type, so the null-element case runs on the SPARK legs only. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
