andygrove opened a new issue, #1823: URL: https://github.com/apache/datafusion-comet/issues/1823
### Describe the bug In PR https://github.com/apache/datafusion-comet/pull/1736 there is a panic during Comet shuffle when building structs. ``` 2025-05-30T18:36:16.9739600Z [0m[[0m[0minfo[0m] [0m[0m[31m- merge with updates to nested struct fields in NOT MATCHED BY SOURCE clauses *** FAILED *** (98 milliseconds)[0m[0m 2025-05-30T18:36:16.9758247Z [0m[[0m[0minfo[0m] [0m[0m[31m org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 341.0 failed 1 times, most recent failure: Lost task 0.0 in stage 341.0 (TID 432) (1840a49b4efb executor driver): org.apache.comet.CometNativeException: StructBuilder (Schema { fields: [Field { name: "a", data_type: List(Field { name: "item", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "m", data_type: Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metada ta: {} }) and field_builder with index 0 (List(Field { name: "item", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })) are of unequal lengths: (1 != 0).[0m[0m 2025-05-30T18:36:16.9764845Z [0m[[0m[0minfo[0m] [0m[0m[31m at comet::errors::init::{{closure}}(__internal__:0)[0m[0m 2025-05-30T18:36:16.9770774Z [0m[[0m[0minfo[0m] [0m[0m[31m at std::panicking::rust_panic_with_hook(__internal__:0)[0m[0m 2025-05-30T18:36:16.9772048Z [0m[[0m[0minfo[0m] [0m[0m[31m at std::panicking::begin_panic_handler::{{closure}}(__internal__:0)[0m[0m 2025-05-30T18:36:16.9773411Z [0m[[0m[0minfo[0m] [0m[0m[31m at std::sys::backtrace::__rust_end_short_backtrace(__internal__:0)[0m[0m 2025-05-30T18:36:16.9774458Z [0m[[0m[0minfo[0m] [0m[0m[31m at __rustc::rust_begin_unwind(__internal__:0)[0m[0m 2025-05-30T18:36:16.9775560Z [0m[[0m[0minfo[0m] [0m[0m[31m at core::panicking::panic_fmt(__internal__:0)[0m[0m 2025-05-30T18:36:16.9777053Z [0m[[0m[0minfo[0m] [0m[0m[31m at arrow_array::builder::struct_builder::StructBuilder::validate_content::{{closure}}::panic_cold_display(__internal__:0)[0m[0m 2025-05-30T18:36:16.9778679Z [0m[[0m[0minfo[0m] [0m[0m[31m at arrow_array::builder::struct_builder::StructBuilder::validate_content(__internal__:0)[0m[0m 2025-05-30T18:36:16.9780025Z [0m[[0m[0minfo[0m] [0m[0m[31m at arrow_array::builder::struct_builder::StructBuilder::finish(__internal__:0)[0m[0m 2025-05-30T18:36:16.9781598Z [0m[[0m[0minfo[0m] [0m[0m[31m at <arrow_array::builder::struct_builder::StructBuilder as arrow_array::builder::ArrayBuilder>::finish(__internal__:0)[0m[0m 2025-05-30T18:36:16.9783111Z [0m[[0m[0minfo[0m] [0m[0m[31m at arrow_array::builder::struct_builder::StructBuilder::finish(__internal__:0)[0m[0m 2025-05-30T18:36:16.9784815Z [0m[[0m[0minfo[0m] [0m[0m[31m at <arrow_array::builder::struct_builder::StructBuilder as arrow_array::builder::ArrayBuilder>::finish(__internal__:0)[0m[0m 2025-05-30T18:36:16.9791924Z [0m[[0m[0minfo[0m] [0m[0m[31m at <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::next(__internal__:0)[0m[0m 2025-05-30T18:36:16.9793670Z [0m[[0m[0minfo[0m] [0m[0m[31m at comet::execution::shuffle::row::process_sorted_row_partition(__internal__:0)[0m[0m 2025-05-30T18:36:16.9795501Z [0m[[0m[0minfo[0m] [0m[0m[31m at comet::execution::jni_api::Java_org_apache_comet_Native_writeSortedFileNative::{{closure}}::{{closure}}(__internal__:0)[0m[0m 2025-05-30T18:36:16.9797049Z [0m[[0m[0minfo[0m] [0m[0m[31m at Java_org_apache_comet_Native_writeSortedFileNative(__internal__:0)[0m[0m `` ### Steps to reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
