ErikBPF commented on PR #5903: URL: https://github.com/apache/datafusion-comet/pull/5903#issuecomment-5655417473
Extended the forced-spill regression to cover arrays of structs as well as structs. Required fields are constructed after the Parquet scan, so the new case exercises recursive nullability beneath an array. Each shape runs both `collect_list` and `collect_set`, compares Spark results, requires native aggregation, and asserts a positive spill count. The PR description also records the DataFusion #24767 integration boundary: when upgrading that dependency, re-check outer collection-element nullability against Comet's intermediate buffer schema and rerun this regression. This follow-up does not change the current production declaration. The new array-only case failed with the original strict buffer declaration restored, reporting the expected nested list/struct nullability mismatch. Restoring the fix made the expanded targeted regression pass on both Spark 4.1.3 and Spark 3.5.9, with one test passed per profile and no failures or cancellations. Both profiles passed formatting/style checks; Spark 3.5 used a clean full-reactor build. The native release library was rebuilt from the current PR source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
