viirya commented on PR #5778: URL: https://github.com/apache/datafusion-comet/pull/5778#issuecomment-5608651508
@andygrove Thanks for re-running the suite and clarifying the remaining points. Your concern about both sides of the Rust comparisons sharing the modified kernel is well taken: agreement alone would not catch a systematic change in hash values. The explicit Spark comparisons provide an independent oracle, with the native-operator assertions guarding against fallback masking a mismatch. I updated the second test’s comment to describe what it actually covers: nested-element fallback and chaining a flat-struct column with a nested column. It no longer claims dictionary encoding, which the test does not assert. On `generateArray`, I checked the generator: with the current `maxDepth = 2`, enabling it produces arrays of primitives, but not `array<struct<...>>`, because container selection and child generation each increment the depth. It would broaden general array coverage without exercising this batching path. I’m keeping the explicit Spark-oracle tests for this PR. A follow-up fuzz extension should ensure the target schemas are generated and compare both hash functions with native execution asserted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
