dwsmith1983 commented on issue #2190: URL: https://github.com/apache/datafusion-comet/issues/2190#issuecomment-5637988605
The short-term goal stated here is in place: `CometScanRule` falls back for any scan whose schema contains a collated string type (spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala), `QueryPlanSerde.supportedScalarSortElementType` rejects collated sort keys, and #4948 added the same for collated predicate operands, so Spark 4 collation queries run correctly on Spark rather than producing wrong byte-order results. What remains is actual collation-aware native execution, which DataFusion still lacks and which parthchandra's research note scopes out as non-trivial; that is a different piece of work from the fallback this issue asked for and deserves its own issue with that note as the starting point, so the tracking here can close. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
