Re: [I] Spark-compatible CAST operation [datafusion]

2025-05-30 Thread via GitHub
Blizzara commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2922682982 Just fwiw, we already use Comet's Cast in our use of DF, through a rewrite rule: ``` impl FunctionRewrite for OurCastRewrite { fn name(&self) -> &str { "

Re: [I] Spark-compatible CAST operation [datafusion]

2025-05-29 Thread via GitHub
alamb commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2919978927 The key will be to find some way to control what PhysicalExpr is used during the physical planner There might already be an API to do this (provide a custom planner for Expr

Re: [I] Spark-compatible CAST operation [datafusion]

2025-05-28 Thread via GitHub
logan-keede commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2917688776 The comet implementation already has a `PhysicalExpr` for cast. I was thinking if we could make it datafusion compatible(perhaps it already is) and while making physical exp

Re: [I] Spark-compatible CAST operation [datafusion]

2025-05-26 Thread via GitHub
alamb commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2909140210 > do we just need to port [cast](https://github.com/apache/datafusion-comet/blob/main/native/spark-expr/src/conversion_funcs/cast.rs) here from comet? I think that is proba

Re: [I] Spark-compatible CAST operation [datafusion]

2025-05-25 Thread via GitHub
logan-keede commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2908018288 do we just need to port [cast](https://github.com/apache/datafusion-comet/blob/main/native/spark-expr/src/conversion_funcs/cast.rs) here from comet? -- This is an automa

Re: [I] Spark-compatible CAST operation [datafusion]

2025-05-05 Thread via GitHub
alamb commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2852188264 Now that we have the datafusion-spark crate, maybe we have a home for this functionality: - https://github.com/apache/datafusion/issues/15914 -- This is an automated message

Re: [I] Spark-compatible CAST operation [datafusion]

2024-07-05 Thread via GitHub
alamb commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2210821002 > For Expr::Cast that doesn't work as it's part of the Expr enum rather than being a ScalarUDF - is that by choice, or just pending migration? But I plan to try if I can use Funct

Re: [I] Spark-compatible CAST operation [datafusion]

2024-07-03 Thread via GitHub
Blizzara commented on issue #11201: URL: https://github.com/apache/datafusion/issues/11201#issuecomment-2206915698 There's definitely difference in some functions as well, but those have been mostly minor so far (I haven't gotten to regexps or date/time stuff yet), like returning null vs th