Blizzara commented on PR #14392: URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2630634253
I'd be very happy to see this go in :) Having the functions here in main datafusion repo makes dealing with any breaks and downstream upgrades easier (we'll only need to track/match DF & object_store, but no more Comet). And it maybe allows reusing some of the internal macros, making it easier to contribute new stuff, and reusing logic across the existing impls and the Spark impls. So, nice! Re the conflicts between Spark and existing impls, not sure what the best there would be, but I think in general the people who care about Spark-compatibility likely don't care about the non-compatible functions, so it could be some kind of toggle on the context/state level. Maybe supplying either a DataFusion-(or Postgres-)FunctionRegistry (default) or a SparkFunctionRegistry, or something like that? Also, this includes things that aren't strictly UD(A/W)Fs, like [cast](https://github.com/apache/datafusion/pull/14392/files#diff-dfa667ba378c7329ba68ff11097b90144fb2704f901849e70254188eacf820c2). We currently use a FunctionRewrite to convert all DF casts to use the Comet cast, but maybe there could be a better way for that as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
