findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2659658450
Sharing WIP status here
- https://github.com/apache/datafusion/pull/14668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2655964195
> downstream crates might need to override the default signature
they would define own function
either via implementation or via "simplify" (not covered by the cur
shehabgamin commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2655822219
@findepi Thank you for thoroughly exploring this topic and creating such a
detailed design document!
@linhr and I have reviewed it and have some initial thoughts:
comphead commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2654675641
Thanks @findepi and everyone, this work is epic, literally. in DataFusion it
was always needed to unify the builtin functions as they implemented by
different developers in dif
Omega359 commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2650993466
I love the work that you've put into this! You use the regexp_like as an
example of what could be handled by if you look at that function it actually
doesn't operate on individ
findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2650048313
Going back to this topic. Not much happened on the Simple Functions front,
but a lot happened in the world. New type system changes (Logical types can be
found in type signature
davidhewitt commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2626830164
Thanks, looks similar to what we've done in `datafusions-functions-json` but
we tried to handle dictionaries without casting them away. I guess we should
just cast them and
Omega359 commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2622912627
https://github.com/apache/datafusion/blob/main/datafusion/functions-nested/src/string.rs
is a good example of what can result with supporting many types and args
--
This is
adriangb commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2622869956
I'll let @davidhewitt chime in but we've experience a lot of generic bloat
from having to implement functions that operate on scalars, arrays, dictionary
arrays and take multip
Omega359 commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2408232282
> @Omega359 this is how the function logic is structured anyway
🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, plea
findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2408139607
@Omega359 this is how the function logic is structured anyway --
https://github.com/apache/datafusion/blob/a08f923c2acb1a46614970231d9a672c36ce3ad2/datafusion/functions/src/math/
Omega359 commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2405845576
```
pub fn call(x: i64, y: i64) -> Result {
datafusion::functions::math::gcd::compute_gcd(x, y)
}
```
Wouldn't this incur a significant amount
comphead commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2405413561
> I experimented with this on the way from DataFusion meetup in Belgrade.
>
> i came up with something like this
>
> function author would write this
> row-
notfilippo commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2404398841
> @notfilippo could we perhaps have a logical types stub in main at some
point?
Planning on opening the PR on `main` soon. Then once it gets merged I can
merge `main`
findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2401892724
I experimented with this on the way from DataFusion meetup in Belgrade.
i came up with something like this
function author would write this
row-oriented
alamb commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2380708925
> . It was an attempt to summarize why we need both: simpler types
(https://github.com/apache/datafusion/issues/11513), more types
(https://github.com/apache/datafusion/issues/126
findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2380660829
FYI i touched upon the topic of types on DataFusion meetup in Belgrade
yesterday.
The slides are here if anyone is interested:
https://docs.google.com/presentation/d/1VW_JCGb
alamb commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2378978643
Just to be clear, what I am imagining comes out of this work is:
1. No changes to `ScalarUDFImpl`
2. Some new way (generic functions, macros, etc) that would make creating
th
comphead commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2378116102
Thanks @findepi I think this process go through iterations, and easier than
was before but still far from perfect.
The ScalarUDFImpl common trait is already a huge help,
alamb commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2377975313
> Currently DataFusion functions are singletons plugged into the execution
engine. They have no way to store and reuse buffers or compiled regular
expressions, etc.
here i
alamb commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2377975178
I think the idea of making it easier to write functions that include
specialized implementations for different types is a great idea. This would
likely both make our code faster (
findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2376910852
FYI: I am doing some experiments how this could look like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and us
findepi commented on issue #12635:
URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2376798706
cc @alamb, @andygrove, @jayzhan211, @ozankabak, @notfilippo, @comphead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
findepi opened a new issue, #12635:
URL: https://github.com/apache/datafusion/issues/12635
### Is your feature request related to a problem or challenge?
### Verbosity
Currently implementing a scalar function is a pretty involved process. For
example a simple function calculati
24 matches
Mail list logo
