alamb commented on PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846971785
> Fyi, the main CI has failed since the PR
@blaginin has fixed it -- it appears to have been a logical conflict
--
This is an automated message from the Apache Git Service.
To
xudong963 commented on PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846060139
Fyi, the main CI has failed since the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
alamb merged PR #15168:
URL: https://github.com/apache/datafusion/pull/15168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusi
alamb commented on PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846025123
I have filed an epic to track filling out the datafusion-spark crate:
- https://github.com/apache/datafusion/issues/15914
I will file some subtickets for follow on work as we
alamb commented on PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846025618
Onward!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubs
alamb commented on PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2840589691
This looks great to me -- I plan to merge it tomorrow and start collecting
next steps in a new `EPIC` ticket unless someone beats me to it
--
This is an automated message from the Ap
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2065542062
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor lic
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2065539727
##
datafusion/spark/src/function/string/ascii.rs:
##
@@ -0,0 +1,210 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2065539353
##
datafusion/sqllogictest/src/engines/conversion.rs:
##
@@ -77,7 +77,21 @@ pub(crate) fn f64_to_str(value: f64) -> String {
} else if value == f64::NEG_I
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2065538751
##
datafusion/spark/src/lib.rs:
##
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2065538632
##
Cargo.lock:
##
@@ -2558,6 +2558,27 @@ dependencies = [
"tokio",
]
+[[package]]
+name = "datafusion-spark"
+version = "47.0.0"
+dependencies = [
+ "arro
alamb commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2065131934
##
datafusion/sqllogictest/test_files/spark/README.md:
##
@@ -0,0 +1,57 @@
+
+
+# Spark Test Files
+
+This directory contains test files for the `spark` test suite.
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2061232126
##
datafusion/sqllogictest/src/engines/datafusion_engine/normalize.rs:
##
@@ -193,7 +192,7 @@ macro_rules! get_row_value {
///
/// Floating numbers are round
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2051099886
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor licen
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2038609229
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor lic
alamb commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2008804385
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license a
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2008444729
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor lic
alamb commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2004518126
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license a
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r2002067067
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor lic
alamb commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1997557845
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor
alamb commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1997557995
##
datafusion/sqllogictest/test_files/spark/string/ascii.slt:
##
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributo
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1995461595
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994892781
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994917169
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994946916
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994929734
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994884151
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994863716
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994823297
##
datafusion/core/src/execution/session_state_defaults.rs:
##
@@ -104,28 +104,55 @@ impl SessionStateDefaults {
/// returns the list of default [`Scala
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994466480
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994471938
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994452292
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994474461
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994472992
##
datafusion/core/src/execution/session_state_defaults.rs:
##
@@ -104,28 +104,55 @@ impl SessionStateDefaults {
/// returns the list of default [`ScalarU
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994457478
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994457586
##
datafusion/sqllogictest/test_files/spark/string/ascii.slt:
##
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994469174
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994457586
##
datafusion/sqllogictest/test_files/spark/string/ascii.slt:
##
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994461973
##
datafusion/sqllogictest/test_files/spark/string/ascii.slt:
##
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more cont
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994461775
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994461511
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994455591
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
andygrove commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994448064
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contribu
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992573548
##
datafusion/spark/src/function/string/ascii.rs:
##
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992589651
##
datafusion/spark/src/function/string/ascii.rs:
##
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992589651
##
datafusion/spark/src/function/string/ascii.rs:
##
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992592514
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor lic
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992574619
##
datafusion/spark/src/function/string/ascii.rs:
##
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992575249
##
datafusion/spark/src/function/math/expm1.rs:
##
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor lic
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992574619
##
datafusion/spark/src/function/string/ascii.rs:
##
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992573548
##
datafusion/spark/src/function/string/ascii.rs:
##
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor l
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992569405
##
datafusion/sqllogictest/test_files/spark/README.md:
##
@@ -0,0 +1,57 @@
+
+
Review Comment:
Testing instructions here
--
This is an automated messag
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992569901
##
datafusion/sqllogictest/test_files/spark/math/expm1.slt:
##
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contri
shehabgamin commented on code in PR #15168:
URL: https://github.com/apache/datafusion/pull/15168#discussion_r1992569624
##
datafusion/spark/README.md:
##
@@ -0,0 +1,38 @@
+
+
+# datafusion-spark: Spark-compatible Expressions
+
+This crate provides Apache Spark-compatible express
shehabgamin opened a new pull request, #15168:
URL: https://github.com/apache/datafusion/pull/15168
## Which issue does this PR close?
Closes https://github.com/apache/datafusion/issues/5600
## Rationale for this change
See discussion in https://github.com/apache/datafusi
andygrove closed pull request #14392: feat: Add `datafusion-spark` crate
URL: https://github.com/apache/datafusion/pull/14392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
shehabgamin commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2708820627
> > > Would you have bandwidth to help with adding some initial tests in the
Comet repo? I think if we have some examples then it will be easier for others
to contribute.
> >
alamb commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2703660394
> > Would you have bandwidth to help with adding some initial tests in the
Comet repo? I think if we have some examples then it will be easier for others
to contribute.
>
> Yeah
shehabgamin commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2703064824
> Would you have bandwidth to help with adding some initial tests in the
Comet repo? I think if we have some examples then it will be easier for others
to contribute.
Yeah
andygrove commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2701788755
> @andygrove Just checking in. I've been adding tons of new functions to the
Sail repo but will port them over after this PR is merged!
Sounds good!
The consensus was
shehabgamin commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2686793279
@andygrove Just checking in. I've been adding tons of new functions to the
Sail repo but will port them over after this PR is merged!
--
This is an automated message from the A
andygrove commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2637613868
I filed an issue in Comet to do the necessary work (testing and examples) to
prepare to move the crate into DataFusion repo.
https://github.com/apache/datafusion-comet/issues
alamb commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2637338434
> All functions that Sail ports over can be tested without the JVM or Spark
Client. Because there is no JVM involvement when running a Sail server, it
would be a relatively straightfor
alamb commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2637335904
> @andygrove @alamb maybe we can do a conditional compilation on function
level instead of separate crate? Sort of introduce a feature spark and have
different implementations
>
Blizzara commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2630634253
I'd be very happy to see this go in :) Having the functions here in main
datafusion repo makes dealing with any breaks and downstream upgrades easier
(we'll only need to track/match
andygrove commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629666721
I'm going to move this to draft for now. Let's keep the conversation going,
though.
I plan on working with the community to start adding tests to the
`datafusion-comet-spark
comphead commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629508048
That is exactly what I was mentioning
https://github.com/apache/datafusion/pull/14392#issuecomment-2628334890 not
sure how to separate out functions with the same names. I think thi
andygrove commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629456979
> For Comet, we need the function names to match Spark.
Actually, this isn't true. In Comet, we just need the Scala wrapper classes
to have the same function names as Spark.
andygrove commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629435886
> @comphead @andygrove @alamb It may be a good idea to prefix all function
names with `spark_` to avoid confusion, conflicts, or unknown behavior between
functions that share the s
shehabgamin commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629317132
> If the behavior is the same we don't have any reason to copy one to spark
crate, adding alias to the function is enough.
@jayzhan211 I’ll need to verify this, but I reca
jayzhan211 commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629310335
> > In optimizer, we rely on the name to do such optimization so if we
rename it to name like 'spark_count' we might need to add the spark name to
those optimize rules as well, wh
shehabgamin commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629306233
> In optimizer, we rely on the name to do such optimization so if we rename
it to name like 'spark_count' we might need to add the spark name to those
optimize rules as well, whi
jayzhan211 commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629290634
> It may be a good idea to prefix all function names with spark_ to avoid
confusion, conflicts, or unknown behavior between functions that share the same
name.
In optimizer
shehabgamin commented on code in PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#discussion_r1938420167
##
datafusion/spark/src/agg_funcs/avg.rs:
##
@@ -0,0 +1,344 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license a
shehabgamin commented on PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2629161924
> cc @Blizzara and @shehabgamin as you have mentioned interest in helping
here
>
Will catch up on this thread tonight. So much happening so fast, exciting!
--
Th
andygrove commented on code in PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#discussion_r1938313126
##
datafusion/spark/src/agg_funcs/avg.rs:
##
@@ -0,0 +1,344 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agr
andygrove commented on code in PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#discussion_r1938313058
##
datafusion/spark/src/agg_funcs/avg.rs:
##
@@ -0,0 +1,344 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agr
alamb commented on code in PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#discussion_r1938312603
##
datafusion/spark/src/comet_scalar_funcs.rs:
##
@@ -0,0 +1,192 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license ag
alamb commented on code in PR #14392:
URL: https://github.com/apache/datafusion/pull/14392#discussion_r1938312400
##
datafusion/spark/src/agg_funcs/avg.rs:
##
@@ -0,0 +1,344 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreeme
79 matches
Mail list logo
