Re: [I] [EPIC] Decouple logical from physical types [datafusion]

2025-01-17 Thread via GitHub
jayzhan211 commented on issue #12622: URL: https://github.com/apache/datafusion/issues/12622#issuecomment-2599579247 > Unfortunately, using Scalar in ColumnarValue and Expr breaks all patterns that try to match the scalar for these types. While we can fix this in the DataFusion code base

Re: [PR] Add a hint about expected extension in error message in register_csv,… [datafusion]

2025-01-17 Thread via GitHub
cj-zhukov commented on code in PR #14168: URL: https://github.com/apache/datafusion/pull/14168#discussion_r1920985877 ## datafusion/core/src/execution/context/mod.rs: ## @@ -1379,6 +1379,29 @@ impl SessionContext { Ok(()) } +fn register_type_check( +&

Re: [I] Spaceship operator (<=>) not supported [datafusion]

2025-01-17 Thread via GitHub
Spaarsh commented on issue #14098: URL: https://github.com/apache/datafusion/issues/14098#issuecomment-2599524998 Apologies for the delay! @ion-elgreco is this the output you are expecting? If so, I shall make a PR including some tests as well! ![Image](https://github.com/user-attachment

Re: [I] Release DataFusion `45.0.0` [datafusion]

2025-01-17 Thread via GitHub
shehabgamin commented on issue #14008: URL: https://github.com/apache/datafusion/issues/14008#issuecomment-2599492511 As promised, Sail is working on porting relevant tests into DataFusion. A good starting point is a regression our tests caught in DataFusion 43, which still seems to p

Re: [PR] fix: partially fix consistency issue of hash functions with decimal input [datafusion-comet]

2025-01-17 Thread via GitHub
wForget commented on code in PR #1295: URL: https://github.com/apache/datafusion-comet/pull/1295#discussion_r1920937037 ## native/spark-expr/src/hash_funcs/utils.rs: ## @@ -104,6 +104,26 @@ macro_rules! hash_array_primitive_float { }; } +#[macro_export] +macro_rules! has

Re: [PR] fix: partially fix consistency issue of hash functions with decimal input [datafusion-comet]

2025-01-17 Thread via GitHub
wForget commented on code in PR #1295: URL: https://github.com/apache/datafusion-comet/pull/1295#discussion_r1920937402 ## native/spark-expr/src/hash_funcs/utils.rs: ## @@ -104,6 +104,26 @@ macro_rules! hash_array_primitive_float { }; } +#[macro_export] +macro_rules! has

Re: [PR] fix: partially fix consistency issue of hash functions with decimal input [datafusion-comet]

2025-01-17 Thread via GitHub
wForget commented on code in PR #1295: URL: https://github.com/apache/datafusion-comet/pull/1295#discussion_r1920937037 ## native/spark-expr/src/hash_funcs/utils.rs: ## @@ -104,6 +104,26 @@ macro_rules! hash_array_primitive_float { }; } +#[macro_export] +macro_rules! has

Re: [PR] chore: [comet-parquet-exec] merge from main 20240116 [datafusion-comet]

2025-01-17 Thread via GitHub
parthchandra commented on PR #1299: URL: https://github.com/apache/datafusion-comet/pull/1299#issuecomment-2599460209 @andygrove I added another fix since you approved. This was also a merge issue and caused 6 test failures if native_datafusion was enabled. Latest count : ``` nat

Re: [I] Error when use `user` field in where caluse [datafusion]

2025-01-17 Thread via GitHub
chenkovsky commented on issue #14141: URL: https://github.com/apache/datafusion/issues/14141#issuecomment-2599457794 > > I try to make genericdialect more customizable, but it seems that it maybe not a good idea. ಥ_ಥ > > Note there is: https://datafusion.apache.org/user-guide/configs.

Re: [I] Comet possibly preventing AQE optimization [datafusion-comet]

2025-01-17 Thread via GitHub
parthchandra commented on issue #1266: URL: https://github.com/apache/datafusion-comet/issues/1266#issuecomment-2599442763 Is there some particular version of Spark that this failure is occurring with? -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Add a hint about expected extension in error message in register_csv,… [datafusion]

2025-01-17 Thread via GitHub
comphead commented on code in PR #14168: URL: https://github.com/apache/datafusion/pull/14168#discussion_r1920874016 ## datafusion/core/src/execution/context/mod.rs: ## @@ -1379,6 +1379,29 @@ impl SessionContext { Ok(()) } +fn register_type_check( +&s

Re: [I] Automate updating sqllogictest updates [datafusion]

2025-01-17 Thread via GitHub
Omega359 commented on issue #14158: URL: https://github.com/apache/datafusion/issues/14158#issuecomment-2599372511 First cut: https://gist.github.com/Omega359/fd30f1e2807f08033c65957b5233fd0b -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] fix: speed up ConcurrentHashMap#computeIfAbsent of JDK8 [datafusion-comet]

2025-01-17 Thread via GitHub
parthchandra commented on PR #1245: URL: https://github.com/apache/datafusion-comet/pull/1245#issuecomment-2599368441 > @mbutrovich, is it better to close this pull request and reopen if needed? Personally, I think so. Also, do we want to continue supporting JDK8 for long (FWIW,

Re: [PR] Add `ScalarValue::try_as_str` to get str value from logical strings [datafusion]

2025-01-17 Thread via GitHub
wiedld commented on code in PR #14167: URL: https://github.com/apache/datafusion/pull/14167#discussion_r1920831004 ## datafusion/common/src/scalar/mod.rs: ## @@ -2849,6 +2849,50 @@ impl ScalarValue { ScalarValue::from(value).cast_to(target_type) } +/// Return

Re: [PR] Add `ScalarValue::try_as_str` to get str value from logical strings [datafusion]

2025-01-17 Thread via GitHub
wiedld commented on code in PR #14167: URL: https://github.com/apache/datafusion/pull/14167#discussion_r1920830750 ## datafusion/common/src/scalar/mod.rs: ## @@ -2849,6 +2849,50 @@ impl ScalarValue { ScalarValue::from(value).cast_to(target_type) } +/// Return

Re: [PR] Add `ScalarValue::try_as_str` to get str value from logical strings [datafusion]

2025-01-17 Thread via GitHub
wiedld commented on code in PR #14167: URL: https://github.com/apache/datafusion/pull/14167#discussion_r1920830750 ## datafusion/common/src/scalar/mod.rs: ## @@ -2849,6 +2849,50 @@ impl ScalarValue { ScalarValue::from(value).cast_to(target_type) } +/// Return

[I] Prepare for DataFusion 45 [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove opened a new issue, #1304: URL: https://github.com/apache/datafusion-comet/issues/1304 ### What is the problem the feature request solves? We should start tracking issues that may affect us related to DF 45. - https://github.com/apache/datafusion/pull/14094 ###

Re: [PR] Add benchmark for planning sorted unions [datafusion]

2025-01-17 Thread via GitHub
comphead commented on code in PR #14157: URL: https://github.com/apache/datafusion/pull/14157#discussion_r1920810030 ## datafusion/core/benches/sql_planner.rs: ## @@ -147,6 +149,77 @@ fn benchmark_with_param_values_many_columns(ctx: &SessionContext, b: &mut Benche }); }

Re: [PR] Make `LexOrdering::inner` non pub, add comments, update usages [datafusion]

2025-01-17 Thread via GitHub
comphead merged PR #14155: URL: https://github.com/apache/datafusion/pull/14155 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Add benchmark for planning sorted unions [datafusion]

2025-01-17 Thread via GitHub
comphead commented on code in PR #14157: URL: https://github.com/apache/datafusion/pull/14157#discussion_r1920808402 ## datafusion/core/benches/sql_planner.rs: ## @@ -147,6 +149,77 @@ fn benchmark_with_param_values_many_columns(ctx: &SessionContext, b: &mut Benche }); }

Re: [PR] Minor: Simplify Bloom Filter Check [datafusion]

2025-01-17 Thread via GitHub
comphead merged PR #14165: URL: https://github.com/apache/datafusion/pull/14165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] fix: add support for Decimal128 and Decimal256 types in interval arithmetic [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #14126: URL: https://github.com/apache/datafusion/pull/14126#discussion_r1920803813 ## datafusion/expr-common/src/interval_arithmetic.rs: ## @@ -76,6 +76,14 @@ macro_rules! get_extreme_value { DataType::Interval(IntervalUnit::MonthDayNa

Re: [PR] chore: [comet-parquet-exec] merge from main 20240116 [datafusion-comet]

2025-01-17 Thread via GitHub
parthchandra commented on code in PR #1299: URL: https://github.com/apache/datafusion-comet/pull/1299#discussion_r1920791563 ## spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala: ## @@ -190,7 +190,7 @@ class CometSparkSessionExtensions // data

Re: [I] Deprecate `datafusion.execution.parquet.max_statistics_size` config option [datafusion]

2025-01-17 Thread via GitHub
logan-keede commented on issue #14172: URL: https://github.com/apache/datafusion/issues/14172#issuecomment-2599235975 Hi @alamb, Please check if my PR #14175 correctly addresses this issue Thanks, Logan -- This is an automated message from the Apache Git Service. To respond to the m

[PR] deprecate max_statistics_size [datafusion]

2025-01-17 Thread via GitHub
logan-keede opened a new pull request, #14175: URL: https://github.com/apache/datafusion/pull/14175 ## Which issue does this PR close? Closes #14172 ## Rationale for this change Mark `max_size_statistics` deprecated ## What changes are included in this PR?

Re: [PR] minor: update compatibility [datafusion-comet]

2025-01-17 Thread via GitHub
kazuyukitanimura commented on PR #1303: URL: https://github.com/apache/datafusion-comet/pull/1303#issuecomment-2599101337 merged, thanks @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] minor: update compatibility [datafusion-comet]

2025-01-17 Thread via GitHub
kazuyukitanimura merged PR #1303: URL: https://github.com/apache/datafusion-comet/pull/1303 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsub

Re: [PR] Comet 0.5.0 blog post [datafusion-site]

2025-01-17 Thread via GitHub
andygrove merged PR #52: URL: https://github.com/apache/datafusion-site/pull/52 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] Find a safe alternative to `LogicalPlan::using_columns()` [datafusion]

2025-01-17 Thread via GitHub
logan-keede commented on issue #14118: URL: https://github.com/apache/datafusion/issues/14118#issuecomment-2599032845 I thought so, I looked at some other avenues, Here are some things that might help anyone trying to solve this in future. Problem:- `exclude_using_columns` return

Re: [PR] chore: [comet-parquet-exec] merge from main 20240116 [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove commented on code in PR #1299: URL: https://github.com/apache/datafusion-comet/pull/1299#discussion_r1920620850 ## spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala: ## @@ -190,7 +190,7 @@ class CometSparkSessionExtensions // data sou

Re: [PR] Comet 0.5.0 blog post [datafusion-site]

2025-01-17 Thread via GitHub
andygrove commented on PR #52: URL: https://github.com/apache/datafusion-site/pull/52#issuecomment-2598971562 @viirya @comphead @alamb Could I get an approval to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Upgrade arrow-rs, parquet to `54.0.0` and pyo3 to `0.23.3` [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #14153: URL: https://github.com/apache/datafusion/pull/14153#discussion_r1920591420 ## datafusion/proto/tests/cases/roundtrip_logical_plan.rs: ## @@ -1815,6 +1815,8 @@ fn round_trip_datatype() { } } +// TODO file a ticket about handling depr

[PR] Minor add ticket references to deprecated code [datafusion]

2025-01-17 Thread via GitHub
alamb opened a new pull request, #14174: URL: https://github.com/apache/datafusion/pull/14174 ## Which issue does this PR close? Follow on to - https://github.com/apache/datafusion/pull/14153 ## Rationale for this change I filed a ticket to track the removal of some de

[I] Remove use of deprecated `dict_id` in datafusion-proto [datafusion]

2025-01-17 Thread via GitHub
alamb opened a new issue, #14173: URL: https://github.com/apache/datafusion/issues/14173 ### Is your feature request related to a problem or challenge? The most recent version of arrow deprecates the dictionary fields from Schema, see - https://github.com/apache/arrow-rs/pull/6873

[I] Deprecate `datafusion.execution.parquet.max_statistics_size` config option [datafusion]

2025-01-17 Thread via GitHub
alamb opened a new issue, #14172: URL: https://github.com/apache/datafusion/issues/14172 ### Is your feature request related to a problem or challenge? The most recent version of arrow deprecates the `max_statistics_size` parquet option - https://github.com/apache/datafusion/pull/1

[PR] minor: update compatibility [datafusion-comet]

2025-01-17 Thread via GitHub
kazuyukitanimura opened a new pull request, #1303: URL: https://github.com/apache/datafusion-comet/pull/1303 Leftover from https://github.com/apache/datafusion-comet/pull/1281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] feat: Support `Utf8View` for `get_wider_type` + `binary_to_string_coercion` functions [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #13370: URL: https://github.com/apache/datafusion/pull/13370#issuecomment-2598934705 We just updated to a version of arrow-rs that I think has the necessary changes: - https://github.com/apache/datafusion/pull/14153 (thanks @Owen-CH-Leung ) -- This is an automa

Re: [PR] WIP Upgrade to arrow-rs/parquet `54.0.0` [datafusion]

2025-01-17 Thread via GitHub
alamb closed pull request #13663: WIP Upgrade to arrow-rs/parquet `54.0.0` URL: https://github.com/apache/datafusion/pull/13663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Upgrade arrow-rs, parquet to `54.0.0` and pyo3 to `0.23.3` [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14153: URL: https://github.com/apache/datafusion/pull/14153#issuecomment-2598933647 Thanks again @Owen-CH-Leung -- very much apprecaited -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] Upgrade to arrow-rs/parquet `54.0.0` [datafusion]

2025-01-17 Thread via GitHub
alamb closed issue #14114: Upgrade to arrow-rs/parquet `54.0.0` URL: https://github.com/apache/datafusion/issues/14114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Upgrade arrow-rs, parquet to `54.0.0` and pyo3 to `0.23.3` [datafusion]

2025-01-17 Thread via GitHub
alamb merged PR #14153: URL: https://github.com/apache/datafusion/pull/14153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Add recursion limit configuration to `DFParser` [datafusion]

2025-01-17 Thread via GitHub
alamb closed pull request #14095: Add recursion limit configuration to `DFParser` URL: https://github.com/apache/datafusion/pull/14095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] fix: partially fix consistency issue of hash functions with decimal input [datafusion-comet]

2025-01-17 Thread via GitHub
kazuyukitanimura commented on code in PR #1295: URL: https://github.com/apache/datafusion-comet/pull/1295#discussion_r1920575456 ## native/spark-expr/src/hash_funcs/utils.rs: ## @@ -104,6 +104,26 @@ macro_rules! hash_array_primitive_float { }; } +#[macro_export] +macro_r

Re: [PR] Add recursion limit configuration to `DFParser` [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14095: URL: https://github.com/apache/datafusion/pull/14095#issuecomment-2598932383 To be clear I think we will still need to increase the recursion limit in DataFusion to allow the sqllogictest to pass. But until the next sqlparser upgrade I don't we can do so safely

Re: [PR] Add recursion limit configuration to `DFParser` [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14095: URL: https://github.com/apache/datafusion/pull/14095#issuecomment-2598930685 Converting this to draft -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] fix: handle scalar predicates in CASE expressions to prevent internal errors for InfallibleExprOrNull eval method [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14156: URL: https://github.com/apache/datafusion/pull/14156#issuecomment-2598926902 Thank you @Omega359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Always add round robin repartitioning to leaves (data sources), benefitting unbalanced / small datasets [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #13707: URL: https://github.com/apache/datafusion/pull/13707#issuecomment-2598924731 Marking this PR as a draft as it has some conflicts now and it doesn't seem like we have consensus on the approach (some will be faster and some slower). Let me know if this isn't righ

Re: [PR] Making the data_imdb and clickbench_1 functions atomic. [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #14129: URL: https://github.com/apache/datafusion/pull/14129#discussion_r1920568890 ## benchmarks/bench.sh: ## @@ -401,9 +405,14 @@ data_clickbench_1() { else URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet";

[PR] chore: [comet-parquet-exec] enable native scan [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove opened a new pull request, #1302: URL: https://github.com/apache/datafusion-comet/pull/1302 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] fix: handle scalar predicates in CASE expressions to prevent internal errors for InfallibleExprOrNull eval method [datafusion]

2025-01-17 Thread via GitHub
Omega359 commented on PR #14156: URL: https://github.com/apache/datafusion/pull/14156#issuecomment-2598879890 Status update: I've rerun the slt generation locally, I just need to apply patches for 2 failures, have a look over the updates, then push a PR. Hopefully by EOD today. -- This i

Re: [I] doc-gen: Migrate builtin scalar functions from code based documentation to attribute based [datafusion]

2025-01-17 Thread via GitHub
alamb commented on issue #13671: URL: https://github.com/apache/datafusion/issues/13671#issuecomment-2598868919 Nice work everyone and thank you @Chen-Yuan-Lai especially 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Make `LexOrdering::inner` non pub, add comments, update usages [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14155: URL: https://github.com/apache/datafusion/pull/14155#issuecomment-2598871191 > LGTM, thank you @alamb. After your work on these equivalence API's, they seem more clear and easy to use now. Thank you -- now I just need to make them faster / more efficient

Re: [PR] IGNORE: chore: Merge comet-parquet-exec into (just to see diff) [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove closed pull request #1296: IGNORE: chore: Merge comet-parquet-exec into (just to see diff) URL: https://github.com/apache/datafusion-comet/pull/1296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Error when use `user` field in where caluse [datafusion]

2025-01-17 Thread via GitHub
alamb commented on issue #14141: URL: https://github.com/apache/datafusion/issues/14141#issuecomment-2598865518 > I try to make genericdialect more customizable, but it seems that it maybe not a good idea. ಥ_ಥ Note there is: https://datafusion.apache.org/user-guide/configs.html

Re: [PR] Enable native scan [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove closed pull request #1301: Enable native scan URL: https://github.com/apache/datafusion-comet/pull/1301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Enable native scan [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove closed pull request #1301: Enable native scan URL: https://github.com/apache/datafusion-comet/pull/1301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[PR] Enable native scan [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove opened a new pull request, #1301: URL: https://github.com/apache/datafusion-comet/pull/1301 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #14094: URL: https://github.com/apache/datafusion/pull/14094#discussion_r1920358041 ## datafusion/expr/src/udf.rs: ## @@ -342,6 +348,56 @@ pub struct ScalarFunctionArgs<'a> { pub return_type: &'a DataType, } +#[derive(Debug)] +pub struct Ret

Re: [PR] Introduce `return_type_from_args ` for ScalarFunction. [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14094: URL: https://github.com/apache/datafusion/pull/14094#issuecomment-2598845894 FYI @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[PR] Chore: Add UDF documentation guide [datafusion]

2025-01-17 Thread via GitHub
xarus01 opened a new pull request, #14171: URL: https://github.com/apache/datafusion/pull/14171 ## Which issue does this PR close? Closes #8366. ## Rationale for this change ## What changes are included in this PR? Added `How to add/edit doc

[I] Parse MySQL `ALGORITHM` and `LOCK` options to `ALTER TABLE` [datafusion-sqlparser-rs]

2025-01-17 Thread via GitHub
mvzink opened a new issue, #1665: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1665 Example: ```sql ALTER TABLE tbl_name MODIFY COLUMN col_name column_definition FIRST, ALGORITHM=INPLACE, LOCK=NONE; ``` See [`ALTER TABLE` docs](https://dev.mysql.com/doc/

Re: [PR] chore: [comet-parquet-exec] Add repro for zip rdd issue [datafusion-comet]

2025-01-17 Thread via GitHub
viirya commented on PR #1298: URL: https://github.com/apache/datafusion-comet/pull/1298#issuecomment-2598796942 > @viirya, I wonder if you could help me and @parthchandra understand why the test in this PR is failing. I may be able to look at this only after I get a new laptop...

Re: [PR] chore: fix flaky tests [datafusion]

2025-01-17 Thread via GitHub
xudong963 merged PR #14170: URL: https://github.com/apache/datafusion/pull/14170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] doc-gen: migrate scalar functions (other, conditional, and struct) documentation [datafusion]

2025-01-17 Thread via GitHub
comphead merged PR #14163: URL: https://github.com/apache/datafusion/pull/14163 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] doc-gen: Migrate builtin scalar functions from code based documentation to attribute based [datafusion]

2025-01-17 Thread via GitHub
comphead closed issue #13671: doc-gen: Migrate builtin scalar functions from code based documentation to attribute based URL: https://github.com/apache/datafusion/issues/13671 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] doc-gen: migrate scalar functions (other, conditional, and struct) documentation [datafusion]

2025-01-17 Thread via GitHub
comphead commented on PR #14163: URL: https://github.com/apache/datafusion/pull/14163#issuecomment-2598761699 Thanks everyone, looks like these was last functions to migrate 🤖 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[PR] Comet 0.5.0 blog post [datafusion-site]

2025-01-17 Thread via GitHub
andygrove opened a new pull request, #52: URL: https://github.com/apache/datafusion-site/pull/52 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Add `ScalarValue::try_as_str` to get str value from logical strings [datafusion]

2025-01-17 Thread via GitHub
xudong963 commented on code in PR #14167: URL: https://github.com/apache/datafusion/pull/14167#discussion_r1920411199 ## datafusion/common/src/scalar/mod.rs: ## @@ -2849,6 +2849,50 @@ impl ScalarValue { ScalarValue::from(value).cast_to(target_type) } +/// Ret

[PR] chore: fix flaky tests [datafusion]

2025-01-17 Thread via GitHub
xudong963 opened a new pull request, #14170: URL: https://github.com/apache/datafusion/pull/14170 ## Rationale for this change I noticed the two tests would be unstable somethings. ## What changes are included in this PR? Add `Order by` to make them stable

Re: [PR] fix: add support for Decimal128 and Decimal256 types in interval arithmetic [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #14126: URL: https://github.com/apache/datafusion/pull/14126#discussion_r1920352694 ## datafusion/expr-common/src/interval_arithmetic.rs: ## @@ -76,6 +76,14 @@ macro_rules! get_extreme_value { DataType::Interval(IntervalUnit::MonthDayNa

Re: [PR] fix: add support for Decimal128 and Decimal256 types in interval arithmetic [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14126: URL: https://github.com/apache/datafusion/pull/14126#issuecomment-2598600811 Marking as draft as I think this PR is no longer waiting on feedback. Please mark it as ready for review when it is ready for another look I think we just need to tweak the max

Re: [PR] chore: [comet-parquet-exec] Add repro for zip rdd issue [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove commented on PR #1298: URL: https://github.com/apache/datafusion-comet/pull/1298#issuecomment-2598596742 I have this test passing now, but I don't know if the approach is correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] Minor: Consolidate dataframe tests into core_integration [datafusion]

2025-01-17 Thread via GitHub
alamb opened a new pull request, #14169: URL: https://github.com/apache/datafusion/pull/14169 ## Which issue does this PR close? Closes #. ## Rationale for this change I think it would be best to consolidate most "end user API tests" into a single file so they ar

Re: [PR] chore: [comet-parquet-exec] Add repro for zip rdd issue [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove commented on PR #1298: URL: https://github.com/apache/datafusion-comet/pull/1298#issuecomment-2598563070 After more debugging, I understand what is happening, but not sure what the correct fix is yet. We set `firstNonBroadcastPlanNumPartitions` to `0` because that is what

Re: [PR] chore: [comet-parquet-exec] Add repro for zip rdd issue [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove commented on PR #1298: URL: https://github.com/apache/datafusion-comet/pull/1298#issuecomment-2598544621 More debug output showing the RDDs being zipped, with partition count: ``` zipping these RDDS: UnionRDD parts = 10 CometBatchRDD parts = 0 ``` --

[I] Comet 0.5.1 Release [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove opened a new issue, #1300: URL: https://github.com/apache/datafusion-comet/issues/1300 ### What is the problem the feature request solves? Major releases are currently happening every 1-2 months, so we may want to create a patch release at some point with bug fixes. T

Re: [I] Unify SQL planning for `ORDER BY`, `HAVING`, `DISTINCT`, etc [datafusion]

2025-01-17 Thread via GitHub
jonahgao commented on issue #10326: URL: https://github.com/apache/datafusion/issues/10326#issuecomment-2598506302 I haven't started working on it yet. Feel free to take it. Thanks @lichuang -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] chore: [comet-parquet-exec] Add repro for zip rdd issue [datafusion-comet]

2025-01-17 Thread via GitHub
andygrove commented on PR #1298: URL: https://github.com/apache/datafusion-comet/pull/1298#issuecomment-2598503895 @viirya, I wonder if you could help me and @parthchandra understand why the test in this PR is failing. The error is: ``` Can't zip RDDs with unequal numbers

Re: [PR] Add supports for Hive's `SELECT ... GROUP BY .. GROUPING SETS` syntax [datafusion-sqlparser-rs]

2025-01-17 Thread via GitHub
wugeer commented on code in PR #1653: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1653#discussion_r1920283018 ## src/ast/query.rs: ## @@ -328,6 +328,10 @@ pub struct Select { pub value_table_mode: Option, /// STARTING WITH .. CONNECT BY pub connec

Re: [PR] fix: add support for Decimal128 and Decimal256 types in interval arithmetic [datafusion]

2025-01-17 Thread via GitHub
waynexia commented on code in PR #14126: URL: https://github.com/apache/datafusion/pull/14126#discussion_r1920271011 ## datafusion/expr-common/src/interval_arithmetic.rs: ## @@ -76,6 +76,14 @@ macro_rules! get_extreme_value { DataType::Interval(IntervalUnit::MonthDa

Re: [I] Unify SQL planning for `ORDER BY`, `HAVING`, `DISTINCT`, etc [datafusion]

2025-01-17 Thread via GitHub
alamb commented on issue #10326: URL: https://github.com/apache/datafusion/issues/10326#issuecomment-2598457665 👋 @lichuang -- I don't know anyone working on it, but maybe @jonahgao knows of something. THank you! -- This is an automated message from the Apache Git Service. To re

Re: [PR] fix: add support for Decimal128 and Decimal256 types in interval arithmetic [datafusion]

2025-01-17 Thread via GitHub
waynexia commented on code in PR #14126: URL: https://github.com/apache/datafusion/pull/14126#discussion_r1920264150 ## datafusion/expr-common/src/interval_arithmetic.rs: ## @@ -76,6 +76,14 @@ macro_rules! get_extreme_value { DataType::Interval(IntervalUnit::MonthDa

Re: [PR] Add related source code locations to errors [datafusion]

2025-01-17 Thread via GitHub
eliaperantoni commented on PR #13664: URL: https://github.com/apache/datafusion/pull/13664#issuecomment-2598471076 @alamb I'm very sorry for the delay. I think your points about not wanting to change the logical types are valid, and I confess it was a bit of a pain to make all the changes t

Re: [PR] Remove dependency on physical-optimizer on functions-aggregates [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #14134: URL: https://github.com/apache/datafusion/pull/14134#discussion_r1920255614 ## datafusion/expr/src/test/function_stub.rs: ## @@ -252,6 +252,10 @@ impl AggregateUDFImpl for Count { Ok(DataType::Int64) } +fn is_nullable(&se

Re: [PR] Remove dependency on physical-optimizer on functions-aggregates [datafusion]

2025-01-17 Thread via GitHub
alamb merged PR #14134: URL: https://github.com/apache/datafusion/pull/14134 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] fix: add support for Decimal128 and Decimal256 types in interval arithmetic [datafusion]

2025-01-17 Thread via GitHub
waynexia commented on code in PR #14126: URL: https://github.com/apache/datafusion/pull/14126#discussion_r1920254738 ## datafusion/expr-common/src/interval_arithmetic.rs: ## @@ -76,6 +76,14 @@ macro_rules! get_extreme_value { DataType::Interval(IntervalUnit::MonthDa

Re: [PR] Remove dependency on physical-optimizer on functions-aggregates [datafusion]

2025-01-17 Thread via GitHub
alamb commented on PR #14134: URL: https://github.com/apache/datafusion/pull/14134#issuecomment-2598460363 Thank you for the review @jonahgao 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[PR] Add a hint about expected extension in error message in register_csv,… [datafusion]

2025-01-17 Thread via GitHub
cj-zhukov opened a new pull request, #14168: URL: https://github.com/apache/datafusion/pull/14168 … register_parquet, register_json, register_avro (#14144) ## Which issue does this PR close? Closes #14144. ## Rationale for this change ## What change

Re: [PR] Implement predicate pruning for `like` expressions (prefix matching) [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #12978: URL: https://github.com/apache/datafusion/pull/12978#discussion_r1920211318 ## datafusion/physical-optimizer/src/pruning.rs: ## @@ -1605,7 +1627,129 @@ fn build_statistics_expr( Ok(statistics_expr) } +/// Convert `column LIKE literal

Re: [PR] Add `ScalarValue::try_as_str` to get str value from logical strings [datafusion]

2025-01-17 Thread via GitHub
alamb commented on code in PR #14167: URL: https://github.com/apache/datafusion/pull/14167#discussion_r1920205796 ## datafusion/common/src/scalar/mod.rs: ## @@ -2849,6 +2849,50 @@ impl ScalarValue { ScalarValue::from(value).cast_to(target_type) } +/// Returns

[PR] Add `ScalarValue::try_as_str` to get str value from logical strings [datafusion]

2025-01-17 Thread via GitHub
alamb opened a new pull request, #14167: URL: https://github.com/apache/datafusion/pull/14167 ## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/14166 ## Rationale for this change See https://github.com/apache/datafusion/issues/141

[I] Add a way to access logical String `ScalarValue`s as `&str` [datafusion]

2025-01-17 Thread via GitHub
alamb opened a new issue, #14166: URL: https://github.com/apache/datafusion/issues/14166 ### Is your feature request related to a problem or challenge? We hit a bug upstream in InfluxDB in pruning where we didn't match a certain kind of string ScalarValue In this case , we want

Re: [I] Fix doctests in ScalarValue [datafusion]

2025-01-17 Thread via GitHub
cj-zhukov commented on issue #14164: URL: https://github.com/apache/datafusion/issues/14164#issuecomment-2598368377 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Unify SQL planning for `ORDER BY`, `HAVING`, `DISTINCT`, etc [datafusion]

2025-01-17 Thread via GitHub
lichuang commented on issue #10326: URL: https://github.com/apache/datafusion/issues/10326#issuecomment-2598350826 Is this issue still on going? If so, let me have a try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Type Coercion fails for List with inner type struct which has large/view types [datafusion]

2025-01-17 Thread via GitHub
kosiew commented on issue #14154: URL: https://github.com/apache/datafusion/issues/14154#issuecomment-2598327221 Good catch @ion-elgreco ☝! I managed to reproduce the error after fetching the latest main from the upstream. I investigated the delta-rs repo, because I earlier tested

Re: [PR] feat: customize function keywords [datafusion-sqlparser-rs]

2025-01-17 Thread via GitHub
chenkovsky commented on PR #1662: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1662#issuecomment-2598317162 > > > > @MartinSahlen @iffyio Please review it again. I know maybe it's not a good idea to solve it in this library. > > > > > > > > > Given how I've shown t

Re: [PR] feat: customize function keywords [datafusion-sqlparser-rs]

2025-01-17 Thread via GitHub
MartinSahlen commented on PR #1662: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1662#issuecomment-2598306277 > > > @MartinSahlen @iffyio Please review it again. I know maybe it's not a good idea to solve it in this library. > > > > > > Given how I've shown that wh

[PR] Add support for qualified column names in JOIN ... USING [datafusion-sqlparser-rs]

2025-01-17 Thread via GitHub
yoavcloud opened a new pull request, #1663: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1663 It appears that Snowflake allows specifying qualified column names in the USING column list. For example: `SELECT * FROM tbl1 AS t1 JOIN tbl2 AS t2 USING(t2.col1)` -- This i

Re: [PR] feat: customize function keywords [datafusion-sqlparser-rs]

2025-01-17 Thread via GitHub
chenkovsky commented on PR #1662: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1662#issuecomment-2598286166 > > @MartinSahlen @iffyio Please review it again. I know maybe it's not a good idea to solve it in this library. > > Given how I've shown that whether this is ev

Re: [I] Error when use `user` field in where caluse [datafusion]

2025-01-17 Thread via GitHub
chenkovsky commented on issue #14141: URL: https://github.com/apache/datafusion/issues/14141#issuecomment-2598273644 I try to make genericdialect more customizable, but it seems that it maybe not a good idea. ಥ_ಥ -- This is an automated message from the Apache Git Service. To respond to t

  1   2   >