Re: [I] PhysicalOptimizer rule 'join_selection' failed [datafusion]

2025-03-11 Thread via GitHub
g3blv commented on issue #15136: URL: https://github.com/apache/datafusion/issues/15136#issuecomment-2712910534 I've tried a couple of potential workarounds without success: 1. Using explicit casting in the query: ```sql SELECT sources.id, media.title FROM sources JOIN media ON

Re: [PR] Parse MySQL `ALTER TABLE DROP FOREIGN KEY` syntax [datafusion-sqlparser-rs]

2025-03-11 Thread via GitHub
iffyio commented on code in PR #1762: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1762#discussion_r1988447089 ## src/parser/mod.rs: ## @@ -8002,6 +8002,11 @@ impl<'a> Parser<'a> { && dialect_of!(self is MySqlDialect | GenericDialect)

Re: [I] Sort query won't get round-robin repartitioned if input is `MemTable` [datafusion]

2025-03-11 Thread via GitHub
alan910127 commented on issue #15088: URL: https://github.com/apache/datafusion/issues/15088#issuecomment-2713273281 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) [datafusion]

2025-03-11 Thread via GitHub
Blizzara commented on PR #15123: URL: https://github.com/apache/datafusion/pull/15123#issuecomment-2713329924 Yeah, from my perspective this one isn't so bad - implementing any of the deprecated methods is kinda okay, since the defaults are wired to propagate the call correctly here. Callin

[PR] chore(deps): bump syn from 2.0.98 to 2.0.100 [datafusion]

2025-03-11 Thread via GitHub
dependabot[bot] opened a new pull request, #15147: URL: https://github.com/apache/datafusion/pull/15147 Bumps [syn](https://github.com/dtolnay/syn) from 2.0.98 to 2.0.100. Release notes Sourced from https://github.com/dtolnay/syn/releases";>syn's releases. 2.0.100 Add

Re: [PR] Fix broken `serde` feature [datafusion]

2025-03-11 Thread via GitHub
vadimpiven commented on PR #15124: URL: https://github.com/apache/datafusion/pull/15124#issuecomment-2713103636 Tests failed because of “no space left on device” error. Can someone please restart it? -- This is an automated message from the Apache Git Service. To respond to the message, p

[I] PartitionBuffers should not have their own MemoryConsumer [datafusion-comet]

2025-03-11 Thread via GitHub
EmilyMatt opened a new issue, #1495: URL: https://github.com/apache/datafusion-comet/issues/1495 ### Describe the bug The ShuffleRepartitioner should be the memory consumer registered with the pool, with each partitionbuffer being a reservation split off from it(using new_empty()), s

[PR] fix: #1495 [datafusion-comet]

2025-03-11 Thread via GitHub
EmilyMatt opened a new pull request, #1496: URL: https://github.com/apache/datafusion-comet/pull/1496 ## Which issue does this PR close? Closes #1495 . ## Rationale for this change It lets MemoryPools have a better grasp of the amount of operators actually runnin

Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]

2025-03-11 Thread via GitHub
alamb commented on issue #13814: URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2713604463 One thing that is strange is that we have moved most of the code out of datafusion-core now. I am still hoping we can get https://github.com/apache/datafusion/tree/main/dat

[PR] Better CI feature checks [datafusion]

2025-03-11 Thread via GitHub
alamb opened a new pull request, #15156: URL: https://github.com/apache/datafusion/pull/15156 ## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/15155 ## Rationale for this change The coverage for feature flags needs to be improved,

Re: [I] Cannot compile datafusion without feature regex_expressions [datafusion]

2025-03-11 Thread via GitHub
alamb commented on issue #5621: URL: https://github.com/apache/datafusion/issues/5621#issuecomment-2714136904 Will bo covered by https://github.com/apache/datafusion/issues/15155 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Cannot compile datafusion without feature regex_expressions [datafusion]

2025-03-11 Thread via GitHub
alamb closed issue #5621: Cannot compile datafusion without feature regex_expressions URL: https://github.com/apache/datafusion/issues/5621 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-11 Thread via GitHub
alan910127 commented on PR #15110: URL: https://github.com/apache/datafusion/pull/15110#issuecomment-2714179107 @findepi so the test cases in `unwrap_cast` are not correct, and they're showing that I'm over-unwrapping. Am I understanding this correctly? -- This is an automated message fro

Re: [I] Parametrize ListArray inner field [datafusion]

2025-03-11 Thread via GitHub
tustvold commented on issue #15162: URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2714823206 I'm not familiar with how comet interops with Spark, but it looks like whatever component is wrapping the spark execution is incorrectly exposing the schema of its outputs? Pro

Re: [I] Parametrize ListArray inner field [datafusion]

2025-03-11 Thread via GitHub
comphead commented on issue #15162: URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2714869946 Thanks @tustvold the requirement is to customize hardcoded inner field for ListType which is hardcoded `item` now. So DF and arrow-rs creates data arrays for ListType ```

Re: [PR] Benchmarking fixes: query 15, update validator, formatting, ray scheduling fix [datafusion-ray]

2025-03-11 Thread via GitHub
andygrove merged PR #83: URL: https://github.com/apache/datafusion-ray/pull/83 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafu

Re: [I] TPC-H benchmark does not run q15 [datafusion-ray]

2025-03-11 Thread via GitHub
andygrove closed issue #81: TPC-H benchmark does not run q15 URL: https://github.com/apache/datafusion-ray/issues/81 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[I] Add dictionary support for read-side padding [datafusion-comet]

2025-03-11 Thread via GitHub
andygrove opened a new issue, #1500: URL: https://github.com/apache/datafusion-comet/issues/1500 ### What is the problem the feature request solves? This issue is to track splitting out the read-side padding changes from https://github.com/apache/datafusion-comet/pull/1482/files#diff-

Re: [I] Parametrize ListArray inner field [datafusion]

2025-03-11 Thread via GitHub
tustvold commented on issue #15162: URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2714900060 > Comet using the DataFusion physical plan expressions directly, there is no coercion phase and schema from Apache Spark for the same comes as IMO this is the issue, the

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
Omega359 commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989463073 ## content/blog/2025-03-11-ordering-analysis.md: ## @@ -0,0 +1,381 @@ +--- +layout: post +title: Using Ordering for Better Plans in Apache DataFusion +date: 2025-0

Re: [PR] chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) [datafusion]

2025-03-11 Thread via GitHub
Weijun-H commented on code in PR #15123: URL: https://github.com/apache/datafusion/pull/15123#discussion_r1987269201 ## datafusion/core/tests/physical_optimizer/projection_pushdown.rs: ## @@ -89,6 +92,10 @@ impl ScalarUDFImpl for DummyUDF { fn return_type(&self, _arg_types:

Re: [PR] fix: PartitionBuffers should not have their own MemoryConsumer [datafusion-comet]

2025-03-11 Thread via GitHub
codecov-commenter commented on PR #1496: URL: https://github.com/apache/datafusion-comet/pull/1496#issuecomment-2714806221 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1496?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

[I] Parametrize ListArray inner field [datafusion]

2025-03-11 Thread via GitHub
comphead opened a new issue, #15162: URL: https://github.com/apache/datafusion/issues/15162 ### Is your feature request related to a problem or challenge? In Apache DataFusion Comet during implementation to handle ARRAY types from Apache Spark it was found that the inner field hardcod

Re: [I] Sort query won't get round-robin repartitioned if input is `MemTable` [datafusion]

2025-03-11 Thread via GitHub
alan910127 commented on issue #15088: URL: https://github.com/apache/datafusion/issues/15088#issuecomment-2714819886 I've tried - this: ```rust fn required_input_distribution(&self) -> Vec { vec![Distribution::UnspecifiedDistribution] } fn benef

Re: [I] Parametrize ListArray inner field [datafusion]

2025-03-11 Thread via GitHub
comphead commented on issue #15162: URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2714809741 @alamb @tustvold appreciate if you can chime in -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] `datafusion-proto` crate feature `json` is broken [datafusion]

2025-03-11 Thread via GitHub
timsaucer commented on issue #15157: URL: https://github.com/apache/datafusion/issues/15157#issuecomment-2714808637 Recommend adding to this issue that we will want to re-enable the `Check datafusion-proto (json)` as part of closing the issue -- This is an automated message from the Apach

Re: [PR] chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) [datafusion]

2025-03-11 Thread via GitHub
goldmedal commented on PR #15123: URL: https://github.com/apache/datafusion/pull/15123#issuecomment-2711195715 Should we remove the deprecated UDF API now? According to our [API health policy](https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines): ```

Re: [PR] Implement tree rendering for `SortPreservingMergeExec` [datafusion]

2025-03-11 Thread via GitHub
irenjj commented on code in PR #15140: URL: https://github.com/apache/datafusion/pull/15140#discussion_r1989258369 ## datafusion/physical-plan/src/sorts/sort_preserving_merge.rs: ## @@ -186,8 +186,15 @@ impl DisplayAs for SortPreservingMergeExec { Ok(())

Re: [PR] Implement tree rendering for `SortPreservingMergeExec` [datafusion]

2025-03-11 Thread via GitHub
irenjj commented on code in PR #15140: URL: https://github.com/apache/datafusion/pull/15140#discussion_r1989258369 ## datafusion/physical-plan/src/sorts/sort_preserving_merge.rs: ## @@ -186,8 +186,15 @@ impl DisplayAs for SortPreservingMergeExec { Ok(())

Re: [PR] Implement tree rendering for `SortPreservingMergeExec` [datafusion]

2025-03-11 Thread via GitHub
irenjj commented on code in PR #15140: URL: https://github.com/apache/datafusion/pull/15140#discussion_r1989268937 ## datafusion/physical-plan/src/sorts/sort_preserving_merge.rs: ## @@ -186,8 +186,15 @@ impl DisplayAs for SortPreservingMergeExec { Ok(())

Re: [PR] Split out `datafusion-substrait` and `datafusion-proto` CI feature checks [datafusion]

2025-03-11 Thread via GitHub
alamb commented on code in PR #15156: URL: https://github.com/apache/datafusion/pull/15156#discussion_r1989216255 ## .github/workflows/rust.yml: ## @@ -66,9 +66,12 @@ jobs: # the changes to `Cargo.lock` after building with the updated manifest. cargo check

Re: [I] Different unnests on `plan_to_sql` are merged [datafusion]

2025-03-11 Thread via GitHub
blaginin commented on issue #15128: URL: https://github.com/apache/datafusion/issues/15128#issuecomment-2714441462 > Perhaps we can simply throw an error I've been thinking about https://github.com/apache/datafusion/pull/15159 - arguably a dirty fix, but I can't come up with anything

Re: [PR] Add `insta` / snapshot testing to CLI & set up AWS mock [datafusion]

2025-03-11 Thread via GitHub
alamb commented on PR #13672: URL: https://github.com/apache/datafusion/pull/13672#issuecomment-2714438025 I merged up to resolve conflicts and plan to merge this when the tests pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [I] [DISCUSS] Release DataFusion `46.0.1` Patcheset (March 2025) [datafusion]

2025-03-11 Thread via GitHub
Omega359 commented on issue #15151: URL: https://github.com/apache/datafusion/issues/15151#issuecomment-2714279380 Seems reasonable to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[I] `datafusion-proto` crate feature `json` is broken [datafusion]

2025-03-11 Thread via GitHub
alamb opened a new issue, #15157: URL: https://github.com/apache/datafusion/issues/15157 ### Describe the bug I am not sure what the `json` feature is supposed to do in datafusion-proto but it is broken ### To Reproduce ```shell cargo check --profile ci --all-targets

Re: [PR] Fix wasm32 build on version 46 [datafusion]

2025-03-11 Thread via GitHub
alamb commented on PR #15102: URL: https://github.com/apache/datafusion/pull/15102#issuecomment-2714390818 Filed a ticket to track adding coverage - https://github.com/apache/datafusion/issues/15158 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] regression: DataFusion 46 wasm compile error with parquet [datafusion]

2025-03-11 Thread via GitHub
alamb closed issue #15150: regression: DataFusion 46 wasm compile error with parquet URL: https://github.com/apache/datafusion/issues/15150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [I] Serde feature is broken [datafusion]

2025-03-11 Thread via GitHub
alamb closed issue #15122: Serde feature is broken URL: https://github.com/apache/datafusion/issues/15122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] Split out `datafusion-substrait` and `datafusion-proto` CI feature checks [datafusion]

2025-03-11 Thread via GitHub
alamb commented on code in PR #15156: URL: https://github.com/apache/datafusion/pull/15156#discussion_r1989360930 ## .github/workflows/rust.yml: ## @@ -79,28 +82,68 @@ jobs: uses: ./.github/actions/setup-builder with: rust-version: stable - - na

Re: [PR] Fix broken `serde` feature [datafusion]

2025-03-11 Thread via GitHub
alamb merged PR #15124: URL: https://github.com/apache/datafusion/pull/15124 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Fix broken `serde` feature [datafusion]

2025-03-11 Thread via GitHub
alamb commented on PR #15124: URL: https://github.com/apache/datafusion/pull/15124#issuecomment-2714425431 Thanks again @vadimpiven and @Weijun-H -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] #15108 issue: "Non Panic Task error" is not an internal error [datafusion]

2025-03-11 Thread via GitHub
Weijun-H commented on code in PR #15109: URL: https://github.com/apache/datafusion/pull/15109#discussion_r1989434623 ## datafusion/physical-plan/src/stream.rs: ## @@ -128,7 +128,7 @@ impl ReceiverStreamBuilder { // the JoinSet were aborted, which in

Re: [PR] Implement tree rendering for `SortPreservingMergeExec` [datafusion]

2025-03-11 Thread via GitHub
irenjj commented on code in PR #15140: URL: https://github.com/apache/datafusion/pull/15140#discussion_r1989258369 ## datafusion/physical-plan/src/sorts/sort_preserving_merge.rs: ## @@ -186,8 +186,15 @@ impl DisplayAs for SortPreservingMergeExec { Ok(())

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
alamb commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989412400 ## content/blog/2025-03-05-ordering-analysis.md: ## @@ -0,0 +1,353 @@ +--- +layout: post +title: Analysis of Ordering for Better Plans +date: 2025-03-05 +author: Must

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
Omega359 commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989461842 ## content/blog/2025-03-11-ordering-analysis.md: ## @@ -0,0 +1,381 @@ +--- +layout: post +title: Using Ordering for Better Plans in Apache DataFusion +date: 2025-0

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
Omega359 commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989464583 ## content/blog/2025-03-11-ordering-analysis.md: ## @@ -0,0 +1,381 @@ +--- +layout: post +title: Using Ordering for Better Plans in Apache DataFusion +date: 2025-0

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
Omega359 commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989466124 ## content/blog/2025-03-11-ordering-analysis.md: ## @@ -0,0 +1,381 @@ +--- +layout: post +title: Using Ordering for Better Plans in Apache DataFusion +date: 2025-0

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
Omega359 commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989467404 ## content/blog/2025-03-11-ordering-analysis.md: ## @@ -0,0 +1,381 @@ +--- +layout: post +title: Using Ordering for Better Plans in Apache DataFusion +date: 2025-0

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
alamb commented on PR #58: URL: https://github.com/apache/datafusion-site/pull/58#issuecomment-2714590589 Thank you so much @akurmustafa and @ozankabak -- I pushed two commits with minor changes (spelling/code formatting) and then to update the date to today. I think this is ready t

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
Omega359 commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989475539 ## content/blog/2025-03-11-ordering-analysis.md: ## @@ -0,0 +1,381 @@ +--- +layout: post +title: Using Ordering for Better Plans in Apache DataFusion +date: 2025-0

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-11 Thread via GitHub
Omega359 commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1989475539 ## content/blog/2025-03-11-ordering-analysis.md: ## @@ -0,0 +1,381 @@ +--- +layout: post +title: Using Ordering for Better Plans in Apache DataFusion +date: 2025-0

[PR] Simpler to see expressions in tree explain mode [datafusion]

2025-03-11 Thread via GitHub
irenjj opened a new pull request, #15163: URL: https://github.com/apache/datafusion/pull/15163 ## Which issue does this PR close? - Closes #15107 ## Rationale for this change ## What changes are included in this PR? ## Are these changes test

Re: [I] Parametrize ListArray inner field [datafusion]

2025-03-11 Thread via GitHub
zhuqi-lucas commented on issue #15162: URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2714878036 Adding a config to support relax name checking is a good solution to me: If field A and field B only the name is different, but we only check other part besides name,

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-11 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1988708402 ## src/ast/mod.rs: ## @@ -6148,12 +6218,12 @@ impl fmt::Display for GrantObjects { #[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash

[PR] Bump `env_logger` from `0.11.6` to `0.11.7` [datafusion]

2025-03-11 Thread via GitHub
mbrobbel opened a new pull request, #15148: URL: https://github.com/apache/datafusion/pull/15148 ## Which issue does this PR close? Remove the `humantime` dependency via `env_logger` (https://rustsec.org/advisories/RUSTSEC-2025-0014.html) To remove it fully we need https://github.c

Re: [PR] #15108 issue: "Non Panic Task error" is not an internal error [datafusion]

2025-03-11 Thread via GitHub
Satyam018 closed pull request #15109: #15108 issue: "Non Panic Task error" is not an internal error URL: https://github.com/apache/datafusion/pull/15109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]

2025-03-11 Thread via GitHub
alamb commented on issue #13814: URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2713644414 I also tried removing all the doctests from lib.rs and that didn't help (I didn't expect it to but I wanted to try) ```diff diff --git a/datafusion/core/src/lib.rs b/data

Re: [I] Sort query won't get round-robin repartitioned if input is `MemTable` [datafusion]

2025-03-11 Thread via GitHub
2010YOUY01 commented on issue #15088: URL: https://github.com/apache/datafusion/issues/15088#issuecomment-2713675854 > Hi [@2010YOUY01](https://github.com/2010YOUY01), > > I've taken a look into this and found that the issue seems to be more related to `SortExec` itself rather than th

[I] The Comet compiled on the Ubuntu operating system cannot run on CentOS 7 [datafusion-comet]

2025-03-11 Thread via GitHub
dpengpeng opened a new issue, #1499: URL: https://github.com/apache/datafusion-comet/issues/1499 ### What is the problem the feature request solves? I compiled the source code of Comet release 0.6.0 on the Ubuntu (18.04.4 LTS) operating system, obtaining a `comet-spark-spark3.4_2.12-0

[PR] feat: topk functionality for aggregates should support utf8view [datafusion]

2025-03-11 Thread via GitHub
zhuqi-lucas opened a new pull request, #15152: URL: https://github.com/apache/datafusion/pull/15152 ## Which issue does this PR close? - Closes sub-task topk functionality for aggregates should support utf8view for [#15096](https://github.com/apache/datafusion/issues/1509

[PR] fix: compound_field_access doesn't identifier qualifier. [datafusion]

2025-03-11 Thread via GitHub
chenkovsky opened a new pull request, #15153: URL: https://github.com/apache/datafusion/pull/15153 ## Which issue does this PR close? - Closes #15118. ## Rationale for this change In sql_compound_field_access_to_expr, datafusion doesn't check whether prefix can be merged

Re: [I] Unparse of logical plans with `LEFT ANTI` and `LEFT SEMI` joins generate invalid SQL [datafusion]

2025-03-11 Thread via GitHub
chenkovsky commented on issue #15127: URL: https://github.com/apache/datafusion/issues/15127#issuecomment-2713894092 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] chore: add an "expr_planners" method to SessionState [datafusion]

2025-03-11 Thread via GitHub
alamb commented on code in PR #15119: URL: https://github.com/apache/datafusion/pull/15119#discussion_r1989022172 ## datafusion/core/src/execution/context/mod.rs: ## @@ -1632,7 +1632,7 @@ impl FunctionRegistry for SessionContext { } fn expr_planners(&self) -> Vec> {

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-11 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1989297099 ## src/ast/mod.rs: ## @@ -6148,12 +6218,12 @@ impl fmt::Display for GrantObjects { #[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash

Re: [PR] Fix wasm32 build on version 46 [datafusion]

2025-03-11 Thread via GitHub
alamb commented on PR #15102: URL: https://github.com/apache/datafusion/pull/15102#issuecomment-2714391440 Thanks again @XiangpengHao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] feat: support customize metadata in alias for dataframe api [datafusion]

2025-03-11 Thread via GitHub
jonahgao commented on code in PR #15120: URL: https://github.com/apache/datafusion/pull/15120#discussion_r1989378352 ## datafusion/expr/src/expr_schema.rs: ## @@ -343,7 +343,17 @@ impl ExprSchemable for Expr { fn metadata(&self, schema: &dyn ExprSchema) -> Result> {

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-11 Thread via GitHub
alamb commented on PR #15110: URL: https://github.com/apache/datafusion/pull/15110#issuecomment-2714477098 (thank you for checking this @findepi -- 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-11 Thread via GitHub
findepi commented on code in PR #15110: URL: https://github.com/apache/datafusion/pull/15110#discussion_r1989510726 ## datafusion/optimizer/src/simplify_expressions/unwrap_cast.rs: ## @@ -177,6 +192,45 @@ pub(super) fn is_cast_expr_and_support_unwrap_cast_in_comparison_for_inli

Re: [I] Comet 0.7.0 (March 2025) [datafusion-comet]

2025-03-11 Thread via GitHub
andygrove commented on issue #1420: URL: https://github.com/apache/datafusion-comet/issues/1420#issuecomment-2714702417 I'd like to create the release candidate sometime this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[PR] Only unnest source for `EmptyRelation` [datafusion]

2025-03-11 Thread via GitHub
blaginin opened a new pull request, #15159: URL: https://github.com/apache/datafusion/pull/15159 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/15128 ## Rationale for this change We currently override source losing some data, see iss

Re: [PR] Config: Add support default sql varchar to view types [datafusion]

2025-03-11 Thread via GitHub
zhuqi-lucas commented on code in PR #15104: URL: https://github.com/apache/datafusion/pull/15104#discussion_r1989511048 ## datafusion/common/src/config.rs: ## @@ -252,6 +252,11 @@ config_namespace! { /// string length and thus DataFusion can not enforce such limits.

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-11 Thread via GitHub
alan910127 commented on PR #15110: URL: https://github.com/apache/datafusion/pull/15110#issuecomment-2714725299 > As in only do the rewrite if the sequence > > - cast to int > - cast (back) to string > > results in the exact same string as went int @alamb I found that

Re: [PR] Statistics: Migrate to Distribution from Precision (#14896) [datafusion]

2025-03-11 Thread via GitHub
cj-zhukov closed pull request #15154: Statistics: Migrate to Distribution from Precision (#14896) URL: https://github.com/apache/datafusion/pull/15154 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] chore(deps): bump tempfile from 3.17.1 to 3.18.0 [datafusion]

2025-03-11 Thread via GitHub
comphead merged PR #15146: URL: https://github.com/apache/datafusion/pull/15146 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Parse MySQL `ALTER TABLE DROP FOREIGN KEY` syntax [datafusion-sqlparser-rs]

2025-03-11 Thread via GitHub
mvzink commented on code in PR #1762: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1762#discussion_r1989712091 ## src/parser/mod.rs: ## @@ -8002,6 +8002,11 @@ impl<'a> Parser<'a> { && dialect_of!(self is MySqlDialect | GenericDialect)

Re: [I] Parametrize ListArray inner field [datafusion]

2025-03-11 Thread via GitHub
tustvold commented on issue #15162: URL: https://github.com/apache/datafusion/issues/15162#issuecomment-2715020448 > Arrow-rs is not very configurable unlike to DataFusion If you have a concrete proposal, feel free to raise an issue. FWIW most kernels do take various options to alter

Re: [PR] chore: Stop disabling readside padding in TPC stability suite [datafusion-comet]

2025-03-11 Thread via GitHub
parthchandra commented on code in PR #1491: URL: https://github.com/apache/datafusion-comet/pull/1491#discussion_r1989718327 ## spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala: ## @@ -278,7 +278,6 @@ trait CometPlanStabilitySuite extends DisableAda

Re: [PR] fix: use common implementation of handling object store and hdfs urls for native_datafusion and native_iceberg_compat [datafusion-comet]

2025-03-11 Thread via GitHub
parthchandra commented on PR #1494: URL: https://github.com/apache/datafusion-comet/pull/1494#issuecomment-2715024318 @mbutrovich, @comphead if you could please review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-11 Thread via GitHub
alan910127 commented on PR #15110: URL: https://github.com/apache/datafusion/pull/15110#issuecomment-2715045737 @findepi sorry I didn't notice your comment and I just pushed a new version with the unwrapping logic deleted. So you think the two optimizations **should** coexist?

[PR] fix: solution on #15126 [datafusion]

2025-03-11 Thread via GitHub
SanjayUG opened a new pull request, #15164: URL: https://github.com/apache/datafusion/pull/15164 Hi there, checkout this pr on #15126 issue, document unique identifiers in MemoryConsumer and MemoryReservation. The unique identifier functionality was already implemented but no

Re: [PR] feat: Attach `Diagnostic` to more than one column errors in scalar_subquery and in_subquery [datafusion]

2025-03-11 Thread via GitHub
eliaperantoni commented on code in PR #15143: URL: https://github.com/apache/datafusion/pull/15143#discussion_r1988585977 ## datafusion/sql/tests/cases/diagnostic.rs: ## @@ -286,3 +286,121 @@ fn test_invalid_function() -> Result<()> { assert_eq!(diag.span, Some(spans["whole

Re: [PR] Improve parsing `extra_info` in tree explain [datafusion]

2025-03-11 Thread via GitHub
irenjj commented on PR #15125: URL: https://github.com/apache/datafusion/pull/15125#issuecomment-2712119745 Thank you for helping me revise the question! @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Do not print logical plans in `tree` explain format [datafusion]

2025-03-11 Thread via GitHub
alamb closed issue #15020: Do not print logical plans in `tree` explain format URL: https://github.com/apache/datafusion/issues/15020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] PhysicalOptimizer rule 'join_selection' failed [datafusion]

2025-03-11 Thread via GitHub
xudong963 commented on issue #15136: URL: https://github.com/apache/datafusion/issues/15136#issuecomment-2713618560 Could you please directly use datafusion-cli to create these two tables and execute join to see what happens? -- This is an automated message from the Apache Git Service. To

Re: [PR] #15108 issue: "Non Panic Task error" is not an internal error [datafusion]

2025-03-11 Thread via GitHub
Satyam018 closed pull request #15109: #15108 issue: "Non Panic Task error" is not an internal error URL: https://github.com/apache/datafusion/pull/15109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] Substrait support for propagating TableScan.filters to Substrait ReadRel [datafusion]

2025-03-11 Thread via GitHub
alamb closed issue #14193: Substrait support for propagating TableScan.filters to Substrait ReadRel URL: https://github.com/apache/datafusion/issues/14193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Int64 as default type for make_array function empty or null case [datafusion]

2025-03-11 Thread via GitHub
joroKr21 commented on code in PR #10790: URL: https://github.com/apache/datafusion/pull/10790#discussion_r1988683281 ## datafusion/functions-array/src/make_array.rs: ## @@ -131,6 +131,11 @@ impl ScalarUDFImpl for MakeArray { } } +// Empty array is a special case that is

Re: [I] Change mapping of SQL `VARCHAR` from `Utf8` to `Utf8View` [datafusion]

2025-03-11 Thread via GitHub
zhuqi-lucas commented on issue #15096: URL: https://github.com/apache/datafusion/issues/15096#issuecomment-2713415370 New sub_task: - [ ] topk functionality for aggregates should support utf8view -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] feat: implement tree explain for `ProjectionExec` [datafusion]

2025-03-11 Thread via GitHub
Standing-Man commented on code in PR #15082: URL: https://github.com/apache/datafusion/pull/15082#discussion_r1989151892 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -539,6 +539,158 @@ physical_plan 17)│ format: arrow │ 18)└

[I] Improved CI test coverage for rust features [datafusion]

2025-03-11 Thread via GitHub
alamb opened a new issue, #15155: URL: https://github.com/apache/datafusion/issues/15155 ### Is your feature request related to a problem or challenge? DataFusion has many rust features (e.g. `--features=parquet`) and recently we hit discovered several failures for features during ref

Re: [I] Attach `Diagnostic` to "more than one column in subquery" error [datafusion]

2025-03-11 Thread via GitHub
eliaperantoni commented on issue #14438: URL: https://github.com/apache/datafusion/issues/14438#issuecomment-2713064816 > > Just one minor thing: it seems like there are scalar subqueries and then "in" subqueries. e.g. SELECT * FROM users WHERE id IN (SELECT id FROM admins). Do you think yo

Re: [PR] fix: unparse for subqueryalias [datafusion]

2025-03-11 Thread via GitHub
goldmedal merged PR #15068: URL: https://github.com/apache/datafusion/pull/15068 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] fix: unparse for subqueryalias [datafusion]

2025-03-11 Thread via GitHub
goldmedal commented on PR #15068: URL: https://github.com/apache/datafusion/pull/15068#issuecomment-2713703765 Thanks @chenkovsky and @alamb for reviewing 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] Statistics: Migrate to Distribution from Precision (#14896) [datafusion]

2025-03-11 Thread via GitHub
cj-zhukov opened a new pull request, #15154: URL: https://github.com/apache/datafusion/pull/15154 ## Which issue does this PR close? - Closes #14896. ## Rationale for this change ## What changes are included in this PR? ## Are these changes

Re: [I] Regression: TypeSignature::Coercible does not respect user-declared type coercions [datafusion]

2025-03-11 Thread via GitHub
niebayes commented on issue #15117: URL: https://github.com/apache/datafusion/issues/15117#issuecomment-2710052300 > btw the equivalent rewrite for the old signature should be > > signature: Signature::one_of( > vec![ > TypeSignature::Coercible

Re: [PR] Int64 as default type for make_array function empty or null case [datafusion]

2025-03-11 Thread via GitHub
joroKr21 commented on code in PR #10790: URL: https://github.com/apache/datafusion/pull/10790#discussion_r1988692075 ## datafusion/functions-array/src/set_ops.rs: ## @@ -259,6 +259,17 @@ fn generic_set_lists( return general_array_distinct::(l, &field); } +//

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-11 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1988719022 ## src/parser/mod.rs: ## @@ -10955,134 +10955,226 @@ impl<'a> Parser<'a> { } else { Some(self.parse_identifier()?)

Re: [PR] Int64 as default type for make_array function empty or null case [datafusion]

2025-03-11 Thread via GitHub
joroKr21 commented on code in PR #10790: URL: https://github.com/apache/datafusion/pull/10790#discussion_r1988720176 ## datafusion/functions-array/src/set_ops.rs: ## @@ -259,6 +259,17 @@ fn generic_set_lists( return general_array_distinct::(l, &field); } +//

[PR] Saner handling of nulls inside arrays [datafusion]

2025-03-11 Thread via GitHub
joroKr21 opened a new pull request, #15149: URL: https://github.com/apache/datafusion/pull/15149 ## Which issue does this PR close? - Part of #7142 ## Rationale for this change ## What changes are included in this PR? ## Are these changes te

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

2025-03-11 Thread via GitHub
xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2713344196 @alamb, I'll also be in charge of this release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

  1   2   3   4   5   6   >