Re: [PR] feat: Add `array_max` function support [datafusion]

2025-03-07 Thread via GitHub
findepi merged PR #14470: URL: https://github.com/apache/datafusion/pull/14470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafu

Re: [PR] Enable Dataframe to be converted into views which can be used in register_table [datafusion-python]

2025-03-07 Thread via GitHub
timsaucer commented on code in PR #1016: URL: https://github.com/apache/datafusion-python/pull/1016#discussion_r1985040308 ## src/dataframe.rs: ## @@ -50,9 +52,79 @@ use crate::{ expr::{sort_expr::PySortExpr, PyExpr}, }; +// https://github.com/apache/datafusion-python/p

Re: [PR] Remove deprecated function `OptimizerRule::try_optimize` [datafusion]

2025-03-07 Thread via GitHub
alamb commented on code in PR #15051: URL: https://github.com/apache/datafusion/pull/15051#discussion_r1985148749 ## datafusion/optimizer/src/optimizer.rs: ## @@ -304,43 +284,25 @@ impl TreeNodeRewriter for Rewriter<'_> { fn f_down(&mut self, node: LogicalPlan) -> Result

[PR] Add object store support [datafusion-ray]

2025-03-07 Thread via GitHub
robtandy opened a new pull request, #78: URL: https://github.com/apache/datafusion-ray/pull/78 # Basic Object store support ### Changes: - Updated `register_parquet`, `register_listing_table`, and added `register_csv` such that they will automatically register an object store b

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-07 Thread via GitHub
comphead commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2706871828 I think the last merge brought up a new test which fails now on schema mismatch, checking this ``` - array_compact *** FAILED *** (204 milliseconds) org.apache.sp

Re: [PR] chore: Refactor CometScanRule to avoid duplication and improve fallback messages [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove commented on PR #1474: URL: https://github.com/apache/datafusion-comet/pull/1474#issuecomment-2706885107 I am going to start again with this, breaking it down into smaller incremental changes. Thanks for the review so far @kazuyukitanimura and @mbutrovich -- This is an automa

Re: [PR] chore: Refactor CometScanRule to avoid duplication and improve fallback messages [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove closed pull request #1474: chore: Refactor CometScanRule to avoid duplication and improve fallback messages URL: https://github.com/apache/datafusion-comet/pull/1474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] chore: Enable CI checks for `native_datafusion` scan [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove commented on code in PR #1479: URL: https://github.com/apache/datafusion-comet/pull/1479#discussion_r1984220205 ## .github/workflows/pr_build.yml: ## @@ -76,6 +76,41 @@ jobs: # upload test reports only for java 17 upload-test-reports: ${{ matrix.j

Re: [I] Support different `EXPLAIN` formats via SQL [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #15021: URL: https://github.com/apache/datafusion/issues/15021#issuecomment-2703715380 I was thinking about implementing this feature I think it would be a relatively straightforward implementation: 1. Extend the DataFusion explain statement to have a format

[I] Overflow happened on: -9223372036854775808 div -1 [datafusion-comet]

2025-03-07 Thread via GitHub
wForget opened a new issue, #1477: URL: https://github.com/apache/datafusion-comet/issues/1477 ### Describe the bug https://github.com/apache/arrow-rs/pull/7159 only avoids overflow of remainder(%), the divide operation still has similar issues as #1412. In spark, the `/` operation

Re: [I] Internal error: Non Panic Task error: task 113 was cancelled. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker [datafu

2025-03-07 Thread via GitHub
chenquan commented on issue #15065: URL: https://github.com/apache/datafusion/issues/15065#issuecomment-2707773312 > Was it during shutdown of the tokio runtime? yes. > Was a thread panicing? no. -- This is an automated message from the Apache Git Service. To r

[I] Implement `tree` explain for `SortExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj opened a new issue, #15076: URL: https://github.com/apache/datafusion/issues/15076 ### Is your feature request related to a problem or challenge? Part of https://github.com/apache/datafusion/issues/14914 ### Describe the solution you'd like Add tree format to the E

Re: [I] Implement `tree` explain for `SortExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj commented on issue #15076: URL: https://github.com/apache/datafusion/issues/15076#issuecomment-2707785826 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] ignore: Refactor CometScanRule [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove commented on PR #1483: URL: https://github.com/apache/datafusion-comet/pull/1483#issuecomment-2707814756 This PR fixes some bugs, but many tests are now failing in `CometColumnarShuffleSuite` and its subclasses when `native_datafusion` is the default. Reasons: - `Co

Re: [I] Upgrade to sqlparser 0.55.0 [datafusion]

2025-03-07 Thread via GitHub
jonahgao commented on issue #15071: URL: https://github.com/apache/datafusion/issues/15071#issuecomment-2707862584 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Handle columns in with_new_exprs with a Join [datafusion]

2025-03-07 Thread via GitHub
jonahgao commented on PR #15055: URL: https://github.com/apache/datafusion/pull/15055#issuecomment-2707863018 Thanks @delamarch3 @niebayes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Consolidate optimizer passes to improve planning speed [datafusion]

2025-03-07 Thread via GitHub
jayzhan211 commented on issue #15045: URL: https://github.com/apache/datafusion/issues/15045#issuecomment-2707883471 If we can move type coercion into LogicalPlan building stage https://github.com/apache/datafusion/issues/14618, I think Union logic is able to resolved at that time, then we

Re: [PR] Refactor EnforceDistribution test cases to demonstrate dependencies across optimizer runs. [datafusion]

2025-03-07 Thread via GitHub
wiedld commented on code in PR #15074: URL: https://github.com/apache/datafusion/pull/15074#discussion_r1985718083 ## datafusion/core/tests/physical_optimizer/enforce_distribution.rs: ## @@ -1542,11 +1562,12 @@ fn multi_smj_joins() -> Result<()> { "DataS

[I] Implement tree rendering for StreamingTableExec [datafusion]

2025-03-07 Thread via GitHub
Standing-Man opened a new issue, #15086: URL: https://github.com/apache/datafusion/issues/15086 ### Is your feature request related to a problem or challenge? Fixing part of #14914. ### Describe the solution you'd like _No response_ ### Describe alternatives you've

[PR] implement tree rendering for StreamingTableExec [datafusion]

2025-03-07 Thread via GitHub
Standing-Man opened a new pull request, #15085: URL: https://github.com/apache/datafusion/pull/15085 ## Which issue does this PR close? - Closes part of #14914. ## Rationale for this change ## What changes are included in this PR? implement tree

Re: [I] Enable `used_underscore_binding` clippy lint [datafusion]

2025-03-07 Thread via GitHub
Standing-Man commented on issue #14649: URL: https://github.com/apache/datafusion/issues/14649#issuecomment-2708093680 Hi @ding-young, how is this going? If you aren't planning on working on them in the near term perhaps i can help you fix it. -- This is an automated message from the Apa

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985605486 ## src/parser/mod.rs: ## @@ -10961,127 +10961,184 @@ impl<'a> Parser<'a> { }) } -pub fn parse_set(&mut self) -> Result { -

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985608039 ## tests/sqlparser_common.rs: ## @@ -14654,3 +14640,23 @@ fn parse_set_names() { dialects.verified_stmt("SET NAMES 'utf8'"); dialects

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985619929 ## src/parser/mod.rs: ## @@ -10961,127 +10961,184 @@ impl<'a> Parser<'a> { }) } -pub fn parse_set(&mut self) -> Result { -

Re: [PR] chore(deps): bump sqllogictest from 0.27.2 to 0.28.0 [datafusion]

2025-03-07 Thread via GitHub
alamb closed pull request #15060: chore(deps): bump sqllogictest from 0.27.2 to 0.28.0 URL: https://github.com/apache/datafusion/pull/15060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
mvzink commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985631206 ## src/parser/mod.rs: ## @@ -10961,127 +10961,182 @@ impl<'a> Parser<'a> { }) } -pub fn parse_set(&mut self) -> Result { -le

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985615457 ## src/parser/mod.rs: ## @@ -10961,127 +10961,182 @@ impl<'a> Parser<'a> { }) } -pub fn parse_set(&mut self) -> Result { -

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985543609 ## src/parser/mod.rs: ## @@ -10961,127 +10961,182 @@ impl<'a> Parser<'a> { }) } -pub fn parse_set(&mut self) -> Result { -

Re: [I] Run DataFusion benchmarks regularly and track performance history over time [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #5504: URL: https://github.com/apache/datafusion/issues/5504#issuecomment-2707185516 I also think it is important not to tie ourselves to any one particular CI framework / infrastructure. It should be possible to run the scripts to gather data in any environment

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
mvzink commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985643698 ## src/parser/mod.rs: ## @@ -10961,127 +10961,182 @@ impl<'a> Parser<'a> { }) } -pub fn parse_set(&mut self) -> Result { -le

Re: [PR] Improve explain tree formatting for longer lines / word wrap [datafusion]

2025-03-07 Thread via GitHub
alamb commented on code in PR #15031: URL: https://github.com/apache/datafusion/pull/15031#discussion_r1985643612 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -245,7 +245,97 @@ physical_plan 05)│ FilterExec│ 06)│ │ 07

Re: [PR] feat: Use official DataFusion 46.0.0 release [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove merged PR #1484: URL: https://github.com/apache/datafusion-comet/pull/1484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2707696640 I'd quite like to merge https://github.com/apache/datafusion-comet/pull/1479 before this one so that we can be sure that no additional test failures are introduced for `native

Re: [PR] chore: Enable CI checks for `native_datafusion` scan [datafusion-comet]

2025-03-07 Thread via GitHub
kazuyukitanimura commented on code in PR #1479: URL: https://github.com/apache/datafusion-comet/pull/1479#discussion_r1985845051 ## .github/workflows/pr_build.yml: ## @@ -76,6 +76,41 @@ jobs: # upload test reports only for java 17 upload-test-reports: ${{ m

[I] Test native_datafusion and native_iceverg_compat with all supported Java, Spark, and Scala versions [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove opened a new issue, #1486: URL: https://github.com/apache/datafusion-comet/issues/1486 ### What is the problem the feature request solves? _No response_ ### Describe the potential solution _No response_ ### Additional context _No response_ -- Th

Re: [PR] chore: Enable CI checks for `native_datafusion` scan [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove commented on code in PR #1479: URL: https://github.com/apache/datafusion-comet/pull/1479#discussion_r1985846108 ## .github/workflows/pr_build.yml: ## @@ -76,6 +76,41 @@ jobs: # upload test reports only for java 17 upload-test-reports: ${{ matrix.j

Re: [PR] Implement `tree` explain for `HashJoinExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj commented on code in PR #15079: URL: https://github.com/apache/datafusion/pull/15079#discussion_r1985989767 ## datafusion/physical-plan/src/joins/hash_join.rs: ## @@ -669,8 +669,15 @@ impl DisplayAs for HashJoinExec { ) } Display

Re: [I] Internal error: Non Panic Task error: task 113 was cancelled. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker [datafu

2025-03-07 Thread via GitHub
chenquan commented on issue #15065: URL: https://github.com/apache/datafusion/issues/15065#issuecomment-2708024266 @alamb Hello, have you made any progress? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] feat: implement tree explain for ProjectionExec [datafusion]

2025-03-07 Thread via GitHub
Standing-Man commented on code in PR #15082: URL: https://github.com/apache/datafusion/pull/15082#discussion_r1986000615 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -519,6 +519,150 @@ physical_plan 17)│ format: arrow │ 18)└

Re: [I] Support Push down expression evaluation in `TableProviders` [datafusion]

2025-03-07 Thread via GitHub
adriangb commented on issue #14993: URL: https://github.com/apache/datafusion/issues/14993#issuecomment-2703778115 Would SchemaAdapter be a good place to implement this functionality? It already has knowledge of the required columns and file schema. We'd need piping around it (changes to Ta

Re: [PR] feat: implement tree explain for ProjectionExec [datafusion]

2025-03-07 Thread via GitHub
irenjj commented on code in PR #15082: URL: https://github.com/apache/datafusion/pull/15082#discussion_r1985993230 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -519,6 +519,150 @@ physical_plan 17)│ format: arrow │ 18)└───┘

Re: [I] Upgrade to sqlparser 0.55.0 [datafusion]

2025-03-07 Thread via GitHub
jonahgao commented on issue #15071: URL: https://github.com/apache/datafusion/issues/15071#issuecomment-2708037360 Thank you @PokIsemaine for your work. Assigned this issue to you ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Implement `tree` explain for `HashJoinExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj commented on code in PR #15079: URL: https://github.com/apache/datafusion/pull/15079#discussion_r1985992579 ## datafusion/physical-plan/src/joins/hash_join.rs: ## @@ -669,8 +669,15 @@ impl DisplayAs for HashJoinExec { ) } Display

Re: [PR] Minor: Improve documentation of `need_handle_count_bug` [datafusion]

2025-03-07 Thread via GitHub
suibianwanwank commented on code in PR #15050: URL: https://github.com/apache/datafusion/pull/15050#discussion_r1985999199 ## datafusion/optimizer/src/decorrelate.rs: ## @@ -56,10 +56,14 @@ pub struct PullUpCorrelatedExpr { /// Indicates if we encounter any correlated expre

Re: [I] Implement `tree` explain for `BoundedWindowAggExec` and `WindowAggExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj commented on issue #15083: URL: https://github.com/apache/datafusion/issues/15083#issuecomment-2708068665 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
iffyio commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1986001377 ## src/ast/mod.rs: ## @@ -2947,6 +2947,17 @@ pub enum Statement { variables: OneOrManyWithParens, value: Vec, }, + +/// ```sq

[PR] Implement `tree explain for `BoundedWindowAggExec` and `WindowAggExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj opened a new pull request, #15084: URL: https://github.com/apache/datafusion/pull/15084 ## Which issue does this PR close? - Closes #15083 ## Rationale for this change ## What changes are included in this PR? ## Are these changes test

[I] Implement `tree` explain for `BoundedWindowAggExec` and `WindowAggExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj opened a new issue, #15083: URL: https://github.com/apache/datafusion/issues/15083 ### Is your feature request related to a problem or challenge? partof #14914 ### Describe the solution you'd like _No response_ ### Describe alternatives you've considered

Re: [I] Timeouts reading large files from object stores on slow connetions [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #15067: URL: https://github.com/apache/datafusion/issues/15067#issuecomment-2706951343 To trigger this error you need a slow internet connection and a parquet file where the row group is "large" The https://datasets.clickhouse.com/hits_compatible/athena_part

Re: [PR] Example for using a separate threadpool for CPU bound work (try 2) [datafusion]

2025-03-07 Thread via GitHub
alamb commented on PR #14286: URL: https://github.com/apache/datafusion/pull/14286#issuecomment-2707071207 > I have one more theory I want to chase down this afternoon and then I will write up my thoughts on next steps here My other theory was that DataFusion might start requests but

Re: [I] Timeouts reading large files from object stores on slow connetions [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #15067: URL: https://github.com/apache/datafusion/issues/15067#issuecomment-2707061036 Maybe it is time for a `object_store_util` crate or similar mirroring `futures_util` that has a bunch of these ObjectStore combinators (chunked paralle, reads, limited request siz

[PR] Cherry-pick fix to issue #14348 [datafusion]

2025-03-07 Thread via GitHub
anlinc opened a new pull request, #15070: URL: https://github.com/apache/datafusion/pull/15070 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [I] Release DataFusion `46.0.0` [datafusion]

2025-03-07 Thread via GitHub
alamb closed issue #14123: Release DataFusion `46.0.0` URL: https://github.com/apache/datafusion/issues/14123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [I] Release DataFusion `46.0.0` [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #14123: URL: https://github.com/apache/datafusion/issues/14123#issuecomment-2707248187 The release was approved and published to crates.io See thread here: https://lists.apache.org/thread/rmvsc1ffpqd2z3m0c9lptjypzl83k2j8 I have filed a ticket for 47

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
mvzink commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1985755405 ## src/ast/mod.rs: ## @@ -5334,6 +5345,16 @@ impl fmt::Display for Statement { Statement::List(command) => write!(f, "LIST {command}"),

Re: [PR] Add `insta` / snapshot testing to CLI & set up AWS mock [datafusion]

2025-03-07 Thread via GitHub
alamb commented on code in PR #13672: URL: https://github.com/apache/datafusion/pull/13672#discussion_r1985792488 ## datafusion-cli/CONTRIBUTING.md: ## @@ -0,0 +1,75 @@ + + +# Development instructions + +## Running Tests + +Tests can be run using `cargo` + +```shell +cargo test

Re: [PR] ignore: Refactor CometScanRule [datafusion-comet]

2025-03-07 Thread via GitHub
codecov-commenter commented on PR #1483: URL: https://github.com/apache/datafusion-comet/pull/1483#issuecomment-2707621165 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1483?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

[PR] Minor: split datafusion-cli testing into its own CI job [datafusion]

2025-03-07 Thread via GitHub
alamb opened a new pull request, #15075: URL: https://github.com/apache/datafusion/pull/15075 ## Which issue does this PR close? Draft as it build on https://github.com/apache/datafusion/pull/13672 - Follow on https://github.com/apache/datafusion/pull/13672 ## Rationa

Re: [PR] Fix array_has_all and array_has_any with empty array [datafusion]

2025-03-07 Thread via GitHub
LuQQiu commented on code in PR #15039: URL: https://github.com/apache/datafusion/pull/15039#discussion_r1985809872 ## datafusion/sqllogictest/test_files/array.slt: ## @@ -5818,6 +5818,13 @@ false false false false false false false false false false false false +query BB Re

Re: [PR] Cherry-pick fix to issue #14348 [datafusion]

2025-03-07 Thread via GitHub
anlinc closed pull request #15070: Cherry-pick fix to issue #14348 URL: https://github.com/apache/datafusion/pull/15070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

[PR] chore: update python dependencies [datafusion-ballista]

2025-03-07 Thread via GitHub
milenkovicm opened a new pull request, #1197: URL: https://github.com/apache/datafusion-ballista/pull/1197 # Which issue does this PR close? Closes #. # Rationale for this change keeping py-ballista dependencies up to date # What changes are included in this PR?

Re: [PR] Handle columns in with_new_exprs with a Join [datafusion]

2025-03-07 Thread via GitHub
jonahgao merged PR #15055: URL: https://github.com/apache/datafusion/pull/15055 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] Bug: calling "with_new_exprs" on join after optimization unexpectedly fails [datafusion]

2025-03-07 Thread via GitHub
jonahgao closed issue #14999: Bug: calling "with_new_exprs" on join after optimization unexpectedly fails URL: https://github.com/apache/datafusion/issues/14999 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove opened a new pull request, #1481: URL: https://github.com/apache/datafusion-comet/pull/1481 ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/1480 ## Rationale for this change ## What changes are included

Re: [PR] Implement `tree` explain for `SortExec` [datafusion]

2025-03-07 Thread via GitHub
2010YOUY01 commented on code in PR #15077: URL: https://github.com/apache/datafusion/pull/15077#discussion_r1985964011 ## datafusion/physical-plan/src/sorts/sort.rs: ## @@ -1005,10 +1005,15 @@ impl DisplayAs for SortExec { None => write!(f, "SortExec: expr=[

Re: [I] Upgrade to sqlparser 0.55.0 [datafusion]

2025-03-07 Thread via GitHub
PokIsemaine commented on issue #15071: URL: https://github.com/apache/datafusion/issues/15071#issuecomment-2707977800 @alamb @jonahgao I have already made some progress on this part. If you don't mind, I would like to finish it. I want to use this update to become more familiar with Dat

Re: [PR] Implement `tree` explain for `HashJoinExec` [datafusion]

2025-03-07 Thread via GitHub
2010YOUY01 commented on code in PR #15079: URL: https://github.com/apache/datafusion/pull/15079#discussion_r1985969186 ## datafusion/physical-plan/src/joins/hash_join.rs: ## @@ -669,8 +669,15 @@ impl DisplayAs for HashJoinExec { ) } Dis

Re: [I] Consolidate optimizer passes to improve planning speed [datafusion]

2025-03-07 Thread via GitHub
clflushopt commented on issue #15045: URL: https://github.com/apache/datafusion/issues/15045#issuecomment-2707988622 This seems like an interesting task, I'd like to work on it ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] feat: implement tree explain for ProjectionExec [datafusion]

2025-03-07 Thread via GitHub
Standing-Man commented on code in PR #15082: URL: https://github.com/apache/datafusion/pull/15082#discussion_r1985987690 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -519,6 +519,150 @@ physical_plan 17)│ format: arrow │ 18)└

[PR] ignore: Refactor scan [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove opened a new pull request, #1483: URL: https://github.com/apache/datafusion-comet/pull/1483 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

[PR] chore(deps): bump sqllogictest from 0.27.2 to 0.28.0 [datafusion]

2025-03-07 Thread via GitHub
dependabot[bot] opened a new pull request, #15060: URL: https://github.com/apache/datafusion/pull/15060 Bumps [sqllogictest](https://github.com/risinglightdb/sqllogictest-rs) from 0.27.2 to 0.28.0. Release notes Sourced from https://github.com/risinglightdb/sqllogictest-rs/releases

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-07 Thread via GitHub
iffyio commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1984492397 ## src/parser/mod.rs: ## @@ -10961,127 +10961,182 @@ impl<'a> Parser<'a> { }) } -pub fn parse_set(&mut self) -> Result { -le

Re: [PR] Document guidelines for physical operator yielding [datafusion]

2025-03-07 Thread via GitHub
berkaysynnada commented on code in PR #15030: URL: https://github.com/apache/datafusion/pull/15030#discussion_r1984885052 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -260,13 +260,30 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { /// used. //

Re: [I] Release DataFusion `46.0.0` [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #14123: URL: https://github.com/apache/datafusion/issues/14123#issuecomment-2706188668 > [@alamb](https://github.com/alamb) i will catch a flight today and have a short vacation, do you have time to do the final release? Yes, absolutely -- thank you for all yo

Re: [I] [Docs] Add info on type system (DFSchema/Schema) to user guide [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #7724: URL: https://github.com/apache/datafusion/issues/7724#issuecomment-2706224746 > Hi, I noticed this ticket hasn’t had updates for a few months. I’d love to work on it—would it be possible to assign it to me? Done! Thank you! -- This is an automated me

Re: [I] PushDownFilter is eliminated with leaf Extension nodes [datafusion]

2025-03-07 Thread via GitHub
sarvagyad2604 commented on issue #15047: URL: https://github.com/apache/datafusion/issues/15047#issuecomment-2706022079 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Handle columns in with_new_exprs with a Join [datafusion]

2025-03-07 Thread via GitHub
delamarch3 commented on PR #15055: URL: https://github.com/apache/datafusion/pull/15055#issuecomment-2706220468 Thanks for the reviews, I've pushed up your suggestions @jonahgao @niebayes -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Blog for DataFusion 46.0.0 [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #15053: URL: https://github.com/apache/datafusion/issues/15053#issuecomment-2706223772 Thank you! I will love to help write it too! Just let me know! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Fix array_has_all and array_has_any with empty array [datafusion]

2025-03-07 Thread via GitHub
Weijun-H commented on code in PR #15039: URL: https://github.com/apache/datafusion/pull/15039#discussion_r1985179602 ## datafusion/functions-nested/src/array_has.rs: ## @@ -439,6 +439,16 @@ fn array_has_all_and_any_dispatch( ) -> Result { let haystack = as_generic_list_arr

Re: [I] `security_audit` CI check is failing [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #15062: URL: https://github.com/apache/datafusion/issues/15062#issuecomment-2706349318 ``` rate: generational-arena Version: 0.2.9 Warning: unmaintained Title: `generational-arena` is unmaintained Date: 2024-02-11 ID:RUSTSEC-

Re: [PR] chore(deps): bump sqllogictest from 0.27.2 to 0.28.0 [datafusion]

2025-03-07 Thread via GitHub
alamb commented on PR #15060: URL: https://github.com/apache/datafusion/pull/15060#issuecomment-2706360863 security audit failure is due to this one I think: - https://github.com/apache/datafusion/issues/15062 -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] Minor: fix CI to make the sqllogic testing result consistent [datafusion]

2025-03-07 Thread via GitHub
alamb commented on PR #15059: URL: https://github.com/apache/datafusion/pull/15059#issuecomment-2706361369 security audit failure is due to this one I think: - https://github.com/apache/datafusion/issues/15062 -- This is an automated message from the Apache Git Service. To respond to th

[I] `security_audit` CI check is failing [datafusion]

2025-03-07 Thread via GitHub
alamb opened a new issue, #15062: URL: https://github.com/apache/datafusion/issues/15062 ### Describe the bug ![Image](https://github.com/user-attachments/assets/05157089-b020-4594-a591-1ebdd80a5393) Here is an example: https://github.com/apache/datafusion/actions/runs/

[PR] Update ring to v0.17.13 [datafusion]

2025-03-07 Thread via GitHub
alamb opened a new pull request, #15063: URL: https://github.com/apache/datafusion/pull/15063 ## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/15062 ## Rationale for this change there is a vulnerability in old versions of rin

Re: [I] Incorrect documentation for docker deployment [datafusion-ballista]

2025-03-07 Thread via GitHub
milenkovicm commented on issue #1198: URL: https://github.com/apache/datafusion-ballista/issues/1198#issuecomment-2706356193 maybe we should move https://github.com/apache/datafusion-ballista/blob/23a5354d57d84e2413e27e801ee170038d59abc7/.github/workflows/rust.yml#L297-L325 to dock

Re: [I] Implement `tree` explain for `AggregateExec` [datafusion]

2025-03-07 Thread via GitHub
alamb commented on issue #15024: URL: https://github.com/apache/datafusion/issues/15024#issuecomment-2706342062 @zebsme how is this going? I noticed you took this and https://github.com/apache/datafusion/issues/15025 and https://github.com/apache/datafusion/issues/15026 If you aren'

Re: [PR] Minor: fix CI to make the sqllogic testing result consistent [datafusion]

2025-03-07 Thread via GitHub
alamb commented on PR #15059: URL: https://github.com/apache/datafusion/pull/15059#issuecomment-2706360512 security audit failure is due to this one I think: - https://github.com/apache/datafusion/issues/15062 -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] Minor: Improve documentation of `need_handle_count_bug` [datafusion]

2025-03-07 Thread via GitHub
comphead commented on code in PR #15050: URL: https://github.com/apache/datafusion/pull/15050#discussion_r1985364551 ## datafusion/optimizer/src/decorrelate.rs: ## @@ -56,10 +56,14 @@ pub struct PullUpCorrelatedExpr { /// Indicates if we encounter any correlated expression

Re: [I] Implement tree explain for `PartialSortExec` [datafusion]

2025-03-07 Thread via GitHub
irenjj commented on issue #15064: URL: https://github.com/apache/datafusion/issues/15064#issuecomment-2706615346 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] fix: Stop copying rust-toolchain to docker file [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove merged PR #1475: URL: https://github.com/apache/datafusion-comet/pull/1475 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

[I] Timeouts reading large files from object stores on slow connetions [datafusion]

2025-03-07 Thread via GitHub
alamb opened a new issue, #15067: URL: https://github.com/apache/datafusion/issues/15067 ### Describe the bug Related to - https://github.com/apache/datafusion/issues/12393 - https://github.com/apache/datafusion/pull/14286 - https://github.com/delta-io/delta-rs/issues/2595

Re: [PR] test: fix Spark 3.5 tests [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove commented on PR #1482: URL: https://github.com/apache/datafusion-comet/pull/1482#issuecomment-2706930850 Thanks for creating this @kazuyukitanimura. The following test is currently failing. I can help look into this. ``` Filters should be pushed down for vectorized Parqu

Re: [PR] Document guidelines for physical operator yielding [datafusion]

2025-03-07 Thread via GitHub
carols10cents commented on code in PR #15030: URL: https://github.com/apache/datafusion/pull/15030#discussion_r1985212925 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -260,13 +260,30 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { /// used. //

Re: [PR] Document guidelines for physical operator yielding [datafusion]

2025-03-07 Thread via GitHub
carols10cents commented on PR #15030: URL: https://github.com/apache/datafusion/pull/15030#issuecomment-2706663774 I just pushed some more commits addressing some comments; there is one TODO commit in there that I will update once https://github.com/apache/datafusion/pull/15054 has been mer

Re: [PR] Document guidelines for physical operator yielding [datafusion]

2025-03-07 Thread via GitHub
carols10cents commented on code in PR #15030: URL: https://github.com/apache/datafusion/pull/15030#discussion_r1985206291 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -260,13 +260,30 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { /// used. //

Re: [PR] Document guidelines for physical operator yielding [datafusion]

2025-03-07 Thread via GitHub
carols10cents commented on code in PR #15030: URL: https://github.com/apache/datafusion/pull/15030#discussion_r1985207813 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -260,13 +260,30 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { /// used. //

Re: [PR] fix: mark ScalarUDFImpl::invoke_batch as deprecated [datafusion]

2025-03-07 Thread via GitHub
Blizzara commented on code in PR #15049: URL: https://github.com/apache/datafusion/pull/15049#discussion_r1985164453 ## datafusion/functions/benches/character_length.rs: ## @@ -26,6 +28,8 @@ fn criterion_benchmark(c: &mut Criterion) { // All benches are single batch run wit

Re: [PR] Minor: fix CI to make the sqllogic testing result consistent [datafusion]

2025-03-07 Thread via GitHub
alamb merged PR #15059: URL: https://github.com/apache/datafusion/pull/15059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Fix CI to make the sqllogic testing result consistent [datafusion]

2025-03-07 Thread via GitHub
alamb closed issue #15058: Fix CI to make the sqllogic testing result consistent URL: https://github.com/apache/datafusion/issues/15058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Enable Dataframe to be converted into views which can be used in register_table [datafusion-python]

2025-03-07 Thread via GitHub
kosiew commented on code in PR #1016: URL: https://github.com/apache/datafusion-python/pull/1016#discussion_r1985237531 ## src/dataframe.rs: ## @@ -50,9 +52,79 @@ use crate::{ expr::{sort_expr::PySortExpr, PyExpr}, }; +// https://github.com/apache/datafusion-python/pull

  1   2   3   >