Re: [I] Attach `Diagnostic` to "more than one column in subquery" error [datafusion]

2025-03-09 Thread via GitHub
changsun20 commented on issue #14438: URL: https://github.com/apache/datafusion/issues/14438#issuecomment-2709581340 Hi @eliaperantoni, After investigating this issue, here are my initial thoughts on implementation: The most straightforward approach would be to add a new `span`

Re: [PR] chore(deps): bump substrait from 0.53.2 to 0.54.0 [datafusion]

2025-03-09 Thread via GitHub
jonahgao commented on code in PR #15043: URL: https://github.com/apache/datafusion/pull/15043#discussion_r1986660443 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -455,6 +455,7 @@ pub fn to_substrait_plan(plan: &LogicalPlan, state: &SessionState) -> Result Result

[I] ## Implement tree explain for `ArrowFileSink` [datafusion]

2025-03-09 Thread via GitHub
shruti2522 opened a new issue, #15112: URL: https://github.com/apache/datafusion/issues/15112 ### Is your feature request related to a problem or challenge? todo for #14914 ### Describe the solution you'd like _No response_ ### Describe alternatives you've consider

Re: [PR] SET with a list of comma separated assignments [datafusion-sqlparser-rs]

2025-03-09 Thread via GitHub
MohamedAbdeen21 commented on code in PR #1757: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1757#discussion_r1986585508 ## src/ast/mod.rs: ## @@ -2947,6 +2947,17 @@ pub enum Statement { variables: OneOrManyWithParens, value: Vec, }, + +

Re: [PR] chore: Add `native_iceberg_compat` CI checks [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on PR #1487: URL: https://github.com/apache/datafusion-comet/pull/1487#issuecomment-2709280755 > lgtm. Are these the only tests that are failing now? These are the only tests failing that were not already skipped in https://github.com/apache/datafusion-comet/pull/

Re: [I] Make it easier to run TPCH queries with datafusion-cli [datafusion]

2025-03-09 Thread via GitHub
clflushopt commented on issue #14608: URL: https://github.com/apache/datafusion/issues/14608#issuecomment-2709364715 Hey @alamb as of today I have a fully working implementation that matches Apache Trino and OLTPBenchmark's, I found the issue I mentionned in the message above which was due

Re: [I] [EPIC] Complete `SQL EXPLAIN` Tree Rendering [datafusion]

2025-03-09 Thread via GitHub
irenjj commented on issue #14914: URL: https://github.com/apache/datafusion/issues/14914#issuecomment-2709297384 > Can make `tree` a subcommand of `explain`, like `explain tree `. It looks like this is something that #15021 is going to handle.👀 -- This is an automated message fr

Re: [I] Implement `tree` explain for `ProjectionExec` [datafusion]

2025-03-09 Thread via GitHub
Standing-Man commented on issue #15023: URL: https://github.com/apache/datafusion/issues/15023#issuecomment-2709301867 Hi @alamb, this issue has been completed and should be closed to avoid any ambiguity, thank you. -- This is an automated message from the Apache Git Service. To respond t

[I] docs: typo in docs of ClickHouse column option MATERIALIZED [datafusion-sqlparser-rs]

2025-03-09 Thread via GitHub
tv42 opened a new issue, #1760: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1760 https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.ColumnOption.html#variant.Materialized sqlparser-rs docs say > `MATERIALIZE Syntax: b INT MATERIALIZE (a + 1)` That's m

[I] docs: ColumnOption::ForeignKey docs have wrong SQL syntax [datafusion-sqlparser-rs]

2025-03-09 Thread via GitHub
tv42 opened a new issue, #1761: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1761 https://docs.rs/sqlparser/0.55.0/sqlparser/ast/enum.ColumnOption.html#variant.ForeignKey says > A referential integrity constraint (`[FOREIGN KEY REFERENCES () { [ON DELETE ] [O

Re: [I] Attach `Diagnostic` to "more than one column in subquery" error [datafusion]

2025-03-09 Thread via GitHub
changsun20 commented on issue #14438: URL: https://github.com/apache/datafusion/issues/14438#issuecomment-2709239602 Hi @irenjj , I noticed this issue was assigned to you a few weeks ago. May I check if there's any progress? If you're open to collaboration, I'd be happy to contribute a P

Re: [PR] refactor: use TypeSignature::Coercible for crypto functions [datafusion]

2025-03-09 Thread via GitHub
jayzhan211 commented on code in PR #14826: URL: https://github.com/apache/datafusion/pull/14826#discussion_r1986495558 ## datafusion/functions/src/crypto/sha512.rs: ## @@ -52,11 +56,16 @@ impl Default for SHA512Func { impl SHA512Func { pub fn new() -> Self { -use

Re: [PR] refactor: use TypeSignature::Coercible for crypto functions [datafusion]

2025-03-09 Thread via GitHub
jayzhan211 commented on code in PR #14826: URL: https://github.com/apache/datafusion/pull/14826#discussion_r1986495558 ## datafusion/functions/src/crypto/sha512.rs: ## @@ -52,11 +56,16 @@ impl Default for SHA512Func { impl SHA512Func { pub fn new() -> Self { -use

Re: [PR] refactor: use TypeSignature::Coercible for crypto functions [datafusion]

2025-03-09 Thread via GitHub
jayzhan211 commented on code in PR #14826: URL: https://github.com/apache/datafusion/pull/14826#discussion_r1986494416 ## datafusion/functions/src/crypto/basic.rs: ## @@ -342,22 +367,27 @@ pub fn digest_process( DataType::LargeBinary => { digest_alg

Re: [PR] refactor: use TypeSignature::Coercible for crypto functions [datafusion]

2025-03-09 Thread via GitHub
jayzhan211 commented on code in PR #14826: URL: https://github.com/apache/datafusion/pull/14826#discussion_r1986494305 ## datafusion/functions/src/crypto/basic.rs: ## @@ -328,6 +322,37 @@ impl DigestAlgorithm { } } } + +pub fn digest_binary_array_i

Re: [PR] implement tree rendering for StreamingTableExec [datafusion]

2025-03-09 Thread via GitHub
Standing-Man commented on code in PR #15085: URL: https://github.com/apache/datafusion/pull/15085#discussion_r1986479692 ## datafusion/physical-plan/src/streaming.rs: ## @@ -209,8 +209,27 @@ impl DisplayAs for StreamingTableExec { Ok(()) }

Re: [I] Bad performance on wide tables (1000+ columns) [datafusion]

2025-03-09 Thread via GitHub
Omega359 commented on issue #7698: URL: https://github.com/apache/datafusion/issues/7698#issuecomment-2709192220 After some work (I had to spin up an ec2 instance to finally get a flamegraph) I've finally gotten what I think is a good flamegraph of 25 calls to `create_physical_plan` for a d

Re: [PR] feat/improve ruff test coverage [datafusion-python]

2025-03-09 Thread via GitHub
timsaucer commented on PR #1055: URL: https://github.com/apache/datafusion-python/pull/1055#issuecomment-2709187047 @CrystalZhou0529 would you mind reviewing the changes I made to `udf` and `udaf`? -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Reject `RESPECT NULLS` and `IGNORE NULLS` for aggregate functions [datafusion]

2025-03-09 Thread via GitHub
huaxingao commented on code in PR #15014: URL: https://github.com/apache/datafusion/pull/15014#discussion_r1986441543 ## datafusion/sql/src/expr/function.rs: ## @@ -349,6 +349,12 @@ impl SqlToRel<'_, S> { } else { // User defined aggregate functions (UDAF)

Re: [PR] feat: implement tree explain for `ProjectionExec` [datafusion]

2025-03-09 Thread via GitHub
Standing-Man commented on code in PR #15082: URL: https://github.com/apache/datafusion/pull/15082#discussion_r1986474472 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -539,6 +539,158 @@ physical_plan 17)│ format: arrow │ 18)└

Re: [PR] Add object store support [datafusion-ray]

2025-03-09 Thread via GitHub
andygrove merged PR #78: URL: https://github.com/apache/datafusion-ray/pull/78 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafu

[I] [DISCUSSION] physical-plan-common crate [datafusion]

2025-03-09 Thread via GitHub
berkaysynnada opened a new issue, #15111: URL: https://github.com/apache/datafusion/issues/15111 Currently, the datasource crate depends on physical-plan, meaning that physical-plan cannot access datasource at all. This design seems strange because none of the physical-plan code can directl

Re: [I] Support SQL pipe operator [datafusion]

2025-03-09 Thread via GitHub
simonvandel commented on issue #14660: URL: https://github.com/apache/datafusion/issues/14660#issuecomment-2709091476 I started work on supporting the syntax in sqlparser-rs here https://github.com/apache/datafusion-sqlparser-rs/pull/1759 -- This is an automated message from the Apache Gi

[PR] Support some of pipe operators [datafusion-sqlparser-rs]

2025-03-09 Thread via GitHub
simonvandel opened a new pull request, #1759: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1759 Part of https://github.com/apache/datafusion-sqlparser-rs/issues/1758 Still missing (ran out of time today, can be done in follow-ups perhaps) - join - union|intersect|e

Re: [I] Implement tree explain for `PartialSortExec` [datafusion]

2025-03-09 Thread via GitHub
alamb closed issue #15064: Implement tree explain for `PartialSortExec` URL: https://github.com/apache/datafusion/issues/15064 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] Simpler / easier to see expressions in `tree` explain mode [datafusion]

2025-03-09 Thread via GitHub
irenjj commented on issue #15107: URL: https://github.com/apache/datafusion/issues/15107#issuecomment-2708903196 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[PR] enhance sql-using-python-udf example [datafusion-python]

2025-03-09 Thread via GitHub
Ritinikhil opened a new pull request, #1054: URL: https://github.com/apache/datafusion-python/pull/1054 [ ] Add comprehensive comments and documentation [ ] Implement multiple data registration methods for API compatibility Add version information printing for debugging [ ] Improve err

Re: [I] Expose global context [datafusion-python]

2025-03-09 Thread via GitHub
timsaucer commented on issue #1045: URL: https://github.com/apache/datafusion-python/issues/1045#issuecomment-2708991997 No, we have plenty of other points in the code where we us the | operator. Also, that issue is related, but not the topic of this issue. We also have a PR up that resolv

[I] CSV file path handling in substrait.py example [datafusion-python]

2025-03-09 Thread via GitHub
Ritinikhil opened a new issue, #1053: URL: https://github.com/apache/datafusion-python/issues/1053 - Add cross-platform path handling using os.path - Add error handling for CSV file registration -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Implement tree explain for AggregateExec [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15103: URL: https://github.com/apache/datafusion/pull/15103#discussion_r1986382181 ## datafusion/physical-plan/src/aggregates/mod.rs: ## @@ -809,8 +809,54 @@ impl DisplayAs for AggregateExec { } } DisplayF

[I] TPC-H benchmark does not run q15 [datafusion-ray]

2025-03-09 Thread via GitHub
andygrove opened a new issue, #81: URL: https://github.com/apache/datafusion-ray/issues/81 The current tpcbench.py runs the first query in each file, so is only running the `CREATE VIEW` from q15 and is not running the `SELECT` query. -- This is an automated message from the Apache Git Se

Re: [I] Expose global context [datafusion-python]

2025-03-09 Thread via GitHub
Spaarsh commented on issue #1045: URL: https://github.com/apache/datafusion-python/issues/1045#issuecomment-2708982762 Should make a PR that doesn't use the ```|``` operator? We simply need to use the ```Union``` function instead. I already have that code running locally. -- This is an a

Re: [PR] Perform hashing in CollectLeft HashJoin in parallel [datafusion]

2025-03-09 Thread via GitHub
ctsk closed pull request #14234: Perform hashing in CollectLeft HashJoin in parallel URL: https://github.com/apache/datafusion/pull/14234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Auto run docker containers needed for tests [datafusion]

2025-03-09 Thread via GitHub
Omega359 commented on issue #15092: URL: https://github.com/apache/datafusion/issues/15092#issuecomment-2708932399 > I think [@Omega359](https://github.com/Omega359) did something similar here with sqllogictests for starting postgres: > > https://github.com/apache/datafusion/blob/mai

Re: [PR] refactor: use TypeSignature::Coercible for crypto functions [datafusion]

2025-03-09 Thread via GitHub
Chen-Yuan-Lai commented on code in PR #14826: URL: https://github.com/apache/datafusion/pull/14826#discussion_r1986347452 ## datafusion/functions/src/crypto/basic.rs: ## @@ -328,6 +322,37 @@ impl DigestAlgorithm { } } } + +pub fn digest_binary_arra

Re: [I] Change mapping of SQL `VARCHAR` from `Utf8` to `Utf8View` [datafusion]

2025-03-09 Thread via GitHub
zhuqi-lucas commented on issue #15096: URL: https://github.com/apache/datafusion/issues/15096#issuecomment-2708780286 I also testing the tcph, it seems also performance improvement which we use sql varchar to default to Utf8View. ```rust ./benchmarks/bench.sh compare main issue_14

Re: [PR] refactor: use TypeSignature::Coercible for crypto functions [datafusion]

2025-03-09 Thread via GitHub
Chen-Yuan-Lai commented on code in PR #14826: URL: https://github.com/apache/datafusion/pull/14826#discussion_r1986346839 ## datafusion/functions/src/crypto/basic.rs: ## @@ -342,22 +367,27 @@ pub fn digest_process( DataType::LargeBinary => { digest_

Re: [I] Datafusion binary size has been getting bigger [datafusion]

2025-03-09 Thread via GitHub
logan-keede commented on issue #13816: URL: https://github.com/apache/datafusion/issues/13816#issuecomment-2708902951 Optimizing binary size https://github.com/apache/datafusion/issues/13816 > [Optimizing DataFusion Binary Size](https://github.com/apache/datafusion/issues/13816) C

Re: [PR] implement tree explain for GlobalLimitExec [datafusion]

2025-03-09 Thread via GitHub
zjregee commented on code in PR #15100: URL: https://github.com/apache/datafusion/pull/15100#discussion_r1986344402 ## datafusion/physical-plan/src/limit.rs: ## @@ -109,8 +109,12 @@ impl DisplayAs for GlobalLimitExec { ) } DisplayFormat

Re: [PR] refactor: use TypeSignature::Coercible for crypto functions [datafusion]

2025-03-09 Thread via GitHub
Chen-Yuan-Lai commented on PR #14826: URL: https://github.com/apache/datafusion/pull/14826#issuecomment-2708900254 > Hi @jayzhan211, It seems all the CI checks were passed (including sqlogicaltest), but when I created and printed a table by datafusion-cli , I got empty result > > ```

Re: [PR] [POC] feat: Add datafusion-storage [datafusion]

2025-03-09 Thread via GitHub
Xuanwo commented on code in PR #15018: URL: https://github.com/apache/datafusion/pull/15018#discussion_r1986342269 ## datafusion/storage/src/write.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] [POC] feat: Add datafusion-storage [datafusion]

2025-03-09 Thread via GitHub
Xuanwo commented on code in PR #15018: URL: https://github.com/apache/datafusion/pull/15018#discussion_r1986342269 ## datafusion/storage/src/write.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] [POC] feat: Add datafusion-storage [datafusion]

2025-03-09 Thread via GitHub
Xuanwo commented on code in PR #15018: URL: https://github.com/apache/datafusion/pull/15018#discussion_r1986342160 ## datafusion/storage/src/read.rs: ## @@ -0,0 +1,149 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] [POC] feat: Add datafusion-storage [datafusion]

2025-03-09 Thread via GitHub
Xuanwo commented on code in PR #15018: URL: https://github.com/apache/datafusion/pull/15018#discussion_r1986341562 ## datafusion/storage/src/write.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [I] Stop running Rust tests in CI for all Java and Spark versions [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on issue #1480: URL: https://github.com/apache/datafusion-comet/issues/1480#issuecomment-2708892862 Through working on this, I discovered that some of the Rust tests actually do interact with the JVM, so we probably do still need to test with different Java versions. We

Re: [PR] chore: Add `native_iceberg_compat` CI checks [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on PR #1487: URL: https://github.com/apache/datafusion-comet/pull/1487#issuecomment-2708892412 @parthchandra Could you review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on PR #1481: URL: https://github.com/apache/datafusion-comet/pull/1481#issuecomment-2708890026 We don't need to test Rust with different Scala and Spark versions. I will revisit this PR another day. Thanks for the reviews, @kazuyukitanimura. -- This is an automated me

Re: [PR] Config: Add support default sql varchar to view types [datafusion]

2025-03-09 Thread via GitHub
zhuqi-lucas commented on code in PR #15104: URL: https://github.com/apache/datafusion/pull/15104#discussion_r1986337572 ## datafusion/core/tests/dataframe/mod.rs: ## @@ -5673,3 +5673,43 @@ async fn test_fill_null_all_columns() -> Result<()> { assert_batches_sorted_eq!(expec

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on code in PR #1481: URL: https://github.com/apache/datafusion-comet/pull/1481#discussion_r1986330188 ## .github/workflows/pr_build.yml: ## @@ -40,12 +40,31 @@ env: RUST_VERSION: stable jobs: + linux-test-rust: +strategy: + matrix: +os

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on PR #1481: URL: https://github.com/apache/datafusion-comet/pull/1481#issuecomment-2708887119 @kazuyukitanimura It looks like I had misunderstood this. The rust tests actually do have some tests that interact with the JVM to load classes. I did not realize this until n

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove closed pull request #1481: chore: Reduce number of runs of Rust unit tests in CI URL: https://github.com/apache/datafusion-comet/pull/1481 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Config: Add support default sql varchar to view types [datafusion]

2025-03-09 Thread via GitHub
zhuqi-lucas commented on code in PR #15104: URL: https://github.com/apache/datafusion/pull/15104#discussion_r1986337689 ## datafusion/common/src/config.rs: ## @@ -252,6 +252,11 @@ config_namespace! { /// string length and thus DataFusion can not enforce such limits.

Re: [I] Upgrade to sqlparser 0.55.0 [datafusion]

2025-03-09 Thread via GitHub
jonahgao commented on issue #15071: URL: https://github.com/apache/datafusion/issues/15071#issuecomment-2708885625 > Do we need to distinguish between `Inner/Join`, `Left/LeftOuter`, and `Right/RightOuter` in `JoinType`, like this? I think we don't need to distinguish them as they

Re: [PR] Config: Add support default sql varchar to view types [datafusion]

2025-03-09 Thread via GitHub
zhuqi-lucas commented on PR #15104: URL: https://github.com/apache/datafusion/pull/15104#issuecomment-2708885055 > Thanks @zhuqi-lucas ! This looks great. I have a few comments. Let me know what you think Thank you @alamb for review, great suggestions. Addressed in latest PR. -- Th

[PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-09 Thread via GitHub
alan910127 opened a new pull request, #15110: URL: https://github.com/apache/datafusion/pull/15110 ## Which issue does this PR close? - Closes #15035. ## Rationale for this change ## What changes are included in this PR? - Add a new function for

Re: [PR] Implement tree explain for AggregateExec [datafusion]

2025-03-09 Thread via GitHub
zebsme commented on code in PR #15103: URL: https://github.com/apache/datafusion/pull/15103#discussion_r1986332402 ## datafusion/physical-plan/src/aggregates/mod.rs: ## @@ -809,8 +809,54 @@ impl DisplayAs for AggregateExec { } } Display

Re: [I] "Non Panic Task error" is not an internal error [datafusion]

2025-03-09 Thread via GitHub
Satyam018 commented on issue #15108: URL: https://github.com/apache/datafusion/issues/15108#issuecomment-2708871872 "take" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] "Non Panic Task error" is not an internal error [datafusion]

2025-03-09 Thread via GitHub
getChan commented on issue #15108: URL: https://github.com/apache/datafusion/issues/15108#issuecomment-2708867405 @Satyam018 thanks. you can assign yourself by comment "take" https://datafusion.apache.org/contributor-guide/index.html#open-contribution-and-assigning-tickets -- This is a

[PR] #15108 issue: "Non Panic Task error" is not an internal error [datafusion]

2025-03-09 Thread via GitHub
Satyam018 opened a new pull request, #15109: URL: https://github.com/apache/datafusion/pull/15109 ## Which issue does this PR close? The issue was to change the internal error to execuriton error - Closes #. ## Rationale for this change The main reason for

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on code in PR #1481: URL: https://github.com/apache/datafusion-comet/pull/1481#discussion_r1986331195 ## .github/workflows/pr_build.yml: ## @@ -40,12 +40,31 @@ env: RUST_VERSION: stable jobs: + linux-test-rust: +strategy: + matrix: +os

Re: [I] "Non Panic Task error" is not an internal error [datafusion]

2025-03-09 Thread via GitHub
Satyam018 commented on issue #15108: URL: https://github.com/apache/datafusion/issues/15108#issuecomment-2708871819 @getChan sure thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on code in PR #1481: URL: https://github.com/apache/datafusion-comet/pull/1481#discussion_r1986330188 ## .github/workflows/pr_build.yml: ## @@ -40,12 +40,31 @@ env: RUST_VERSION: stable jobs: + linux-test-rust: +strategy: + matrix: +os

Re: [PR] Implement tree explain for `NestedLoopJoinExec`, `CrossJoinExec`, `So… [datafusion]

2025-03-09 Thread via GitHub
irenjj commented on PR #15081: URL: https://github.com/apache/datafusion/pull/15081#issuecomment-2708868217 Thanks @alamb for your review, have resolved conflicts, PTAL :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on code in PR #1481: URL: https://github.com/apache/datafusion-comet/pull/1481#discussion_r1986329465 ## .github/workflows/pr_build.yml: ## @@ -182,20 +207,29 @@ jobs: with: rust-version: ${{env.RUST_VERSION}} jdk-version: ${{ m

Re: [PR] chore: Reduce number of runs of Rust unit tests in CI [datafusion-comet]

2025-03-09 Thread via GitHub
andygrove commented on code in PR #1481: URL: https://github.com/apache/datafusion-comet/pull/1481#discussion_r1986329258 ## .github/workflows/pr_build.yml: ## @@ -40,12 +40,31 @@ env: RUST_VERSION: stable jobs: + linux-test-rust: +strategy: + matrix: +os

Re: [I] Enable `used_underscore_binding` clippy lint [datafusion]

2025-03-09 Thread via GitHub
ding-young commented on issue #14649: URL: https://github.com/apache/datafusion/issues/14649#issuecomment-2708867108 @Standing-Man I’ve already added a lint to suppress the conflict (the second option) locally, but I couldn’t submit a PR because I think the first approach(moving the logic a

Re: [PR] Implement tree explain for `NestedLoopJoinExec`, `CrossJoinExec`, `So… [datafusion]

2025-03-09 Thread via GitHub
irenjj commented on PR #15081: URL: https://github.com/apache/datafusion/pull/15081#issuecomment-2708864726 > Hi @irenjj -- this PR has some merge conflicts > > Any chance you can resolve them? > > Thanks again! Thanks @alamb for your review, have resolved conflicts, PTAL

Re: [PR] Minor: final `46.0.0` release tweaks: changelog + instructions [datafusion]

2025-03-09 Thread via GitHub
alamb commented on PR #15073: URL: https://github.com/apache/datafusion/pull/15073#issuecomment-2708860931 Thank you @jonahgao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Minor: final `46.0.0` release tweaks: changelog + instructions [datafusion]

2025-03-09 Thread via GitHub
alamb merged PR #15073: URL: https://github.com/apache/datafusion/pull/15073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Implement tree explain for `NestedLoopJoinExec`, `CrossJoinExec`, `So… [datafusion]

2025-03-09 Thread via GitHub
alamb commented on PR #15081: URL: https://github.com/apache/datafusion/pull/15081#issuecomment-2708858793 Hi @irenjj -- this PR has some merge conflicts Any chance you can resolve them? Thanks again! -- This is an automated message from the Apache Git Service. To respond to

Re: [I] "Non Panic Task error" is not an internal error [datafusion]

2025-03-09 Thread via GitHub
Satyam018 commented on issue #15108: URL: https://github.com/apache/datafusion/issues/15108#issuecomment-2708857228 @alamb I am working on this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Add ruff check for missing futures import [datafusion-python]

2025-03-09 Thread via GitHub
timsaucer opened a new pull request, #1052: URL: https://github.com/apache/datafusion-python/pull/1052 # Which issue does this PR close? None # Rationale for this change During testing a user discovered a missing futures import that broke importing datafusion. #

Re: [I] Implement `tree` explain for `SortExec` [datafusion]

2025-03-09 Thread via GitHub
alamb closed issue #15076: Implement `tree` explain for `SortExec` URL: https://github.com/apache/datafusion/issues/15076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Implement `tree` explain for `SortExec` [datafusion]

2025-03-09 Thread via GitHub
alamb merged PR #15077: URL: https://github.com/apache/datafusion/pull/15077 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Make it easier to run TPCH queries with datafusion-cli [datafusion]

2025-03-09 Thread via GitHub
alamb commented on issue #14608: URL: https://github.com/apache/datafusion/issues/14608#issuecomment-2708856273 > My goal is to potentially donate it to the [datafusion-contrib ](https://github.com/datafusion-contrib) organization and then keep maintaining it there this way we can coordinat

Re: [PR] Fix wasm32 build on version 46 [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15102: URL: https://github.com/apache/datafusion/pull/15102#discussion_r1986321168 ## datafusion/wasmtest/Cargo.toml: ## @@ -45,7 +45,7 @@ chrono = { version = "0.4", features = ["wasmbind"] } # all the `std::fmt` and `std::panicking` infrastructu

Re: [PR] Reject `RESPECT NULLS` and `IGNORE NULLS` for aggregate functions [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15014: URL: https://github.com/apache/datafusion/pull/15014#discussion_r1986319445 ## datafusion/sql/src/expr/function.rs: ## @@ -349,6 +349,12 @@ impl SqlToRel<'_, S> { } else { // User defined aggregate functions (UDAF) have

Re: [PR] fix: unparse for subqueryalias [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15068: URL: https://github.com/apache/datafusion/pull/15068#discussion_r1986318680 ## datafusion/sql/src/unparser/plan.rs: ## @@ -984,11 +984,18 @@ impl Unparser<'_> { Ok(Some(builder.build()?)) } LogicalP

Re: [PR] Fix wasm32 build on version 46 [datafusion]

2025-03-09 Thread via GitHub
XiangpengHao commented on code in PR #15102: URL: https://github.com/apache/datafusion/pull/15102#discussion_r1986318574 ## datafusion/wasmtest/Cargo.toml: ## @@ -45,7 +45,7 @@ chrono = { version = "0.4", features = ["wasmbind"] } # all the `std::fmt` and `std::panicking` infra

Re: [PR] Implement tree explain for PartialSortExec [datafusion]

2025-03-09 Thread via GitHub
alamb commented on PR #15066: URL: https://github.com/apache/datafusion/pull/15066#issuecomment-2708828878 Thanks again @irenjj -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[I] "Non Panic Task error" is not an internal error [datafusion]

2025-03-09 Thread via GitHub
alamb opened a new issue, #15108: URL: https://github.com/apache/datafusion/issues/15108 ### Describe the bug This code is marked as an [internal error ](https://docs.rs/datafusion/latest/datafusion/common/enum.DataFusionError.html#variant.Internal)(meaning it should not happen unles

Re: [I] Internal error: Non Panic Task error: task 113 was cancelled. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker [datafu

2025-03-09 Thread via GitHub
alamb commented on issue #15065: URL: https://github.com/apache/datafusion/issues/15065#issuecomment-2708845343 I also filed a ticket to fix the error type - https://github.com/apache/datafusion/issues/15108 -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Simpler / easier to see expressions in tree explain mode [datafusion]

2025-03-09 Thread via GitHub
alamb commented on issue #15107: URL: https://github.com/apache/datafusion/issues/15107#issuecomment-2708836074 If anyone is interested in this I think the first thing to do would be to get a function like this working: ```rust fn simple_display(expr: &dyn PhysicalExpr) -> String

Re: [I] Internal error: Non Panic Task error: task 113 was cancelled. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker [datafu

2025-03-09 Thread via GitHub
alamb commented on issue #15065: URL: https://github.com/apache/datafusion/issues/15065#issuecomment-2708843380 > Thanks again, I think I've found the reason. Out of curiosity, what did you find? -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] implement tree explain for GlobalLimitExec [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15100: URL: https://github.com/apache/datafusion/pull/15100#discussion_r1986316154 ## datafusion/physical-plan/src/limit.rs: ## @@ -109,8 +109,12 @@ impl DisplayAs for GlobalLimitExec { ) } DisplayFormatTy

Re: [PR] Implement `tree` explain for `SortExec` [datafusion]

2025-03-09 Thread via GitHub
alamb commented on PR #15077: URL: https://github.com/apache/datafusion/pull/15077#issuecomment-2708842413 I merged up to resolve a conflict -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Internal error: Non Panic Task error: task 113 was cancelled. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker [datafu

2025-03-09 Thread via GitHub
chenquan closed issue #15065: Internal error: Non Panic Task error: task 113 was cancelled. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker URL: https://github.com/apache/datafusion/issues/15065 -- This is an automate

Re: [PR] Config: Add support default sql varchar to view types [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15104: URL: https://github.com/apache/datafusion/pull/15104#discussion_r1986315228 ## datafusion/core/tests/dataframe/mod.rs: ## @@ -5673,3 +5673,43 @@ async fn test_fill_null_all_columns() -> Result<()> { assert_batches_sorted_eq!(expected, &

Re: [PR] feat: implement tree explain for `ProjectionExec` [datafusion]

2025-03-09 Thread via GitHub
alamb merged PR #15082: URL: https://github.com/apache/datafusion/pull/15082 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Implement tree explain for AggregateExec [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15103: URL: https://github.com/apache/datafusion/pull/15103#discussion_r1986313731 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -124,25 +124,39 @@ logical_plan physical_plan 01)┌───┐ 02)│ AggregateE

Re: [PR] feat: implement tree explain for `ProjectionExec` [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15082: URL: https://github.com/apache/datafusion/pull/15082#discussion_r1986312553 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -539,6 +539,158 @@ physical_plan 17)│ format: arrow │ 18)└───┘

[I] Simpler / easier to see expressions in tree explain mode [datafusion]

2025-03-09 Thread via GitHub
alamb opened a new issue, #15107: URL: https://github.com/apache/datafusion/issues/15107 ### Is your feature request related to a problem or challenge? - Part of https://github.com/apache/datafusion/issues/14914 Thanks to @Standing-Man , tree explains are looking nice - https

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-03-09 Thread via GitHub
shehabgamin commented on PR #14392: URL: https://github.com/apache/datafusion/pull/14392#issuecomment-2708820627 > > > Would you have bandwidth to help with adding some initial tests in the Comet repo? I think if we have some examples then it will be easier for others to contribute. > >

Re: [PR] Add `insta` / snapshot testing to CLI & set up AWS mock [datafusion]

2025-03-09 Thread via GitHub
alamb commented on PR #13672: URL: https://github.com/apache/datafusion/pull/13672#issuecomment-2708825239 I plan to merge this on Tuesday unless anyone needs additional time to review. I may then organize some tickets to port existing tests to use the new framework (especially as we

Re: [PR] Implement tree explain for PartialSortExec [datafusion]

2025-03-09 Thread via GitHub
alamb merged PR #15066: URL: https://github.com/apache/datafusion/pull/15066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] [POC] feat: Add datafusion-storage [datafusion]

2025-03-09 Thread via GitHub
tustvold commented on PR #15018: URL: https://github.com/apache/datafusion/pull/15018#issuecomment-2708822164 So I've not had time to look in huge detail, and would echo Andrew's concerns around AsyncWrite and friends, ObjectStore intentionally is not formulated in terms of them. How

Re: [I] Implement `tree` explain for `JsonSink` [datafusion]

2025-03-09 Thread via GitHub
irenjj commented on issue #15089: URL: https://github.com/apache/datafusion/issues/15089#issuecomment-2708813611 @Shreyaskr1409 Maybe you can refer to the test cases in `repartition_scan.slt`, like ``` explain COPY (VALUES (1), (2), (3), (4), (5)) TO 'test_files/scratch/repartit

Re: [PR] [POC] feat: Add datafusion-storage [datafusion]

2025-03-09 Thread via GitHub
alamb commented on PR #15018: URL: https://github.com/apache/datafusion/pull/15018#issuecomment-2708808553 FYI @crepererum @jayzhan211 @tustvold @berkaysynnada -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] [POC] feat: Add datafusion-storage [datafusion]

2025-03-09 Thread via GitHub
alamb commented on code in PR #15018: URL: https://github.com/apache/datafusion/pull/15018#discussion_r1986295023 ## datafusion/storage/src/write.rs: ## @@ -0,0 +1,79 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. S

Re: [I] Support User-Defined Sorting [datafusion]

2025-03-09 Thread via GitHub
tobixdev commented on issue #14828: URL: https://github.com/apache/datafusion/issues/14828#issuecomment-2708786065 Here is a start for a discussing a possible implementation: [15106](https://github.com/apache/datafusion/pull/15106) -- This is an automated message from the Apache Git Servi

  1   2   >