Re: [PR] Fix build failure caused by new `CoalescePartitionsExec::with_fetch` method [datafusion]

2025-04-24 Thread via GitHub
lewiszlw merged PR #15849: URL: https://github.com/apache/datafusion/pull/15849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

[PR] Handle missing login in changelog generate script [datafusion-sqlparser-rs]

2025-04-24 Thread via GitHub
iffyio opened a new pull request, #1823: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1823 Noticed during #1822 that the script initially failed due to [this commit](https://github.com/apache/datafusion-sqlparser-rs/commit/10cf7c164ee0bae8a71e1d8f0af5851b96465692) no longe

[PR] Prepare for 0.56.0 release: Version and CHANGELOG [datafusion-sqlparser-rs]

2025-04-24 Thread via GitHub
iffyio opened a new pull request, #1822: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1822 Part of - #1756 1. Updates the version 2. Adds a changelog. See [rendered version here](https://github.com/validio-io/sqlparser-rs/blob/prepare_0.56.0/changelog/0.56.0.

Re: [PR] fix: fetch is missed during EnforceDistribution [datafusion]

2025-04-24 Thread via GitHub
xudong963 closed pull request #14207: fix: fetch is missed during EnforceDistribution URL: https://github.com/apache/datafusion/pull/14207 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[I] `select count(distinct col)` query doesn't go the specialized distinct accumulator [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 opened a new issue, #15850: URL: https://github.com/apache/datafusion/issues/15850 ### Is your feature request related to a problem or challenge? ``` statement count 0 create table t(a int) as values (1), (2); query I select count(distinct a) from t;

[PR] Fix build failure caused by new `CoalescePartitionsExec::with_fetch` [datafusion]

2025-04-24 Thread via GitHub
lewiszlw opened a new pull request, #15849: URL: https://github.com/apache/datafusion/pull/15849 ## Which issue does this PR close? - Closes #. ## Rationale for this change There was a new pr (https://github.com/apache/datafusion/pull/15842) merged. ##

Re: [PR] Update extending-operators.md [datafusion]

2025-04-24 Thread via GitHub
Adez017 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2829373011 Hi @xudong963 , i think it is ready , give it a check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Fix `CoalescePartitionsExec` proto serialization [datafusion]

2025-04-24 Thread via GitHub
lewiszlw merged PR #15824: URL: https://github.com/apache/datafusion/pull/15824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
crystalxyz commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059575045 ## docs/source/library-user-guide/upgrading.md: ## @@ -19,6 +19,33 @@ # Upgrade Guides +## DataFusion `48.0.0` + +### Processing `Field` instead of `DataTy

Re: [PR] Update extending-operators.md [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2829380416 You can refer to the doc: https://datafusion.apache.org/library-user-guide/custom-table-providers.html. It should contain the real code https://github.com/apache/datafusion/

Re: [I] Optimized spill file format [datafusion]

2025-04-24 Thread via GitHub
getChan commented on issue #14078: URL: https://github.com/apache/datafusion/issues/14078#issuecomment-2829344168 update : skip validation is applied when reading spill files. by #15454 -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Fix: fetch is missing in `plan_with_order_breaking_variants` method [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on PR #15842: URL: https://github.com/apache/datafusion/pull/15842#issuecomment-2829300863 Thanks @xudong963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Fix: fetch is missing in `plan_with_order_breaking_variants` method [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 merged PR #15842: URL: https://github.com/apache/datafusion/pull/15842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] Fix `CoalescePartitionsExec` proto serialization [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on code in PR #15824: URL: https://github.com/apache/datafusion/pull/15824#discussion_r2059510351 ## datafusion/physical-optimizer/src/enforce_distribution.rs: ## @@ -1036,9 +1036,9 @@ pub fn replace_order_preserving_variants( if is_sort_preserving_merge

Re: [I] `physical_planner::tests::test_optimization_invariant_checker` fails on `release-nonlto` profile but not `test` profile [datafusion]

2025-04-24 Thread via GitHub
qazxcdswe123 commented on issue #15848: URL: https://github.com/apache/datafusion/issues/15848#issuecomment-2829251167 I'm currently running it in `release` profile but it may take a while.. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[I] `physical_planner::tests::test_optimization_invariant_checker` fails on `release-nonlto` profile but not `dev` profile [datafusion]

2025-04-24 Thread via GitHub
qazxcdswe123 opened a new issue, #15848: URL: https://github.com/apache/datafusion/issues/15848 ### Describe the bug With `RUST_BACKTRACE=1 cargo test physical_planner::tests::test_optimization_invariant_checker --profile release-nonlto` ``` running 1 test test physical_

Re: [PR] Fix `CoalescePartitionsExec` proto serialization [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on code in PR #15824: URL: https://github.com/apache/datafusion/pull/15824#discussion_r2059464810 ## datafusion/physical-optimizer/src/enforce_distribution.rs: ## @@ -1036,9 +1036,9 @@ pub fn replace_order_preserving_variants( if is_sort_preserving_merge

Re: [PR] improve eliminate_outer_join rule [datafusion]

2025-04-24 Thread via GitHub
github-actions[bot] closed pull request #13249: improve eliminate_outer_join rule URL: https://github.com/apache/datafusion/pull/13249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[PR] deprecate schema expressions [datafusion]

2025-04-24 Thread via GitHub
ajita-asthana opened a new pull request, #15847: URL: https://github.com/apache/datafusion/pull/15847 ## Which issue does this PR close? - Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes t

Re: [I] [Experimental scans] schema adapter does not apply required schema for structs within lists [datafusion-comet]

2025-04-24 Thread via GitHub
comphead commented on issue #1681: URL: https://github.com/apache/datafusion-comet/issues/1681#issuecomment-2829102935 another test to reproduce ``` test("native reader - read a STRUCT subfield from ARRAY of STRUCTS") { testSingleLineQuery( """ | selec

Re: [PR] chore: match Maven plugin versions with Spark 3.5 [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove merged PR #1668: URL: https://github.com/apache/datafusion-comet/pull/1668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] fix: Fall back to Spark when reading Parquet files with structs inside arrays [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove closed pull request #1682: fix: Fall back to Spark when reading Parquet files with structs inside arrays URL: https://github.com/apache/datafusion-comet/pull/1682 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[PR] feat(benchmark): collect benchmarks for last 5 versions in line protocol format [datafusion]

2025-04-24 Thread via GitHub
saraghds opened a new pull request, #15846: URL: https://github.com/apache/datafusion/pull/15846 ## Which issue does this PR close? - Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested

Re: [PR] Feat/auto bench [datafusion]

2025-04-24 Thread via GitHub
saraghds closed pull request #15845: Feat/auto bench URL: https://github.com/apache/datafusion/pull/15845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

[PR] Feat/auto bench [datafusion]

2025-04-24 Thread via GitHub
saraghds opened a new pull request, #15845: URL: https://github.com/apache/datafusion/pull/15845 ## Which issue does this PR close? - Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested

[I] How to install ballista python package? [datafusion-ballista]

2025-04-24 Thread via GitHub
Wuerike opened a new issue, #1257: URL: https://github.com/apache/datafusion-ballista/issues/1257 Sorry for this (probably) dumb question, but I've couldn't figure out how to install de python package -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] fix: Fall back to Spark when reading Parquet files with structs inside arrays [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove commented on PR #1682: URL: https://github.com/apache/datafusion-comet/pull/1682#issuecomment-2828901727 Moving this to draft because many tests are failing now that were previously passing. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
timsaucer commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059247381 ## docs/source/library-user-guide/upgrading.md: ## @@ -19,6 +19,33 @@ # Upgrade Guides +## DataFusion `48.0.0` + +### Processing `Field` instead of `DataTyp

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
timsaucer commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059245151 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -71,11 +71,23 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash { /

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
timsaucer commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059245151 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -71,11 +71,23 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash { /

[PR] WIP: scalar UDFs with metadata [datafusion-python]

2025-04-24 Thread via GitHub
timsaucer opened a new pull request, #1110: URL: https://github.com/apache/datafusion-python/pull/1110 # Which issue does this PR close? Closes #. # Rationale for this change # What changes are included in this PR? # Are there any user-facing chang

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-04-24 Thread via GitHub
blaginin commented on code in PR #15793: URL: https://github.com/apache/datafusion/pull/15793#discussion_r2059229206 ## datafusion/common/src/format.rs: ## @@ -27,7 +27,3 @@ pub const DEFAULT_CAST_OPTIONS: CastOptions<'static> = CastOptions { safe: false, format_optio

Re: [PR] feat: More warning info for users [datafusion-comet]

2025-04-24 Thread via GitHub
hsiang-c commented on code in PR #1667: URL: https://github.com/apache/datafusion-comet/pull/1667#discussion_r2059163344 ## spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala: ## @@ -161,10 +161,28 @@ class CometSparkSessionExtensions }

[PR] fix: Fall back to Spark when reading Parquet files with structs inside arrays [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove opened a new pull request, #1682: URL: https://github.com/apache/datafusion-comet/pull/1682 ## Which issue does this PR close? Workaround for https://github.com/apache/datafusion-comet/issues/1681 ## Rationale for this change ## What changes are

[I] [Experimental scans] schema adapter is not fully recursive [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove opened a new issue, #1681: URL: https://github.com/apache/datafusion-comet/issues/1681 ### Describe the bug The following test currently fails: ```scala test("nested data - array of struct") { val data = (1 to 10).map(i => Tuple1(Seq(i -> s"val_$i")))

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2059092200 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

Re: [PR] chore: Update viable crates [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove merged PR #1677: URL: https://github.com/apache/datafusion-comet/pull/1677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Support more types when pruning Parquet data [datafusion]

2025-04-24 Thread via GitHub
etseidl commented on issue #15742: URL: https://github.com/apache/datafusion/issues/15742#issuecomment-2828518601 Fixed by #15764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Support more types when pruning Parquet data [datafusion]

2025-04-24 Thread via GitHub
etseidl closed issue #15742: Support more types when pruning Parquet data URL: https://github.com/apache/datafusion/issues/15742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add `DECLARE ... CURSOR FOR` support for SQL Server [datafusion-sqlparser-rs]

2025-04-24 Thread via GitHub
iffyio merged PR #1821: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1821 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

[PR] Fix `from_unixtime` function documentation [datafusion]

2025-04-24 Thread via GitHub
Viicos opened a new pull request, #15844: URL: https://github.com/apache/datafusion/pull/15844 ## Which issue does this PR close? - Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested?

Re: [PR] Added SQL Example for `Aggregate Functions` [datafusion]

2025-04-24 Thread via GitHub
Adez017 closed pull request #15778: Added SQL Example for `Aggregate Functions` URL: https://github.com/apache/datafusion/pull/15778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058898316 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "jem

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058901724 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "jem

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058898316 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "jem

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
parthchandra commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058887934 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "j

Re: [PR] feat(datafusion-functions-aggregate): add support for lists and other nested types in `min` and `max` [datafusion]

2025-04-24 Thread via GitHub
rluvaton commented on PR #13991: URL: https://github.com/apache/datafusion/pull/13991#issuecomment-2828299361 Sorry, I could not find time today, you can take this but please add me as co-author for my changes: [Creating a commit with multiple authors](https://docs.github.com/en/pull

[PR] feat: make execution_graph.stages() public [datafusion-ballista]

2025-04-24 Thread via GitHub
milenkovicm opened a new pull request, #1256: URL: https://github.com/apache/datafusion-ballista/pull/1256 # Which issue does this PR close? Closes #. # Rationale for this change this is follow up on ##1243 where I missed to name `stages()` public. Making stages public

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2058172660 ## datafusion/core/src/datasource/listing/table.rs: ## @@ -982,18 +980,6 @@ impl TableProvider for ListingTable { return Ok(TableProvide

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2058837968 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2058262957 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on PR #15769: URL: https://github.com/apache/datafusion/pull/15769#issuecomment-2828267340 I do think you make a good point of "can we keep the current thing and add the new one". It's worth a shot, at least to split the PR into two. And if that's too complicated or if we

Re: [PR] refactor filter pushdown apis [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2828261533 Sounds good happy to wait for your suggestions. And we can always iterate again! -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on PR #15769: URL: https://github.com/apache/datafusion/pull/15769#issuecomment-2828241765 > I'm not really sure how this degrades anything. The end result is the same, users won't see any difference. Logical planning results are changing. We are also using DF

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
parthchandra commented on PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#issuecomment-2828241291 Restarted the test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] refactor filter pushdown apis [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2828219078 I understand your concerns and motivations. Since you're leading this work, I'm happy to follow your direction. I'm sure we both aim to do what's right, even if our priorities

Re: [I] Cannot use Projection::new_from_schema to set parquet field ids. [datafusion]

2025-04-24 Thread via GitHub
init-js commented on issue #15837: URL: https://github.com/apache/datafusion/issues/15837#issuecomment-2828204628 I may be getting lost in the layers, I'm not well versed in datafusion yet, but when building the physical plan, I notice that the schema part of the Projection is ignore

Re: [PR] Fix `ILIKE` expression support in SQL unparser [datafusion]

2025-04-24 Thread via GitHub
comphead commented on PR #15820: URL: https://github.com/apache/datafusion/pull/15820#issuecomment-2827893961 > Correct, DataFusion was correctly handling Like and ILike - but DataFusion stores that as a single `Expr::Like ` with a boolean for whether its case insensitive. > > When t

[PR] chore: More details to `No UDF registered` error [datafusion]

2025-04-24 Thread via GitHub
comphead opened a new pull request, #15843: URL: https://github.com/apache/datafusion/pull/15843 ## Which issue does this PR close? - Closes #. ## Rationale for this change Currently when getting the error when custom UDF is not registered ``` called `Result

Re: [I] Spark SQL test failures in native_datafusion scan [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on issue #1545: URL: https://github.com/apache/datafusion-comet/issues/1545#issuecomment-2828029094 ``` core 1: Failed: Total 9138, Failed 38, Errors 0, Passed 9100, Ignored 292, Canceled 6 core 2: Failed: Total 2655, Failed 23, Errors 0, Passed 2632, Ignored 387

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#issuecomment-2827971141 The Spark SQL test does not seem related to the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
codecov-commenter commented on PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#issuecomment-2827961029 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1679?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] Make `Diagnostic` easy/convinient to attach by using macro and avoiding `map_err` [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15796: URL: https://github.com/apache/datafusion/pull/15796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] benchmarks: Read SessionConfig from Environment [datafusion]

2025-04-24 Thread via GitHub
comphead closed issue #15684: benchmarks: Read SessionConfig from Environment URL: https://github.com/apache/datafusion/issues/15684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] predicate pruning: support cast and try_cast for more types [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15764: URL: https://github.com/apache/datafusion/pull/15764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] predicate pruning: support cast and try_cast for more types [datafusion]

2025-04-24 Thread via GitHub
comphead commented on PR #15764: URL: https://github.com/apache/datafusion/pull/15764#issuecomment-2827908748 Thanks everyone! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Feature/benchmark config from env [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15782: URL: https://github.com/apache/datafusion/pull/15782 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Fix `ILIKE` expression support in SQL unparser [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15820: URL: https://github.com/apache/datafusion/pull/15820 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-04-24 Thread via GitHub
NevroHelios commented on PR #15841: URL: https://github.com/apache/datafusion/pull/15841#issuecomment-2827852025 Hi @xudong963 I made the changes and pushed the updates. Could you please initiate the CI workflows when you get a chance? -- This is an automated message from the Apache G

Re: [PR] Minor: fix potential flaky test in aggregate.slt [datafusion]

2025-04-24 Thread via GitHub
bikbov commented on PR #15829: URL: https://github.com/apache/datafusion/pull/15829#issuecomment-2827855525 Thanks @Weijun-H @xudong963 for review 🙂 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] refactor filter pushdown apis [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2827824299 I don't understand where this `fetch()` method would go or how it will help. I feel that what you are proposing is not necessarily any simpler. It requires understanding sever

Re: [PR] Minor: fix potential flaky test in aggregate.slt [datafusion]

2025-04-24 Thread via GitHub
Weijun-H commented on PR #15829: URL: https://github.com/apache/datafusion/pull/15829#issuecomment-2827813221 LGTM! Thanks @bikbov and @xudong963 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Potential flaky tests [datafusion]

2025-04-24 Thread via GitHub
Weijun-H closed issue #15789: Potential flaky tests URL: https://github.com/apache/datafusion/issues/15789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Minor: fix potential flaky test in aggregate.slt [datafusion]

2025-04-24 Thread via GitHub
Weijun-H merged PR #15829: URL: https://github.com/apache/datafusion/pull/15829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2058548271 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on PR #15769: URL: https://github.com/apache/datafusion/pull/15769#issuecomment-2827757032 I'm not really sure how this degrades anything. The end result is the same, users won't see any difference. What ListingTable does currently is misguided and wrong since it is

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.44.2 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1092: URL: https://github.com/apache/datafusion-python/pull/1092#issuecomment-2827673126 Looks like tokio is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] build(deps): bump arrow from 54.3.0 to 54.3.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1093: build(deps): bump arrow from 54.3.0 to 54.3.1 URL: https://github.com/apache/datafusion-python/pull/1093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] build(deps): bump pyo3-build-config from 0.23.4 to 0.24.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1094: URL: https://github.com/apache/datafusion-python/pull/1094#issuecomment-2827673666 Looks like pyo3-build-config is no longer updatable, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] build(deps): bump arrow from 54.3.0 to 54.3.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1093: URL: https://github.com/apache/datafusion-python/pull/1093#issuecomment-2827673009 Looks like arrow is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] build(deps): bump pyo3-build-config from 0.23.4 to 0.24.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1094: build(deps): bump pyo3-build-config from 0.23.4 to 0.24.1 URL: https://github.com/apache/datafusion-python/pull/1094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.44.2 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1092: build(deps): bump tokio from 1.43.0 to 1.44.2 URL: https://github.com/apache/datafusion-python/pull/1092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.43.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1098: build(deps): bump tokio from 1.43.0 to 1.43.1 URL: https://github.com/apache/datafusion-python/pull/1098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.43.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1098: URL: https://github.com/apache/datafusion-python/pull/1098#issuecomment-2827673057 Looks like tokio is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[PR] build(deps): bump pyo3 from 0.23.4 to 0.24.1 in /examples/ffi-table-provider [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] opened a new pull request, #1109: URL: https://github.com/apache/datafusion-python/pull/1109 Bumps [pyo3](https://github.com/pyo3/pyo3) from 0.23.4 to 0.24.1. Release notes Sourced from https://github.com/pyo3/pyo3/releases";>pyo3's releases. PyO3 0.24.1 T

Re: [PR] feat: update datafusion dependency 47 [datafusion-python]

2025-04-24 Thread via GitHub
timsaucer merged PR #1107: URL: https://github.com/apache/datafusion-python/pull/1107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [I] Support dot graph output in explain (analyze) [datafusion]

2025-04-24 Thread via GitHub
16pierre commented on issue #3606: URL: https://github.com/apache/datafusion/issues/3606#issuecomment-2827660759 Related but not 100% same topic as original issue - happy to take the conversation elsewhere if you prefer: I'm interested in Dalibo visualizations because the UI is quite

Re: [PR] refactor filter pushdown apis [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2827570314 > 1. Not requiring downcast matching of specific `ExecutionPlan`s (as discussed previously this could be resolved by adding a new method, but in the end both approaches requir

[PR] Fix: fetch is missing in plan_with_order_breaking_variants method [datafusion]

2025-04-24 Thread via GitHub
xudong963 opened a new pull request, #15842: URL: https://github.com/apache/datafusion/pull/15842 ## Which issue does this PR close? - Closes #. ## Rationale for this change Another `fetch` missing ## What changes are included in this PR?

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
crystalxyz commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2058241720 ## docs/source/library-user-guide/upgrading.md: ## @@ -19,6 +19,33 @@ # Upgrade Guides +## DataFusion `48.0.0` + +### Processing `Field` instead of `DataTy

Re: [PR] Partial fix for #1078 — [Add Dataframe display config] [datafusion-python]

2025-04-24 Thread via GitHub
kosiew commented on PR #1086: URL: https://github.com/apache/datafusion-python/pull/1086#issuecomment-2827426701 @timsaucer Should I move max_table_bytes min_table_rows: usize, max_table_rows_in_repr: usize, to the python DataFrameHtmlFormatter class as well?

Re: [PR] Fix: fetch is missing in `EnforceSorting` optimizer (two places) [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on PR #15822: URL: https://github.com/apache/datafusion/pull/15822#issuecomment-2827198842 Thank you @jayzhan211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Fix: fetch is missing in `EnforceSorting` optimizer (two places) [datafusion]

2025-04-24 Thread via GitHub
xudong963 merged PR #15822: URL: https://github.com/apache/datafusion/pull/15822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

[PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-04-24 Thread via GitHub
NevroHelios opened a new pull request, #15841: URL: https://github.com/apache/datafusion/pull/15841 … evaluation ## Which issue does this PR close? [Eliminate the function call in xxx_or (e.g. unwrap_or("".to_string()) #15802](https://github.com/apache/datafusion/issues/15802)

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
vadimpiven commented on PR #15811: URL: https://github.com/apache/datafusion/pull/15811#issuecomment-2827139938 Close in favour of https://github.com/apache/datafusion/pull/15825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Inline table scan drops projection [datafusion]

2025-04-24 Thread via GitHub
vadimpiven commented on issue #15810: URL: https://github.com/apache/datafusion/issues/15810#issuecomment-2827140507 Closed by https://github.com/apache/datafusion/pull/15825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
vadimpiven closed pull request #15811: Preserve projection for inline scan URL: https://github.com/apache/datafusion/pull/15811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Inline table scan drops projection [datafusion]

2025-04-24 Thread via GitHub
vadimpiven closed issue #15810: Inline table scan drops projection URL: https://github.com/apache/datafusion/issues/15810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on PR #15825: URL: https://github.com/apache/datafusion/pull/15825#issuecomment-2827031377 Thanks @xudong963 @vadimpiven -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Fix: fetch is missing in `EnforceSorting` optimizer (two places) [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on code in PR #15822: URL: https://github.com/apache/datafusion/pull/15822#discussion_r2058019883 ## datafusion/physical-optimizer/src/enforce_sorting/replace_with_order_preserving_variants.rs: ## @@ -138,6 +138,21 @@ fn plan_with_order_preserving_variants(

  1   2   >