[PR] Fix: fetch is missing in plan_with_order_breaking_variants method [datafusion]

2025-04-24 Thread via GitHub
xudong963 opened a new pull request, #15842: URL: https://github.com/apache/datafusion/pull/15842 ## Which issue does this PR close? - Closes #. ## Rationale for this change Another `fetch` missing ## What changes are included in this PR?

Re: [PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-04-24 Thread via GitHub
NevroHelios commented on PR #15841: URL: https://github.com/apache/datafusion/pull/15841#issuecomment-2827852025 Hi @xudong963 I made the changes and pushed the updates. Could you please initiate the CI workflows when you get a chance? -- This is an automated message from the Apache G

Re: [PR] Fix `ILIKE` expression support in SQL unparser [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15820: URL: https://github.com/apache/datafusion/pull/15820 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Feature/benchmark config from env [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15782: URL: https://github.com/apache/datafusion/pull/15782 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] predicate pruning: support cast and try_cast for more types [datafusion]

2025-04-24 Thread via GitHub
comphead commented on PR #15764: URL: https://github.com/apache/datafusion/pull/15764#issuecomment-2827908748 Thanks everyone! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[PR] chore: More details to `No UDF registered` error [datafusion]

2025-04-24 Thread via GitHub
comphead opened a new pull request, #15843: URL: https://github.com/apache/datafusion/pull/15843 ## Which issue does this PR close? - Closes #. ## Rationale for this change Currently when getting the error when custom UDF is not registered ``` called `Result

Re: [PR] feat: update datafusion dependency 47 [datafusion-python]

2025-04-24 Thread via GitHub
timsaucer merged PR #1107: URL: https://github.com/apache/datafusion-python/pull/1107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

[PR] build(deps): bump pyo3 from 0.23.4 to 0.24.1 in /examples/ffi-table-provider [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] opened a new pull request, #1109: URL: https://github.com/apache/datafusion-python/pull/1109 Bumps [pyo3](https://github.com/pyo3/pyo3) from 0.23.4 to 0.24.1. Release notes Sourced from https://github.com/pyo3/pyo3/releases";>pyo3's releases. PyO3 0.24.1 T

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.43.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1098: URL: https://github.com/apache/datafusion-python/pull/1098#issuecomment-2827673057 Looks like tokio is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.43.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1098: build(deps): bump tokio from 1.43.0 to 1.43.1 URL: https://github.com/apache/datafusion-python/pull/1098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.44.2 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1092: build(deps): bump tokio from 1.43.0 to 1.44.2 URL: https://github.com/apache/datafusion-python/pull/1092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] build(deps): bump pyo3-build-config from 0.23.4 to 0.24.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1094: build(deps): bump pyo3-build-config from 0.23.4 to 0.24.1 URL: https://github.com/apache/datafusion-python/pull/1094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] build(deps): bump arrow from 54.3.0 to 54.3.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1093: URL: https://github.com/apache/datafusion-python/pull/1093#issuecomment-2827673009 Looks like arrow is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] build(deps): bump pyo3-build-config from 0.23.4 to 0.24.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1094: URL: https://github.com/apache/datafusion-python/pull/1094#issuecomment-2827673666 Looks like pyo3-build-config is no longer updatable, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] build(deps): bump arrow from 54.3.0 to 54.3.1 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] closed pull request #1093: build(deps): bump arrow from 54.3.0 to 54.3.1 URL: https://github.com/apache/datafusion-python/pull/1093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] build(deps): bump tokio from 1.43.0 to 1.44.2 [datafusion-python]

2025-04-24 Thread via GitHub
dependabot[bot] commented on PR #1092: URL: https://github.com/apache/datafusion-python/pull/1092#issuecomment-2827673126 Looks like tokio is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Spark SQL test failures in native_datafusion scan [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on issue #1545: URL: https://github.com/apache/datafusion-comet/issues/1545#issuecomment-2828029094 ``` core 1: Failed: Total 9138, Failed 38, Errors 0, Passed 9100, Ignored 292, Canceled 6 core 2: Failed: Total 2655, Failed 23, Errors 0, Passed 2632, Ignored 387

Re: [PR] Fix: fetch is missing in `EnforceSorting` optimizer (two places) [datafusion]

2025-04-24 Thread via GitHub
xudong963 merged PR #15822: URL: https://github.com/apache/datafusion/pull/15822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] Fix: fetch is missing in `EnforceSorting` optimizer (two places) [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on PR #15822: URL: https://github.com/apache/datafusion/pull/15822#issuecomment-2827198842 Thank you @jayzhan211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
crystalxyz commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2058241720 ## docs/source/library-user-guide/upgrading.md: ## @@ -19,6 +19,33 @@ # Upgrade Guides +## DataFusion `48.0.0` + +### Processing `Field` instead of `DataTy

Re: [PR] Minor: fix potential flaky test in aggregate.slt [datafusion]

2025-04-24 Thread via GitHub
Weijun-H merged PR #15829: URL: https://github.com/apache/datafusion/pull/15829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] Potential flaky tests [datafusion]

2025-04-24 Thread via GitHub
Weijun-H closed issue #15789: Potential flaky tests URL: https://github.com/apache/datafusion/issues/15789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Re: [PR] Minor: fix potential flaky test in aggregate.slt [datafusion]

2025-04-24 Thread via GitHub
Weijun-H commented on PR #15829: URL: https://github.com/apache/datafusion/pull/15829#issuecomment-2827813221 LGTM! Thanks @bikbov and @xudong963 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] refactor filter pushdown apis [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2827824299 I don't understand where this `fetch()` method would go or how it will help. I feel that what you are proposing is not necessarily any simpler. It requires understanding sever

Re: [PR] Minor: fix potential flaky test in aggregate.slt [datafusion]

2025-04-24 Thread via GitHub
bikbov commented on PR #15829: URL: https://github.com/apache/datafusion/pull/15829#issuecomment-2827855525 Thanks @Weijun-H @xudong963 for review 🙂 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Support dot graph output in explain (analyze) [datafusion]

2025-04-24 Thread via GitHub
16pierre commented on issue #3606: URL: https://github.com/apache/datafusion/issues/3606#issuecomment-2827660759 Related but not 100% same topic as original issue - happy to take the conversation elsewhere if you prefer: I'm interested in Dalibo visualizations because the UI is quite

Re: [PR] Make `Diagnostic` easy/convinient to attach by using macro and avoiding `map_err` [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15796: URL: https://github.com/apache/datafusion/pull/15796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [I] benchmarks: Read SessionConfig from Environment [datafusion]

2025-04-24 Thread via GitHub
comphead closed issue #15684: benchmarks: Read SessionConfig from Environment URL: https://github.com/apache/datafusion/issues/15684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] predicate pruning: support cast and try_cast for more types [datafusion]

2025-04-24 Thread via GitHub
comphead merged PR #15764: URL: https://github.com/apache/datafusion/pull/15764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
codecov-commenter commented on PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#issuecomment-2827961029 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1679?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#issuecomment-2827971141 The Spark SQL test does not seem related to the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Partial fix for #1078 — [Add Dataframe display config] [datafusion-python]

2025-04-24 Thread via GitHub
kosiew commented on PR #1086: URL: https://github.com/apache/datafusion-python/pull/1086#issuecomment-2827426701 @timsaucer Should I move max_table_bytes min_table_rows: usize, max_table_rows_in_repr: usize, to the python DataFrameHtmlFormatter class as well?

Re: [PR] refactor filter pushdown apis [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2827570314 > 1. Not requiring downcast matching of specific `ExecutionPlan`s (as discussed previously this could be resolved by adding a new method, but in the end both approaches requir

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on PR #15769: URL: https://github.com/apache/datafusion/pull/15769#issuecomment-2827757032 I'm not really sure how this degrades anything. The end result is the same, users won't see any difference. What ListingTable does currently is misguided and wrong since it is

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2058548271 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2058262957 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

Re: [I] Cannot use Projection::new_from_schema to set parquet field ids. [datafusion]

2025-04-24 Thread via GitHub
init-js commented on issue #15837: URL: https://github.com/apache/datafusion/issues/15837#issuecomment-2828204628 I may be getting lost in the layers, I'm not well versed in datafusion yet, but when building the physical plan, I notice that the schema part of the Projection is ignore

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
parthchandra commented on PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#issuecomment-2828241291 Restarted the test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
parthchandra commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058887934 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "j

Re: [PR] chore: Update viable crates [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove merged PR #1677: URL: https://github.com/apache/datafusion-comet/pull/1677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2059092200 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

[PR] feat(benchmark): collect benchmarks for last 5 versions in line protocol format [datafusion]

2025-04-24 Thread via GitHub
saraghds opened a new pull request, #15846: URL: https://github.com/apache/datafusion/pull/15846 ## Which issue does this PR close? - Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested

Re: [I] Incorrect field indices for right‑side columns in Substrait ProjectRel after [datafusion]

2025-04-24 Thread via GitHub
chenkovsky commented on issue #15765: URL: https://github.com/apache/datafusion/issues/15765#issuecomment-2826836394 ```rust #[tokio::test] async fn test_substrait_post_join_project() -> Result<()> { let ctx = create_context().await?; let sql = " SELECT

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
vadimpiven commented on code in PR #15825: URL: https://github.com/apache/datafusion/pull/15825#discussion_r2057920398 ## datafusion/core/tests/execution/logical_plan.rs: ## @@ -96,3 +100,37 @@ where }; element } + +#[test] +fn inline_scan_projection_test() -> Result<

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
vadimpiven commented on PR #15811: URL: https://github.com/apache/datafusion/pull/15811#issuecomment-2826910009 Updated the test code so we check that the projection is applied, not how it is applied. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
vadimpiven commented on code in PR #15825: URL: https://github.com/apache/datafusion/pull/15825#discussion_r2057920398 ## datafusion/core/tests/execution/logical_plan.rs: ## @@ -96,3 +100,37 @@ where }; element } + +#[test] +fn inline_scan_projection_test() -> Result<

Re: [I] Inline table scan drops projection [datafusion]

2025-04-24 Thread via GitHub
vadimpiven closed issue #15810: Inline table scan drops projection URL: https://github.com/apache/datafusion/issues/15810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
vadimpiven closed pull request #15811: Preserve projection for inline scan URL: https://github.com/apache/datafusion/pull/15811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Inline table scan drops projection [datafusion]

2025-04-24 Thread via GitHub
vadimpiven commented on issue #15810: URL: https://github.com/apache/datafusion/issues/15810#issuecomment-2827140507 Closed by https://github.com/apache/datafusion/pull/15825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] chore(deps): bump pyo3 from 0.24.1 to 0.24.2 [datafusion]

2025-04-24 Thread via GitHub
dependabot[bot] opened a new pull request, #15838: URL: https://github.com/apache/datafusion/pull/15838 Bumps [pyo3](https://github.com/pyo3/pyo3) from 0.24.1 to 0.24.2. Release notes Sourced from https://github.com/pyo3/pyo3/releases";>pyo3's releases. PyO3 0.24.2 This i

Re: [PR] Fix: fetch is missing in `EnforceSorting` optimizer (two places) [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on code in PR #15822: URL: https://github.com/apache/datafusion/pull/15822#discussion_r2058019883 ## datafusion/physical-optimizer/src/enforce_sorting/replace_with_order_preserving_variants.rs: ## @@ -138,6 +138,21 @@ fn plan_with_order_preserving_variants(

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on PR #15825: URL: https://github.com/apache/datafusion/pull/15825#issuecomment-2827031377 Thanks @xudong963 @vadimpiven -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[I] Avro reader fails when query columns are reordered in SELECT statement [datafusion]

2025-04-24 Thread via GitHub
nantunes opened a new issue, #15839: URL: https://github.com/apache/datafusion/issues/15839 ### Describe the bug When querying an Avro file table in DataFusion, column selection works fine when columns are in schema order or a subset of columns in any order. However, if the column or

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 merged PR #15825: URL: https://github.com/apache/datafusion/pull/15825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] chore(deps): bump pyo3 from 0.24.1 to 0.24.2 [datafusion]

2025-04-24 Thread via GitHub
xudong963 merged PR #15838: URL: https://github.com/apache/datafusion/pull/15838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

[PR] fix(avro): Respect projection order in Avro reader [datafusion]

2025-04-24 Thread via GitHub
nantunes opened a new pull request, #15840: URL: https://github.com/apache/datafusion/pull/15840 ## Which issue does this PR close? - Fixes #15839 ## Rationale for this change When querying Avro files with columns in a different order than the original schema, the reader

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on code in PR #15811: URL: https://github.com/apache/datafusion/pull/15811#discussion_r2058012111 ## datafusion/core/tests/execution/logical_plan.rs: ## @@ -96,3 +99,32 @@ where }; element } + +#[test] +fn inline_scan_projection_test() -> Result<(

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on code in PR #15811: URL: https://github.com/apache/datafusion/pull/15811#discussion_r2058012111 ## datafusion/core/tests/execution/logical_plan.rs: ## @@ -96,3 +99,32 @@ where }; element } + +#[test] +fn inline_scan_projection_test() -> Result<(

Re: [PR] Minor: cleanup hash table after emit all [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on PR #15834: URL: https://github.com/apache/datafusion/pull/15834#issuecomment-2827040305 Thank @xudong963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Minor: cleanup hash table after emit all [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 merged PR #15834: URL: https://github.com/apache/datafusion/pull/15834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [I] Custom sort order for column [datafusion]

2025-04-24 Thread via GitHub
kosiew commented on issue #15819: URL: https://github.com/apache/datafusion/issues/15819#issuecomment-2826690238 Thanks for raising this! This is a very relevant use case—particularly in genomics, where the desired sort order of chromosome labels (e.g., `chr1`, `chr2`, `chrX`, `chrY`, `chrM

Re: [I] Eliminate the function call in `xxx_or (e.g. unwrap_or("".to_string())` [datafusion]

2025-04-24 Thread via GitHub
lifan-ake commented on issue #15802: URL: https://github.com/apache/datafusion/issues/15802#issuecomment-2826773861 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Preserve projection for inline scan [datafusion]

2025-04-24 Thread via GitHub
vadimpiven commented on PR #15811: URL: https://github.com/apache/datafusion/pull/15811#issuecomment-2827139938 Close in favour of https://github.com/apache/datafusion/pull/15825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-04-24 Thread via GitHub
NevroHelios opened a new pull request, #15841: URL: https://github.com/apache/datafusion/pull/15841 … evaluation ## Which issue does this PR close? [Eliminate the function call in xxx_or (e.g. unwrap_or("".to_string()) #15802](https://github.com/apache/datafusion/issues/15802)

Re: [I] Support more types when pruning Parquet data [datafusion]

2025-04-24 Thread via GitHub
etseidl commented on issue #15742: URL: https://github.com/apache/datafusion/issues/15742#issuecomment-2828518601 Fixed by #15764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] replace reassign_predicate_columns helper with PhysicalExpr::with_schema [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on code in PR #15779: URL: https://github.com/apache/datafusion/pull/15779#discussion_r2058837968 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -333,6 +333,15 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash {

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-04-24 Thread via GitHub
berkaysynnada commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2058172660 ## datafusion/core/src/datasource/listing/table.rs: ## @@ -982,18 +980,6 @@ impl TableProvider for ListingTable { return Ok(TableProvide

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058901724 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "jem

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058898316 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "jem

Re: [PR] feat: add jemalloc as optional custom allocator [datafusion-comet]

2025-04-24 Thread via GitHub
mbutrovich commented on code in PR #1679: URL: https://github.com/apache/datafusion-comet/pull/1679#discussion_r2058898316 ## native/core/src/lib.rs: ## @@ -50,6 +54,10 @@ pub mod execution; mod jvm_bridge; pub mod parquet; +#[cfg(all(not(target_env = "msvc"), feature = "jem

Re: [PR] Added SQL Example for `Aggregate Functions` [datafusion]

2025-04-24 Thread via GitHub
Adez017 closed pull request #15778: Added SQL Example for `Aggregate Functions` URL: https://github.com/apache/datafusion/pull/15778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-04-24 Thread via GitHub
adriangb commented on PR #15769: URL: https://github.com/apache/datafusion/pull/15769#issuecomment-2828267340 I do think you make a good point of "can we keep the current thing and add the new one". It's worth a shot, at least to split the PR into two. And if that's too complicated or if we

[PR] WIP: scalar UDFs with metadata [datafusion-python]

2025-04-24 Thread via GitHub
timsaucer opened a new pull request, #1110: URL: https://github.com/apache/datafusion-python/pull/1110 # Which issue does this PR close? Closes #. # Rationale for this change # What changes are included in this PR? # Are there any user-facing chang

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
timsaucer commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059245151 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -71,11 +71,23 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash { /

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
timsaucer commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059245151 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -71,11 +71,23 @@ pub trait PhysicalExpr: Send + Sync + Display + Debug + DynEq + DynHash { /

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
timsaucer commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059247381 ## docs/source/library-user-guide/upgrading.md: ## @@ -19,6 +19,33 @@ # Upgrade Guides +## DataFusion `48.0.0` + +### Processing `Field` instead of `DataTyp

Re: [PR] Fix `CoalescePartitionsExec` proto serialization [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on code in PR #15824: URL: https://github.com/apache/datafusion/pull/15824#discussion_r2059464810 ## datafusion/physical-optimizer/src/enforce_distribution.rs: ## @@ -1036,9 +1036,9 @@ pub fn replace_order_preserving_variants( if is_sort_preserving_merge

Re: [PR] Fix build failure caused by new `CoalescePartitionsExec::with_fetch` method [datafusion]

2025-04-24 Thread via GitHub
lewiszlw merged PR #15849: URL: https://github.com/apache/datafusion/pull/15849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] improve eliminate_outer_join rule [datafusion]

2025-04-24 Thread via GitHub
github-actions[bot] closed pull request #13249: improve eliminate_outer_join rule URL: https://github.com/apache/datafusion/pull/13249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] `physical_planner::tests::test_optimization_invariant_checker` fails on `release-nonlto` profile but not `test` profile [datafusion]

2025-04-24 Thread via GitHub
qazxcdswe123 commented on issue #15848: URL: https://github.com/apache/datafusion/issues/15848#issuecomment-2829251167 I'm currently running it in `release` profile but it may take a while.. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Fix `CoalescePartitionsExec` proto serialization [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on code in PR #15824: URL: https://github.com/apache/datafusion/pull/15824#discussion_r2059510351 ## datafusion/physical-optimizer/src/enforce_distribution.rs: ## @@ -1036,9 +1036,9 @@ pub fn replace_order_preserving_variants( if is_sort_preserving_merge

Re: [PR] Fix: fetch is missing in `plan_with_order_breaking_variants` method [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 merged PR #15842: URL: https://github.com/apache/datafusion/pull/15842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] Fix: fetch is missing in `plan_with_order_breaking_variants` method [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 commented on PR #15842: URL: https://github.com/apache/datafusion/pull/15842#issuecomment-2829300863 Thanks @xudong963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[PR] Fix build failure caused by new `CoalescePartitionsExec::with_fetch` [datafusion]

2025-04-24 Thread via GitHub
lewiszlw opened a new pull request, #15849: URL: https://github.com/apache/datafusion/pull/15849 ## Which issue does this PR close? - Closes #. ## Rationale for this change There was a new pr (https://github.com/apache/datafusion/pull/15842) merged. ##

[I] `select count(distinct col)` query doesn't go the specialized distinct accumulator [datafusion]

2025-04-24 Thread via GitHub
jayzhan211 opened a new issue, #15850: URL: https://github.com/apache/datafusion/issues/15850 ### Is your feature request related to a problem or challenge? ``` statement count 0 create table t(a int) as values (1), (2); query I select count(distinct a) from t;

Re: [PR] fix: Fall back to Spark when reading Parquet files with structs inside arrays [datafusion-comet]

2025-04-24 Thread via GitHub
andygrove commented on PR #1682: URL: https://github.com/apache/datafusion-comet/pull/1682#issuecomment-2828901727 Moving this to draft because many tests are failing now that were previously passing. -- This is an automated message from the Apache Git Service. To respond to the message,

[I] How to install ballista python package? [datafusion-ballista]

2025-04-24 Thread via GitHub
Wuerike opened a new issue, #1257: URL: https://github.com/apache/datafusion-ballista/issues/1257 Sorry for this (probably) dumb question, but I've couldn't figure out how to install de python package -- This is an automated message from the Apache Git Service. To respond to the message,

[I] `physical_planner::tests::test_optimization_invariant_checker` fails on `release-nonlto` profile but not `dev` profile [datafusion]

2025-04-24 Thread via GitHub
qazxcdswe123 opened a new issue, #15848: URL: https://github.com/apache/datafusion/issues/15848 ### Describe the bug With `RUST_BACKTRACE=1 cargo test physical_planner::tests::test_optimization_invariant_checker --profile release-nonlto` ``` running 1 test test physical_

Re: [PR] Update extending-operators.md [datafusion]

2025-04-24 Thread via GitHub
xudong963 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2829380416 You can refer to the doc: https://datafusion.apache.org/library-user-guide/custom-table-providers.html. It should contain the real code https://github.com/apache/datafusion/

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-24 Thread via GitHub
crystalxyz commented on code in PR #15646: URL: https://github.com/apache/datafusion/pull/15646#discussion_r2059575045 ## docs/source/library-user-guide/upgrading.md: ## @@ -19,6 +19,33 @@ # Upgrade Guides +## DataFusion `48.0.0` + +### Processing `Field` instead of `DataTy

Re: [PR] Fix `CoalescePartitionsExec` proto serialization [datafusion]

2025-04-24 Thread via GitHub
lewiszlw merged PR #15824: URL: https://github.com/apache/datafusion/pull/15824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Update extending-operators.md [datafusion]

2025-04-24 Thread via GitHub
Adez017 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2829373011 Hi @xudong963 , i think it is ready , give it a check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Optimized spill file format [datafusion]

2025-04-24 Thread via GitHub
getChan commented on issue #14078: URL: https://github.com/apache/datafusion/issues/14078#issuecomment-2829344168 update : skip validation is applied when reading spill files. by #15454 -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] fix: fetch is missed during EnforceDistribution [datafusion]

2025-04-24 Thread via GitHub
xudong963 closed pull request #14207: fix: fetch is missed during EnforceDistribution URL: https://github.com/apache/datafusion/pull/14207 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[PR] Prepare for 0.56.0 release: Version and CHANGELOG [datafusion-sqlparser-rs]

2025-04-24 Thread via GitHub
iffyio opened a new pull request, #1822: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1822 Part of - #1756 1. Updates the version 2. Adds a changelog. See [rendered version here](https://github.com/validio-io/sqlparser-rs/blob/prepare_0.56.0/changelog/0.56.0.

[PR] Handle missing login in changelog generate script [datafusion-sqlparser-rs]

2025-04-24 Thread via GitHub
iffyio opened a new pull request, #1823: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1823 Noticed during #1822 that the script initially failed due to [this commit](https://github.com/apache/datafusion-sqlparser-rs/commit/10cf7c164ee0bae8a71e1d8f0af5851b96465692) no longe

Re: [PR] Add `DECLARE ... CURSOR FOR` support for SQL Server [datafusion-sqlparser-rs]

2025-04-24 Thread via GitHub
iffyio merged PR #1821: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1821 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

[PR] Feat/auto bench [datafusion]

2025-04-24 Thread via GitHub
saraghds opened a new pull request, #15845: URL: https://github.com/apache/datafusion/pull/15845 ## Which issue does this PR close? - Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested

Re: [PR] Feat/auto bench [datafusion]

2025-04-24 Thread via GitHub
saraghds closed pull request #15845: Feat/auto bench URL: https://github.com/apache/datafusion/pull/15845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

[PR] feat: make execution_graph.stages() public [datafusion-ballista]

2025-04-24 Thread via GitHub
milenkovicm opened a new pull request, #1256: URL: https://github.com/apache/datafusion-ballista/pull/1256 # Which issue does this PR close? Closes #. # Rationale for this change this is follow up on ##1243 where I missed to name `stages()` public. Making stages public

  1   2   >