Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996516558 ## datafusion/functions-nested/src/dimension.rs: ## @@ -204,59 +194,57 @@ pub fn array_dims_inner(args: &[ArrayRef]) -> Result { let [array] = take_functi

Re: [PR] fix: nested window function [datafusion]

2025-03-14 Thread via GitHub
2010YOUY01 commented on code in PR #15033: URL: https://github.com/apache/datafusion/pull/15033#discussion_r1996542785 ## datafusion/sql/src/select.rs: ## @@ -887,29 +888,42 @@ fn match_window_definitions( named_windows: &[NamedWindowDefinition], ) -> Result<()> { for

[PR] Minor: exclude datafusion-cli testing for mac [datafusion]

2025-03-14 Thread via GitHub
zhuqi-lucas opened a new pull request, #15240: URL: https://github.com/apache/datafusion/pull/15240 ## Which issue does this PR close? - Closes [#15226](https://github.com/apache/datafusion/issues/15226) ## Rationale for this change exclude datafusion-cli testing for mac

Re: [I] Consider only runnning datafusion-cli tests for linux (not mac) [datafusion]

2025-03-14 Thread via GitHub
zhuqi-lucas commented on issue #15226: URL: https://github.com/apache/datafusion/issues/15226#issuecomment-2726189353 One more question, do we want to exclude more testing cases for mac, i see the linux case exclude the following: ```rust --exclude datafusion-exa

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996526390 ## datafusion/sqllogictest/test_files/array.slt: ## @@ -6232,12 +6244,12 @@ select array_intersect(arrow_cast([1, 1, 2, 2, 3, 3], 'LargeList(Int64)'), null) q

Re: [I] Consider only runnning datafusion-cli tests for linux (not mac) [datafusion]

2025-03-14 Thread via GitHub
zhuqi-lucas commented on issue #15226: URL: https://github.com/apache/datafusion/issues/15226#issuecomment-2726187464 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[PR] Re-enable github discussion [datafusion]

2025-03-14 Thread via GitHub
2010YOUY01 opened a new pull request, #15241: URL: https://github.com/apache/datafusion/pull/15241 ## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/15235 ## Rationale for this change This PR follows the fix in `arrow-rs`

Re: [I] [DISCUSSION] physical-plan-common crate ~and Revert the datasource - physical-plan Dependency~ [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on issue #15111: URL: https://github.com/apache/datafusion/issues/15111#issuecomment-2726071858 The dependency is like this if we have `physical-plan-common` ```mermaid graph TD; datasource-->physical-plan-common; physical-plan-->physical-pla

Re: [PR] Snowflake: Support dollar quoted comments [datafusion-sqlparser-rs]

2025-03-14 Thread via GitHub
iffyio merged PR #1755: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [I] Add support for PostgreSQL LISTEN/NOTIFY [datafusion-sqlparser-rs]

2025-03-14 Thread via GitHub
iffyio closed issue #1481: Add support for PostgreSQL LISTEN/NOTIFY URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1481 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Add upgrade notes for array signatures [datafusion]

2025-03-14 Thread via GitHub
jkosh44 commented on code in PR #15237: URL: https://github.com/apache/datafusion/pull/15237#discussion_r1996445583 ## docs/source/library-user-guide/upgrading.md: ## @@ -212,4 +212,84 @@ To include special characters (such as newlines via `\n`) you can use an `E` lit Elapsed

Re: [PR] Fix wildcard dataframe case [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15230: URL: https://github.com/apache/datafusion/pull/15230#discussion_r1995600861 ## datafusion/core/src/dataframe/mod.rs: ## @@ -342,13 +343,28 @@ impl DataFrame { /// # Ok(()) /// # } /// ``` -pub fn select(self, expr_lis

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996513936 ## datafusion/expr/src/type_coercion/functions.rs: ## @@ -364,98 +366,73 @@ fn get_valid_types( return Ok(vec![vec![]]); } -let

Re: [I] Emit warning with attached `Diagnostic` when doing `= NULL` [datafusion]

2025-03-14 Thread via GitHub
changsun20 commented on issue #14434: URL: https://github.com/apache/datafusion/issues/14434#issuecomment-2726143005 Hi @eliaperantoni, Thank you raising this issue. As I explore potential implementations, I’d appreciate your insights on a few key points: **1. Warning Scope fo

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996516558 ## datafusion/functions-nested/src/dimension.rs: ## @@ -204,59 +194,57 @@ pub fn array_dims_inner(args: &[ArrayRef]) -> Result { let [array] = take_functi

Re: [PR] Simpler to see expressions in explain `tree` mode [datafusion]

2025-03-14 Thread via GitHub
irenjj commented on code in PR #15163: URL: https://github.com/apache/datafusion/pull/15163#discussion_r1996471175 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -739,43 +736,42 @@ physical_plan 01)┌───┐ 02)│ ProjectionExec │

[I] Simplify the printing of all plans containing `expr` in `tree` mode. [datafusion]

2025-03-14 Thread via GitHub
irenjj opened a new issue, #15238: URL: https://github.com/apache/datafusion/issues/15238 ### Is your feature request related to a problem or challenge? part of #14914 #15163 add a new function `fmt_sql` for all expressions to simplify the printing of all expression(`binary`, `cast

Re: [I] Simplify the printing of all plans containing `expr` in `tree` mode. [datafusion]

2025-03-14 Thread via GitHub
irenjj commented on issue #15238: URL: https://github.com/apache/datafusion/issues/15238#issuecomment-2726138696 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Implement `tree` explain for `ArrowFileSink`, fix original URL [datafusion]

2025-03-14 Thread via GitHub
alamb commented on PR #15206: URL: https://github.com/apache/datafusion/pull/15206#issuecomment-2725662551 Love it -- thanks again @irenjj -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996526106 ## datafusion/sqllogictest/test_files/array.slt: ## @@ -4408,12 +4422,10 @@ select array_union(arrow_cast([], 'LargeList(Int64)'), arrow_cast([], 'LargeList q

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996526816 ## datafusion/sqllogictest/test_files/array.slt: ## @@ -3075,22 +3086,26 @@ select array_concat( [1, 2, 3] -# Concatenating Mixed types (doesn't work)

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996528078 ## datafusion/functions-nested/src/extract.rs: ## @@ -200,6 +199,7 @@ fn array_element_inner(args: &[ArrayRef]) -> Result { let [array, indexes] = take_fu

[PR] Add upgrade notes for array signatures [datafusion]

2025-03-14 Thread via GitHub
jkosh44 opened a new pull request, #15237: URL: https://github.com/apache/datafusion/pull/15237 This commit adds upgrade notes for 469f18be1c594b07e4b235f3404419792ed3c24f. Resolves #15105 ## Which issue does this PR close? - Closes #15105. ## Rationale for this

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 commented on code in PR #15149: URL: https://github.com/apache/datafusion/pull/15149#discussion_r1996511821 ## datafusion/expr/src/type_coercion/functions.rs: ## @@ -364,98 +366,73 @@ fn get_valid_types( return Ok(vec![vec![]]); } -let

Re: [PR] Fix broken `serde` feature [datafusion]

2025-03-14 Thread via GitHub
alamb commented on PR #15124: URL: https://github.com/apache/datafusion/pull/15124#issuecomment-2724651837 Backport PR: - https://github.com/apache/datafusion/pull/15227 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] [DISCUSS] Release DataFusion `46.0.1` Patch or `46.1.0` minor release (March 2025) [datafusion]

2025-03-14 Thread via GitHub
alamb commented on issue #15151: URL: https://github.com/apache/datafusion/issues/15151#issuecomment-2724699117 I have made the backport PRs: - https://github.com/apache/datafusion/pull/15227 - https://github.com/apache/datafusion/pull/15228 - https://github.com/apache/datafusion/pul

Re: [PR] datafusion-cli: add streaming state for printing logic [datafusion]

2025-03-14 Thread via GitHub
shruti2522 closed pull request #14961: datafusion-cli: add streaming state for printing logic URL: https://github.com/apache/datafusion/pull/14961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Allow providing Arrow schema when scanning Parquet files [datafusion]

2025-03-14 Thread via GitHub
HawaiianSpork commented on issue #5950: URL: https://github.com/apache/datafusion/issues/5950#issuecomment-2724596144 > > This should be fixed now by https://github.com/apache/datafusion/pull/10515. You can now override the schema used in the file scanner using the SchemaAdapter. > >

Re: [PR] [branch-46] Fix wasm32 build on version 46 [datafusion]

2025-03-14 Thread via GitHub
alamb commented on PR #15229: URL: https://github.com/apache/datafusion/pull/15229#issuecomment-2724704204 Security Audit CI should be fixed by - https://github.com/apache/datafusion/pull/15228 -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] [branch-46] Fix wasm32 build on version 46 [datafusion]

2025-03-14 Thread via GitHub
alamb opened a new pull request, #15229: URL: https://github.com/apache/datafusion/pull/15229 ## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/15151 - Related to https://github.com/apache/datafusion/issues/15150 ## Rationale for this ch

[PR] [branch-46] Fix broken `serde` feature (#15124) [datafusion]

2025-03-14 Thread via GitHub
alamb opened a new pull request, #15227: URL: https://github.com/apache/datafusion/pull/15227 - Part of https://github.com/apache/datafusion/issues/15151 - Backport https://github.com/apache/datafusion/issues/15122 / https://github.com/apache/datafusion/pull/15124 to `branch-46` --

Re: [PR] Renaming Internal Structs [datafusion-python]

2025-03-14 Thread via GitHub
Spaarsh commented on PR #1059: URL: https://github.com/apache/datafusion-python/pull/1059#issuecomment-2724612608 > Thank you for all the work on this. I wasn't of much help in the tests part. Thanks for your patience!! -- This is an automated message from the Apache Git Service. T

[PR] Fix wildcard dataframe case [datafusion]

2025-03-14 Thread via GitHub
jayzhan211 opened a new pull request, #15230: URL: https://github.com/apache/datafusion/pull/15230 ## Which issue does this PR close? - Closes #15218 . ## Rationale for this change ## What changes are included in this PR? ## Are these change

Re: [I] Upgrade Guide for DataFusion 46 does not include the array signatures change [datafusion]

2025-03-14 Thread via GitHub
Omega359 commented on issue #15105: URL: https://github.com/apache/datafusion/issues/15105#issuecomment-2724800170 @jkosh44 Would you be able to add a note about array signature changes to https://github.com/apache/datafusion/blob/main/docs/source/library-user-guide/upgrading.md ? -- Thi

Re: [PR] docs: various improvements to tuning guide [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove commented on code in PR #1525: URL: https://github.com/apache/datafusion-comet/pull/1525#discussion_r1995643547 ## docs/source/user-guide/tuning.md: ## @@ -17,18 +17,94 @@ specific language governing permissions and limitations under the License. --> -# Tuning Guid

Re: [PR] docs: various improvements to tuning guide [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove commented on code in PR #1525: URL: https://github.com/apache/datafusion-comet/pull/1525#discussion_r1995645694 ## docs/source/user-guide/tuning.md: ## @@ -143,28 +182,20 @@ Once it is disabled, Comet will fall back to the default Spark shuffle manager. ### Shuffle

Re: [I] [DISCUSS] Release DataFusion `46.0.1` Patch or `46.1.0` minor release (March 2025) [datafusion]

2025-03-14 Thread via GitHub
xudong963 commented on issue #15151: URL: https://github.com/apache/datafusion/issues/15151#issuecomment-2725044156 > I'll try and get them merged today and then I think we can make release notes and prep an RC Hi @alamb , I'm going to bed, I'll start the release process tomorrow --

Re: [PR] Implement tree explain for `LocalLimitExec` [datafusion]

2025-03-14 Thread via GitHub
comphead commented on code in PR #15232: URL: https://github.com/apache/datafusion/pull/15232#discussion_r1995801147 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -246,6 +252,72 @@ physical_plan 11)│format: csv│ 12)└───┘

Re: [PR] chore(deps): bump tokio-util from 0.7.13 to 0.7.14 [datafusion]

2025-03-14 Thread via GitHub
comphead merged PR #15223: URL: https://github.com/apache/datafusion/pull/15223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] chore(deps): bump bzip2 from 0.5.1 to 0.5.2 [datafusion]

2025-03-14 Thread via GitHub
comphead merged PR #15221: URL: https://github.com/apache/datafusion/pull/15221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] datafusion-cli: add streaming state struct [datafusion]

2025-03-14 Thread via GitHub
matthewmturner commented on PR #15234: URL: https://github.com/apache/datafusion/pull/15234#issuecomment-2725076546 I only had time to take a quick glance - but could this functionality be added to datafusion so it could be used by other apps that have CLIs built on datafusion? -- This i

[I] Write blog post for Comet 0.7.0 release [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove opened a new issue, #1531: URL: https://github.com/apache/datafusion-comet/issues/1531 ### What is the problem the feature request solves? I created a Google doc where we can collaborate on this: I started a Google doc where we can collaborate on the Comet 0.7.0 blog p

[I] Consider only runnning datafusion-cli tests for linux (not mac) [datafusion]

2025-03-14 Thread via GitHub
alamb opened a new issue, #15226: URL: https://github.com/apache/datafusion/issues/15226 lgtm thanks @alamb its about time, for each platform we recompile and run tests for datafusion cli, although just having a linux is enough _Originally posted by @comphead in https:

Re: [PR] Renaming Internal Structs [datafusion-python]

2025-03-14 Thread via GitHub
timsaucer merged PR #1059: URL: https://github.com/apache/datafusion-python/pull/1059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [PR] _repr_ and _html_repr_ show '... and additional rows' message [datafusion-python]

2025-03-14 Thread via GitHub
Spaarsh closed pull request #1041: _repr_ and _html_repr_ show '... and additional rows' message URL: https://github.com/apache/datafusion-python/pull/1041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] Add additional ruff suggestions [datafusion-python]

2025-03-14 Thread via GitHub
Spaarsh opened a new pull request, #1062: URL: https://github.com/apache/datafusion-python/pull/1062 # Which issue does this PR close? Closes #1056 # Rationale for this change Need for enabling ruff rules. # What changes are included in this PR? En

Re: [I] `core_expressions` feature is broken in the `datafusion-functions` [datafusion]

2025-03-14 Thread via GitHub
Omega359 commented on issue #15207: URL: https://github.com/apache/datafusion/issues/15207#issuecomment-2724786354 I had no clue that feature even existed :/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] Add pytest-asyncio unit tests [datafusion-python]

2025-03-14 Thread via GitHub
jsai28 commented on issue #991: URL: https://github.com/apache/datafusion-python/issues/991#issuecomment-2724790361 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Expose additional regexp functions [datafusion-python]

2025-03-14 Thread via GitHub
nirnayroy commented on issue #803: URL: https://github.com/apache/datafusion-python/issues/803#issuecomment-2724800423 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [branch-46] Update ring to v0.17.13 (#15063) [datafusion]

2025-03-14 Thread via GitHub
xudong963 merged PR #15228: URL: https://github.com/apache/datafusion/pull/15228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] fix: unparsing left/ right semi/mark join [datafusion]

2025-03-14 Thread via GitHub
chenkovsky commented on PR #15212: URL: https://github.com/apache/datafusion/pull/15212#issuecomment-2724945557 > Thanks @chenkovsky. I think the results of `left mark join`, `right semi join` and `right anit join` aren't correct (they aren't executable). I left some comments for them. Coul

[I] Generate the common SQL for the unparsing result of the unnest [datafusion]

2025-03-14 Thread via GitHub
goldmedal opened a new issue, #15233: URL: https://github.com/apache/datafusion/issues/15233 ### Is your feature request related to a problem or challenge? After https://github.com/apache/datafusion/pull/15090, we remove the wildacrd expression from the logical plan layer. The unparse

[PR] test: add pytest asyncio tests [datafusion-python]

2025-03-14 Thread via GitHub
jsai28 opened a new pull request, #1063: URL: https://github.com/apache/datafusion-python/pull/1063 # Which issue does this PR close? Closes #991. # Rationale for this change Adding tests for Async Iteration of `RecordBatchStream`. # What changes are included in this PR?

Re: [PR] Reuse alias if possible [datafusion]

2025-03-14 Thread via GitHub
goldmedal commented on code in PR #14781: URL: https://github.com/apache/datafusion/pull/14781#discussion_r1995765092 ## datafusion/sql/src/unparser/plan.rs: ## @@ -860,8 +877,12 @@ impl Unparser<'_> { query: &mut Option, select: &mut SelectBuilder, re

Re: [PR] fix: unparsing left/ right semi/mark join [datafusion]

2025-03-14 Thread via GitHub
goldmedal commented on PR #15212: URL: https://github.com/apache/datafusion/pull/15212#issuecomment-2725013843 > for stackoverflow problem. do you have any idea? No, I don't have any idea currently 🤔. I'm not sure, but I guess `recursive_protection` is the right direction. -- This

[PR] chore: Prepare for 0.8.0 development [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove opened a new pull request, #1530: URL: https://github.com/apache/datafusion-comet/pull/1530 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] Implement tree explain for `LocalLimitExec` [datafusion]

2025-03-14 Thread via GitHub
shruti2522 commented on code in PR #15232: URL: https://github.com/apache/datafusion/pull/15232#discussion_r1995913794 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -246,6 +252,72 @@ physical_plan 11)│format: csv│ 12)└───

[I] DataFusion discussions are missing [datafusion]

2025-03-14 Thread via GitHub
alamb opened a new issue, #15235: URL: https://github.com/apache/datafusion/issues/15235 ### Describe the bug DataFusion used to have gituhub disucssions However now they are gone ![Image](https://github.com/user-attachments/assets/a956471f-e3f9-4a98-84b4-24bce52c0c9f)

Re: [PR] [branch-46] Fix broken `serde` feature (#15124) [datafusion]

2025-03-14 Thread via GitHub
alamb commented on PR #15227: URL: https://github.com/apache/datafusion/pull/15227#issuecomment-2725233595 > Strange, can't rebase the branch-46 to the PR even if the branch-46 has been updated Yeah it is weird -- I merged it locally and pushed up the changes -- This is an automate

Re: [I] Browser-accessible official DataFusion playground / DataFusion fiddle [datafusion]

2025-03-14 Thread via GitHub
pranavJibhakate commented on issue #13818: URL: https://github.com/apache/datafusion/issues/13818#issuecomment-2725258319 Hi @XiangpengHao and @waynexia is there a way to make the a website like in [parquet-viewer](https://parquet-viewer.xiangpeng.systems/) without reading the whole file f

Re: [I] DataFusion discussions are missing [datafusion]

2025-03-14 Thread via GitHub
alamb commented on issue #15235: URL: https://github.com/apache/datafusion/issues/15235#issuecomment-2725259670 See a fix for arrow-rs that is likely related from @assignUser - https://github.com/apache/arrow-rs/pull/7288 -- This is an automated message from the Apache Git Service. To

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-14 Thread via GitHub
alamb commented on PR #15110: URL: https://github.com/apache/datafusion/pull/15110#issuecomment-2725533031 > Actually I'm quite curious is string literal really an issue? If we want string, we can have query with quote `select * from t1 where column1 < '10';`, while if we want numeric, we c

Re: [PR] Use insta for `DataFrame` tests [datafusion]

2025-03-14 Thread via GitHub
alamb commented on PR #15165: URL: https://github.com/apache/datafusion/pull/15165#issuecomment-2725523499 I merged this branch up from main to rerun the tests. Once CI passes I'll plan to merge this in -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] chore: Drop support for Spark 3.3 (EOL) [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove merged PR #1529: URL: https://github.com/apache/datafusion-comet/pull/1529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Implement `tree` explain for `LocalLimitExec` [datafusion]

2025-03-14 Thread via GitHub
alamb closed issue #15025: Implement `tree` explain for `LocalLimitExec` URL: https://github.com/apache/datafusion/issues/15025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Implement tree explain for `LocalLimitExec` [datafusion]

2025-03-14 Thread via GitHub
alamb merged PR #15232: URL: https://github.com/apache/datafusion/pull/15232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Snowflake: Support dollar quoted comment when creating tables, views, and their fields [datafusion-sqlparser-rs]

2025-03-14 Thread via GitHub
7phs commented on code in PR #1755: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1755#discussion_r1995866489 ## src/parser/mod.rs: ## @@ -6926,14 +6927,16 @@ impl<'a> Parser<'a> { let comment = if self.parse_keyword(Keyword::COMMENT) { let h

Re: [PR] fix: `core_expressions` feature flag broken,, move `coverlay` into `core` functions [datafusion]

2025-03-14 Thread via GitHub
shruti2522 commented on PR #15217: URL: https://github.com/apache/datafusion/pull/15217#issuecomment-2725633230 > Thank you @shruti2522 🙏 > > The only concern I have about this PR is that now overlay() is not listed as a string function, which is kind of strange. > > It also me

Re: [PR] fix: remove code duplication in native_datafusion and native_iceberg_compat implementations [datafusion-comet]

2025-03-14 Thread via GitHub
comphead commented on code in PR #1443: URL: https://github.com/apache/datafusion-comet/pull/1443#discussion_r1996144152 ## native/core/src/parquet/mod.rs: ## @@ -620,12 +619,21 @@ fn get_batch_context<'a>(handle: jlong) -> Result<&'a mut BatchContext, CometErr } } -/*

Re: [PR] docs: various improvements to tuning guide [datafusion-comet]

2025-03-14 Thread via GitHub
kazuyukitanimura commented on code in PR #1525: URL: https://github.com/apache/datafusion-comet/pull/1525#discussion_r1996191336 ## docs/source/user-guide/tuning.md: ## @@ -17,18 +17,96 @@ specific language governing permissions and limitations under the License. --> -# Tuni

[PR] Added wrapper for regex_count function [datafusion-python]

2025-03-14 Thread via GitHub
nirnayroy opened a new pull request, #1066: URL: https://github.com/apache/datafusion-python/pull/1066 # Which issue does this PR close? Closes (https://github.com/apache/datafusion-python/issues/803) partially # Rationale for this change Expose additional regex function

Re: [PR] Remove inline table scan analyzer rule [datafusion]

2025-03-14 Thread via GitHub
alamb commented on PR #15201: URL: https://github.com/apache/datafusion/pull/15201#issuecomment-2725551994 I ran planning performance benchmarks and I would say they showed no discernible difference with this branch Details ``` group

Re: [PR] chore: re-enable GitHub discussions [datafusion-comet]

2025-03-14 Thread via GitHub
codecov-commenter commented on PR #1532: URL: https://github.com/apache/datafusion-comet/pull/1532#issuecomment-2725591331 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1532?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] chore: Drop support for Spark 3.3 (EOL) [datafusion-comet]

2025-03-14 Thread via GitHub
kazuyukitanimura commented on PR #1529: URL: https://github.com/apache/datafusion-comet/pull/1529#issuecomment-2725591671 cc @huaxingao you may be able to remove the some of the duplicated code for Iceberg after this PR -- This is an automated message from the Apache Git Service. To resp

Re: [PR] chore: Drop support for Spark 3.3 (EOL) [datafusion-comet]

2025-03-14 Thread via GitHub
kazuyukitanimura commented on code in PR #1529: URL: https://github.com/apache/datafusion-comet/pull/1529#discussion_r1995947311 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -2716,17 +2705,15 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] Implement tree explain for `LocalLimitExec` [datafusion]

2025-03-14 Thread via GitHub
alamb commented on code in PR #15232: URL: https://github.com/apache/datafusion/pull/15232#discussion_r1996145712 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -246,6 +251,69 @@ physical_plan 11)│format: csv│ 12)└───┘ +

Re: [I] Auto-update mechanism for dataframe test [datafusion]

2025-03-14 Thread via GitHub
alamb closed issue #10373: Auto-update mechanism for dataframe test URL: https://github.com/apache/datafusion/issues/10373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Fix invalid schema for unions in ViewTables [datafusion]

2025-03-14 Thread via GitHub
Friede80 commented on code in PR #15135: URL: https://github.com/apache/datafusion/pull/15135#discussion_r1996023999 ## datafusion/expr/src/logical_plan/builder.rs: ## @@ -776,8 +777,32 @@ impl LogicalPlanBuilder { &missing_cols, is_distinct, )

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-03-14 Thread via GitHub
shehabgamin commented on code in PR #15168: URL: https://github.com/apache/datafusion/pull/15168#discussion_r1994917169 ## datafusion/sqllogictest/test_files/spark/math/expm1.slt: ## @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contri

Re: [PR] chore: Attach Diagnostic to "incompatible type in unary expression" error [datafusion]

2025-03-14 Thread via GitHub
eliaperantoni commented on code in PR #15209: URL: https://github.com/apache/datafusion/pull/15209#discussion_r1995063039 ## datafusion/sql/src/expr/unary_op.rs: ## @@ -45,7 +45,13 @@ impl SqlToRel<'_, S> { { Ok(operand) } e

Re: [PR] chore: Attach Diagnostic to "incompatible type in unary expression" error [datafusion]

2025-03-14 Thread via GitHub
onlyjackfrost commented on PR #15209: URL: https://github.com/apache/datafusion/pull/15209#issuecomment-2724260125 @eliaperantoni, > Is there any way you can extend it to all unary expressions Sure. I was thinking about doing the same data type check in other unary operator.

Re: [PR] fix: handle duplicate WindowFunction expressions in Substrait consumer [datafusion]

2025-03-14 Thread via GitHub
Blizzara commented on PR #15211: URL: https://github.com/apache/datafusion/pull/15211#issuecomment-2725937469 > Does this error come from Datafusion or from Substrait? I wouldn't think Substrait would care about duplicate names (since it doesn't use names). DataFusion. Substrait indee

Re: [PR] chore: Upgrade `rand` crate and some other minor crates [datafusion]

2025-03-14 Thread via GitHub
comphead commented on PR #14967: URL: https://github.com/apache/datafusion/pull/14967#issuecomment-2725922642 Depends on https://github.com/apache/arrow-rs/pull/7293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] chore: Prepare for 0.8.0 development [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove merged PR #1530: URL: https://github.com/apache/datafusion-comet/pull/1530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] chore: Upgrade `rand` crate and some other minor crates [datafusion]

2025-03-14 Thread via GitHub
comphead commented on PR #14967: URL: https://github.com/apache/datafusion/pull/14967#issuecomment-2725969346 wasm pack fails because `getrandom 0.2.x` attached to `rand 0.8.x`. wasm requires `js` feature to be enabled, but the feature renamed from `js` to `wasm-js` in `getrandom 0.3` and

Re: [PR] chore: Upgrade `rand` crate and some other minor crates [datafusion]

2025-03-14 Thread via GitHub
comphead commented on PR #14967: URL: https://github.com/apache/datafusion/pull/14967#issuecomment-2725970120 @mbrobbel cc as you working on the migration of arrow-rs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Support default values for columns in SchemaAdapter [datafusion]

2025-03-14 Thread via GitHub
adriangb commented on issue #15220: URL: https://github.com/apache/datafusion/issues/15220#issuecomment-2726006376 > The other thing to note is that adjustments are needed in filter pushdown I’ll note that this is currently broken without any new features. I can give an example later

Re: [PR] _repr_ and _html_repr_ show '... and additional rows' message [datafusion-python]

2025-03-14 Thread via GitHub
Spaarsh commented on PR #1041: URL: https://github.com/apache/datafusion-python/pull/1041#issuecomment-2724534357 @timsaucer Closing this since the changes were incorporated into #1036. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Simpler to see expressions in tree explain mode [datafusion]

2025-03-14 Thread via GitHub
alamb commented on code in PR #15163: URL: https://github.com/apache/datafusion/pull/15163#discussion_r1996216639 ## datafusion/physical-expr-common/src/physical_expr.rs: ## @@ -379,3 +396,53 @@ where DisplayWrapper(exprs.into_iter()) } + +/// Prints a [`PhysicalExpr`] i

Re: [PR] Add upgrade notes for array signatures [datafusion]

2025-03-14 Thread via GitHub
jkosh44 commented on code in PR #15237: URL: https://github.com/apache/datafusion/pull/15237#discussion_r1996444071 ## docs/source/library-user-guide/upgrading.md: ## @@ -212,4 +212,79 @@ To include special characters (such as newlines via `\n`) you can use an `E` lit Elapsed

Re: [PR] Implement `tree` explain for `ArrowFileSink`, fix original URL [datafusion]

2025-03-14 Thread via GitHub
alamb merged PR #15206: URL: https://github.com/apache/datafusion/pull/15206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] chore: Drop support for Spark 3.3 (EOL) [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove commented on PR #1529: URL: https://github.com/apache/datafusion-comet/pull/1529#issuecomment-2725598231 Thanks for the review @kazuyukitanimura -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Publish official Docker images to Docker Hub under Apache account [datafusion-comet]

2025-03-14 Thread via GitHub
andygrove commented on issue #1510: URL: https://github.com/apache/datafusion-comet/issues/1510#issuecomment-2726035533 The repo has been created: https://hub.docker.com/r/apache/datafusion-comet Next, I would like to take our already published image `ghcr.io/apache/datafusion-comet:

Re: [PR] Add upgrade notes for array signatures [datafusion]

2025-03-14 Thread via GitHub
jkosh44 commented on code in PR #15237: URL: https://github.com/apache/datafusion/pull/15237#discussion_r1996457821 ## docs/source/library-user-guide/upgrading.md: ## @@ -212,4 +212,99 @@ To include special characters (such as newlines via `\n`) you can use an `E` lit Elapsed

[I] union by name doesn't seem to be working correctly [datafusion]

2025-03-14 Thread via GitHub
Omega359 opened a new issue, #15236: URL: https://github.com/apache/datafusion/issues/15236 ### Describe the bug union by name seems to not be working as expected: ```sql > create table t1 (x varchar(255), y varchar(255), z varchar(255)); 0 row(s) fetched. Elapsed 0.005

Re: [I] union by name doesn't seem to be working correctly [datafusion]

2025-03-14 Thread via GitHub
Omega359 commented on issue #15236: URL: https://github.com/apache/datafusion/issues/15236#issuecomment-2725289081 fyi @rkrishn7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Add regexp function - regexp_instr() [datafusion]

2025-03-14 Thread via GitHub
nirnayroy commented on issue #13009: URL: https://github.com/apache/datafusion/issues/13009#issuecomment-2725298062 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] `core_expressions` feature is broken in the `datafusion-functions` [datafusion]

2025-03-14 Thread via GitHub
Omega359 commented on issue #15207: URL: https://github.com/apache/datafusion/issues/15207#issuecomment-2725361697 > Me neither -- maybe we just cargo culted it 🤷 Wow, two new terms in one day. First was 'vibe coding' from a coworker, now 'cargo cult'. Still can teach old dogs new tri

Re: [PR] Implement `tree` explain for `CsvSink` [datafusion]

2025-03-14 Thread via GitHub
alamb commented on code in PR #15204: URL: https://github.com/apache/datafusion/pull/15204#discussion_r1993507486 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -1725,6 +1725,22 @@ physical_plan 09)│ rows: 1 │ 10)└───┘

Re: [I] Browser-accessible official DataFusion playground / DataFusion fiddle [datafusion]

2025-03-14 Thread via GitHub
XiangpengHao commented on issue #13818: URL: https://github.com/apache/datafusion/issues/13818#issuecomment-2725268714 > Hi [@XiangpengHao](https://github.com/XiangpengHao) and [@waynexia](https://github.com/waynexia) is there a way to make the a website like in [parquet-viewer](https://par

  1   2   3   >