Re: [I] Create UNION plan node with correct schema [datafusion]

2025-02-07 Thread via GitHub
goldmedal commented on issue #14380: URL: https://github.com/apache/datafusion/issues/14380#issuecomment-2644487765 > Perhaps [@goldmedal](https://github.com/goldmedal) have some thoughts on it. The original discussion about wildcard expansion is https://github.com/apache/datafusion/

[I] Decorrelate scalar subqueries with more complex filter expressions [datafusion]

2025-02-07 Thread via GitHub
duongcongtoai opened a new issue, #14554: URL: https://github.com/apache/datafusion/issues/14554 ### Is your feature request related to a problem or challenge? Datafusion already support decorrelating simple scalar subqueries in this PR: https://github.com/apache/datafusion/pull/6457

Re: [PR] refactor: collect dataframe as stream in `__repr__` [datafusion-python]

2025-02-07 Thread via GitHub
konjac commented on PR #1015: URL: https://github.com/apache/datafusion-python/pull/1015#issuecomment-2644492439 Hi @timsaucer , could you help to trigger CI and also review the proposed change? Thank you! -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [I] Make it easier to create a ScalarValure representing typed `null` [datafusion]

2025-02-07 Thread via GitHub
cj-zhukov commented on issue #14548: URL: https://github.com/apache/datafusion/issues/14548#issuecomment-2644513159 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Update expected error messages after upstream change [datafusion-testing]

2025-02-07 Thread via GitHub
comphead merged PR #6: URL: https://github.com/apache/datafusion-testing/pull/6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Improve documentation about extended tests [datafusion]

2025-02-07 Thread via GitHub
Omega359 commented on PR #14320: URL: https://github.com/apache/datafusion/pull/14320#issuecomment-2644280441 @comphead , @findepi - either of you want to merge this in? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] 14044/enhancement/add xxhash algorithms in expression api [datafusion]

2025-02-07 Thread via GitHub
Omega359 commented on PR #14367: URL: https://github.com/apache/datafusion/pull/14367#issuecomment-2644285338 Thanks for applying the updates! I'll do one final review over the weekend then if it's good (I suspect it is!) then I'll ask a Committer to merge it in. -- This is an automated m

Re: [PR] some dependency removals and setup for refactor of `FileScanConfig` [datafusion]

2025-02-07 Thread via GitHub
logan-keede commented on PR #14543: URL: https://github.com/apache/datafusion/pull/14543#issuecomment-2643963541 cc @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Comet can produce different results to Spark when averaging a decimal [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove closed issue #1354: Comet can produce different results to Spark when averaging a decimal URL: https://github.com/apache/datafusion-comet/issues/1354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] fix: Mark cast from float/double to decimal as incompatible [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove merged PR #1372: URL: https://github.com/apache/datafusion-comet/pull/1372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] Implement FROM-first selects [datafusion-sqlparser-rs]

2025-02-07 Thread via GitHub
iffyio commented on code in PR #1713: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1713#discussion_r1947123514 ## tests/sqlparser_duckdb.rs: ## @@ -246,6 +246,67 @@ fn test_create_table_macro() { assert_eq!(expected, macro_); } +#[test] +fn test_select_fr

Re: [PR] Allow constructing ScalarUDF from shared implementation [datafusion]

2025-02-07 Thread via GitHub
findepi commented on code in PR #14541: URL: https://github.com/apache/datafusion/pull/14541#discussion_r1947197259 ## datafusion/expr/src/udaf.rs: ## @@ -119,9 +119,12 @@ impl AggregateUDF { where F: AggregateUDFImpl + 'static, { -Self { -

Re: [PR] Allow constructing ScalarUDF from shared implementation [datafusion]

2025-02-07 Thread via GitHub
findepi merged PR #14541: URL: https://github.com/apache/datafusion/pull/14541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafu

Re: [I] External sorting not working for (maybe only for string columns??) [datafusion]

2025-02-07 Thread via GitHub
xuchen-plus commented on issue #12136: URL: https://github.com/apache/datafusion/issues/12136#issuecomment-264317 > [@2010YOUY01](https://github.com/2010YOUY01) Your solution does not work for me. I did play around with `sort_spill_reservation_bytes` when I originally encountered this b

[PR] Minor: Add docs and examples for `DataFusionErrorBuilder` [datafusion]

2025-02-07 Thread via GitHub
alamb opened a new pull request, #14551: URL: https://github.com/apache/datafusion/pull/14551 ## Which issue does this PR close? - Follow on https://github.com/apache/datafusion/pull/14439 ## Rationale for this change @eliaperantoni added the nice `DataFusionErrorBuilder`

Re: [PR] Add `DataFusionError::Collection` to return multiple `DataFusionError`s [datafusion]

2025-02-07 Thread via GitHub
alamb commented on code in PR #14439: URL: https://github.com/apache/datafusion/pull/14439#discussion_r1947188396 ## datafusion/common/src/error.rs: ## @@ -569,6 +601,63 @@ impl DataFusionError { DiagnosticsIterator { head: self }.next() } + +/// Sometimes Da

Re: [PR] Validate and unpack function arguments tersely [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14513: URL: https://github.com/apache/datafusion/pull/14513#issuecomment-2644124413 > thank you @alamb @mbrobbel @comphead for your reviews! > > > BTW I bet others would love a chance to clean up the code using this function if we filed a ticket for them to do s

Re: [I] Buildable release builds [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #14479: URL: https://github.com/apache/datafusion/issues/14479#issuecomment-2644127779 > Let's do this. we upgrade compiler version manually anyway, so we can update one more place i believe. It would help us in InfluxDB too as it we have a patched version of

Re: [PR] some dependency removals and setup for refactor of `FileScanConfig` [datafusion]

2025-02-07 Thread via GitHub
logan-keede commented on code in PR #14543: URL: https://github.com/apache/datafusion/pull/14543#discussion_r1947182581 ## datafusion/core/src/datasource/data_source.rs: ## @@ -62,4 +62,8 @@ pub trait FileSource: Send + Sync { fn fmt_extra(&self, _t: DisplayFormatType, _f:

Re: [I] [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #13525: URL: https://github.com/apache/datafusion/issues/13525#issuecomment-2644126863 > DataFusion. Currently this may or may not work - [#14479](https://github.com/apache/datafusion/issues/14479). @mbrobbel has a nice PR to lock dependencies here: - http

Re: [PR] feat: Add fair unified memory pool [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove commented on PR #1369: URL: https://github.com/apache/datafusion-comet/pull/1369#issuecomment-2644248809 > @andygrove I would like to make sure to include this for the 0.6.0 release I added this PR to https://github.com/apache/datafusion-comet/issues/1361 -- This is an au

Re: [PR] Add `DataFusionError::Collection` to return multiple `DataFusionError`s [datafusion]

2025-02-07 Thread via GitHub
eliaperantoni commented on PR #14439: URL: https://github.com/apache/datafusion/pull/14439#issuecomment-2644244588 > Thanks again @eliaperantoni Thank you @alamb! It was very kind of you to fix the test 😊 -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] Add `DataFusionError::Collection` to return multiple `DataFusionError`s [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14439: URL: https://github.com/apache/datafusion/pull/14439#issuecomment-2644253796 > > Thanks again @eliaperantoni > > Thank you @alamb! It was very kind of you to fix the test 😊 I don't really know why it took so much finagling to be honest 🤷 -- This

Re: [PR] fix: disable checking for uint_8 and uint_16 if complex type readers are enabled [datafusion-comet]

2025-02-07 Thread via GitHub
parthchandra commented on code in PR #1376: URL: https://github.com/apache/datafusion-comet/pull/1376#discussion_r1946997661 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -125,6 +125,26 @@ class CometExpressionSuite extends CometTestBase with Adapti

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
timsaucer commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643740088 @kevinjqliu Are you getting an error on this unit test? I'm seeing it pass. Is it possible there's an issue with your submodule that contains the data? It looks like CI unit

Re: [PR] refactor: remove remaining uses of `arrow_array` and use reexport in `arrow` instead [datafusion]

2025-02-07 Thread via GitHub
alamb commented on code in PR #14528: URL: https://github.com/apache/datafusion/pull/14528#discussion_r1947014629 ## datafusion/common/Cargo.toml: ## @@ -51,7 +51,6 @@ apache-avro = { version = "0.17", default-features = false, features = [ "zstandard", ], optional = true

Re: [I] Update ClickBench benchmarks with DataFusion `44.0.0` [datafusion]

2025-02-07 Thread via GitHub
alamb closed issue #13983: Update ClickBench benchmarks with DataFusion `44.0.0` URL: https://github.com/apache/datafusion/issues/13983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Signature::Coercible with user defined implicit casting [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14440: URL: https://github.com/apache/datafusion/pull/14440#issuecomment-2643815648 I will try and review this carefully over the weekend Maybe @shehabgamin has some time to take a look too -- This is an automated message from the Apache Git Service. To respo

Re: [I] Add `try_new` for `LogicalPlan::Join` `Join` and others [datafusion]

2025-02-07 Thread via GitHub
Spaarsh commented on issue #14363: URL: https://github.com/apache/datafusion/issues/14363#issuecomment-2643836272 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] Add `try_new` for `LogicalPlan::Join` `Join` and others [datafusion]

2025-02-07 Thread via GitHub
Spaarsh commented on issue #14363: URL: https://github.com/apache/datafusion/issues/14363#issuecomment-2643836111 I would like to work on this. I will be taking a look at a few ```try_new``` implementations in [benchmarks/src/clickbench.rs](https://github.com/apache/datafusion/blob/fc1835db

Re: [I] Redundancy/Repeated calls in query function [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #14448: URL: https://github.com/apache/datafusion/issues/14448#issuecomment-2643832838 Thanks @TheBitsmith -- it would be great if you could post any information you are able to find out to this ticket (like what the stack traces are) -- This is an automated mes

Re: [PR] fix: Mark cast from float/double to decimal as incompatible [datafusion-comet]

2025-02-07 Thread via GitHub
kazuyukitanimura commented on code in PR #1372: URL: https://github.com/apache/datafusion-comet/pull/1372#discussion_r1946987121 ## spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala: ## @@ -32,13 +32,31 @@ import org.apache.spark.sql.internal.SQLConf import

[I] Make it easier to create a ScalarValure representing typed `null` [datafusion]

2025-02-07 Thread via GitHub
alamb opened a new issue, #14548: URL: https://github.com/apache/datafusion/issues/14548 ### Is your feature request related to a problem or challenge? @findepi notes in https://github.com/apache/datafusion/pull/14541#discussion_r1946891887> Into/From is very important for rea

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
timsaucer commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643862582 Do you think this is ready? If so and you move it out of draft I can merge it. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] feat: Upgrade to DataFusion 45 [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove merged PR #1364: URL: https://github.com/apache/datafusion-comet/pull/1364 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

[I] Construct source plan schema with correct nullability during `INSERT` planning. [datafusion]

2025-02-07 Thread via GitHub
rkrishn7 opened a new issue, #14550: URL: https://github.com/apache/datafusion/issues/14550 ### Describe the bug In, #14394, it was reported that while attempting to implement a `DataSink` different schemas for the record batches were being given than per the `RecordBatchStream`.

Re: [I] DataSink::write_all given invalid RecordBatchStream [datafusion]

2025-02-07 Thread via GitHub
rkrishn7 commented on issue #14394: URL: https://github.com/apache/datafusion/issues/14394#issuecomment-2644048701 @jonahgao Thank you for calling this out. I think you're right! In fact, I think we could say more generally, this issue arises when the schema of the source of an `INSER

Re: [PR] Improve documentation about extended tests [datafusion]

2025-02-07 Thread via GitHub
comphead commented on code in PR #14320: URL: https://github.com/apache/datafusion/pull/14320#discussion_r1947333625 ## docs/source/contributor-guide/testing.md: ## @@ -58,6 +58,19 @@ Like similar systems such as [DuckDB](https://duckdb.org/dev/testing), DataFusio DataFusion

[I] Support accessing a map with non-literal key [datafusion]

2025-02-07 Thread via GitHub
findepi opened a new issue, #14552: URL: https://github.com/apache/datafusion/issues/14552 Accessing a map with a non-string key doesn't work (at least via DataFusion SQL) and is covered by https://github.com/apache/datafusion/issues/11785 Accessing a map with a literal string key wor

Re: [PR] some dependency removals and setup for refactor of `FileScanConfig` [datafusion]

2025-02-07 Thread via GitHub
alamb commented on code in PR #14543: URL: https://github.com/apache/datafusion/pull/14543#discussion_r1947226459 ## datafusion/core/src/datasource/data_source.rs: ## @@ -62,4 +62,8 @@ pub trait FileSource: Send + Sync { fn fmt_extra(&self, _t: DisplayFormatType, _f: &mut F

Re: [PR] some dependency removals and setup for refactor of `FileScanConfig` [datafusion]

2025-02-07 Thread via GitHub
logan-keede commented on code in PR #14543: URL: https://github.com/apache/datafusion/pull/14543#discussion_r1947229098 ## datafusion/core/src/datasource/data_source.rs: ## @@ -62,4 +62,8 @@ pub trait FileSource: Send + Sync { fn fmt_extra(&self, _t: DisplayFormatType, _f:

Re: [PR] feat: Add `array_min` function support [datafusion]

2025-02-07 Thread via GitHub
jayzhan211 commented on PR #14417: URL: https://github.com/apache/datafusion/pull/14417#issuecomment-2644276922 > DuckDB has list_max, and our array semantics are supposed to model Duck's list semantics, thus it makes sense to add to DataFusion core. > > Review-wise, let's do array_ma

Re: [PR] PoC Adaptive round robin repartitioning [datafusion]

2025-02-07 Thread via GitHub
github-actions[bot] commented on PR #13699: URL: https://github.com/apache/datafusion/pull/13699#issuecomment-2644423936 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] improve eliminate_outer_join rule [datafusion]

2025-02-07 Thread via GitHub
github-actions[bot] commented on PR #13249: URL: https://github.com/apache/datafusion/pull/13249#issuecomment-2644423972 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Reorganize the Parser module [datafusion-sqlparser-rs]

2025-02-07 Thread via GitHub
github-actions[bot] commented on PR #1581: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1581#issuecomment-2644425356 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or

Re: [PR] fix: Mark cast from float/double to decimal as incompatible [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove commented on code in PR #1372: URL: https://github.com/apache/datafusion-comet/pull/1372#discussion_r1947005968 ## spark/src/test/scala/org/apache/comet/exec/CometAggregateSuite.scala: ## @@ -32,13 +32,31 @@ import org.apache.spark.sql.internal.SQLConf import org.ap

Re: [I] Test DataFusion 45.0.0 with Sail [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #14408: URL: https://github.com/apache/datafusion/issues/14408#issuecomment-2643703822 From my perspective it seems like all the tests in DataFusion are passing, so clearly we have some sort of test coverage / gap @shehabgamin I wonder if this would be a goo

Re: [PR] Add `DataFusionError::Collection` to return multiple `DataFusionError`s [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14439: URL: https://github.com/apache/datafusion/pull/14439#issuecomment-2643724150 I merged up from main to resolve a conflict -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] refactor: switch BooleanBufferBuilder to NullBufferBuilder in MaybeNullBufferBuilder [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14504: URL: https://github.com/apache/datafusion/pull/14504#issuecomment-2643732450 Pretty stoked about this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] refactor: switch BooleanBufferBuilder to NullBufferBuilder in MaybeNullBufferBuilder [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14504: URL: https://github.com/apache/datafusion/pull/14504#issuecomment-2643732036 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] refactor: switch BooleanBufferBuilder to NullBufferBuilder in MaybeNullBufferBuilder [datafusion]

2025-02-07 Thread via GitHub
alamb merged PR #14504: URL: https://github.com/apache/datafusion/pull/14504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Allow constructing ScalarUDF from shared implementation [datafusion]

2025-02-07 Thread via GitHub
alamb commented on code in PR #14541: URL: https://github.com/apache/datafusion/pull/14541#discussion_r1946990866 ## datafusion/expr/src/udaf.rs: ## @@ -119,9 +119,12 @@ impl AggregateUDF { where F: AggregateUDFImpl + 'static, { -Self { -in

Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]

2025-02-07 Thread via GitHub
Xuanwo commented on issue #14535: URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2643713623 > I bet we are not the only project that would like to have SQLLancer type support in Rust That's really cool! Databend will definitely be interesting in building this pro

Re: [PR] minor: polish MemoryStream related code [datafusion]

2025-02-07 Thread via GitHub
alamb merged PR #14537: URL: https://github.com/apache/datafusion/pull/14537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
kevinjqliu commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643727227 Thank you @timsaucer i verified locally that that indeed resolved the issue for `test_string_functions` I also pushed up a fix for `test_relational_expr` (which was o

Re: [PR] Minor: Update changelog for `45.0.0` and tweak `CHANGELOG` docs [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14545: URL: https://github.com/apache/datafusion/pull/14545#issuecomment-2643724722 Thank you @comphead for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Allow constructing ScalarUDF from shared implementation [datafusion]

2025-02-07 Thread via GitHub
findepi commented on code in PR #14541: URL: https://github.com/apache/datafusion/pull/14541#discussion_r1947197039 ## datafusion/expr/src/udaf.rs: ## @@ -119,9 +119,12 @@ impl AggregateUDF { where F: AggregateUDFImpl + 'static, { -Self { -

Re: [PR] Add `DataFusionError::Collection` to return multiple `DataFusionError`s [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14439: URL: https://github.com/apache/datafusion/pull/14439#issuecomment-2644153033 I am really sorry -- somehow I have messed up the tests on this PR. I will monkey around with them to get them passing 🐒 🤔 -- This is an automated message from the Apache Git Servic

Re: [PR] some dependency removals and setup for refactor of `FileScanConfig` [datafusion]

2025-02-07 Thread via GitHub
alamb commented on code in PR #14543: URL: https://github.com/apache/datafusion/pull/14543#discussion_r1947215863 ## datafusion/core/src/datasource/data_source.rs: ## @@ -62,4 +62,8 @@ pub trait FileSource: Send + Sync { fn fmt_extra(&self, _t: DisplayFormatType, _f: &mut F

Re: [I] Access a Map with a non-string keys [datafusion]

2025-02-07 Thread via GitHub
findepi commented on issue #11785: URL: https://github.com/apache/datafusion/issues/11785#issuecomment-2644158363 This is still a problem. The error message requires the index to be Int64, but it doesn't work even if it is Int64 ``` query error SELECT column1[arrow_cast(1, 'Int6

Re: [PR] some dependency removals and setup for refactor of `FileScanConfig` [datafusion]

2025-02-07 Thread via GitHub
logan-keede commented on code in PR #14543: URL: https://github.com/apache/datafusion/pull/14543#discussion_r1947224752 ## datafusion/core/src/datasource/data_source.rs: ## @@ -62,4 +62,8 @@ pub trait FileSource: Send + Sync { fn fmt_extra(&self, _t: DisplayFormatType, _f:

Re: [PR] Extending support for INDEX parsing [datafusion-sqlparser-rs]

2025-02-07 Thread via GitHub
LucaCappelletti94 commented on code in PR #1707: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1707#discussion_r1946968076 ## src/ast/ddl.rs: ## @@ -1030,20 +1030,29 @@ impl fmt::Display for KeyOrIndexDisplay { /// [1]: https://dev.mysql.com/doc/refman/8.0/en/cre

Re: [PR] Allow constructing ScalarUDF from shared implementation [datafusion]

2025-02-07 Thread via GitHub
alamb commented on code in PR #14541: URL: https://github.com/apache/datafusion/pull/14541#discussion_r1946983077 ## datafusion/expr/src/udaf.rs: ## @@ -119,9 +119,12 @@ impl AggregateUDF { where F: AggregateUDFImpl + 'static, { -Self { -in

Re: [I] Question: `to_char(date, timstamp format)` [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #14536: URL: https://github.com/apache/datafusion/issues/14536#issuecomment-2643690203 @xudong963 maybe we should make the error message better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #14535: URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2643689301 I bet we are not the only project that would like to have SQLLancer type support in Rust Perhaps we can join forces (aka with Risingwave) ps. @Xuanwo -- here is anot

Re: [I] Report multiple errors, not just the first one [datafusion]

2025-02-07 Thread via GitHub
alamb closed issue #13676: Report multiple errors, not just the first one URL: https://github.com/apache/datafusion/issues/13676 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add `DataFusionError::Collection` to return multiple `DataFusionError`s [datafusion]

2025-02-07 Thread via GitHub
alamb merged PR #14439: URL: https://github.com/apache/datafusion/pull/14439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Add `DataFusionError::Collection` to return multiple `DataFusionError`s [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14439: URL: https://github.com/apache/datafusion/pull/14439#issuecomment-2644228452 Thanks again @eliaperantoni -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] feat: Add fair unified memory pool [datafusion-comet]

2025-02-07 Thread via GitHub
kazuyukitanimura commented on PR #1369: URL: https://github.com/apache/datafusion-comet/pull/1369#issuecomment-2644227538 @andygrove I would like to make sure to include this for the 0.6.0 release -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] some dependency removals and setup for refactor of `FileScanConfig` [datafusion]

2025-02-07 Thread via GitHub
alamb merged PR #14543: URL: https://github.com/apache/datafusion/pull/14543 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

[PR] fix: Do not add implicit groupBy expressions when building logical plans from Substrait [datafusion]

2025-02-07 Thread via GitHub
anlinc opened a new pull request, #14553: URL: https://github.com/apache/datafusion/pull/14553 ## Which issue does this PR close? Closes #14348 ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested?

Re: [PR] feat: metadata columns [datafusion]

2025-02-07 Thread via GitHub
chenkovsky commented on PR #14057: URL: https://github.com/apache/datafusion/pull/14057#issuecomment-2644389349 > > as I previously asked, in your implementation "a system column stops being a system column once it's projected" ? If this is correct, then as you said there's no need to a

Re: [PR] Minor: Update changelog for `45.0.0` and tweak `CHANGELOG` docs [datafusion]

2025-02-07 Thread via GitHub
alamb merged PR #14545: URL: https://github.com/apache/datafusion/pull/14545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Always use `StringViewArray` as output of `substr` [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14498: URL: https://github.com/apache/datafusion/pull/14498#issuecomment-2643744227 Thank you @Kev1n8 and @2010YOUY01 -- I am running some benchmarks on this PR to see if it makes any difference -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Fix: Avoid recursive external error wrapping [datafusion]

2025-02-07 Thread via GitHub
alamb commented on PR #14371: URL: https://github.com/apache/datafusion/pull/14371#issuecomment-2643730192 > The extended tests are going to fail on main until we merge [apache/datafusion-testing#6](https://github.com/apache/datafusion-testing/pull/6) and update the testing pin Filed

[I] Extended tests are failing on main [datafusion]

2025-02-07 Thread via GitHub
alamb opened a new issue, #14549: URL: https://github.com/apache/datafusion/issues/14549 ### Describe the bug Example: https://github.com/apache/datafusion/actions/runs/13203566677/job/36861234807 ``` External error: query is expected to fail with error: (regex) Data

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
kevinjqliu commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643871160 cool as long as CI works, i think we're ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] Upgrade to DataFusion 45 [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove closed issue #1304: Upgrade to DataFusion 45 URL: https://github.com/apache/datafusion-comet/issues/1304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] Feb 4, 2025: This week(s) in DataFusion [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #14491: URL: https://github.com/apache/datafusion/issues/14491#issuecomment-2643831696 > Interesting I can not see Empathic as DataFusion users Maybe it will be V2 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] perf: improve performance of update metrics [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove commented on code in PR #1329: URL: https://github.com/apache/datafusion-comet/pull/1329#discussion_r1947082907 ## native/core/src/execution/jni_api.rs: ## @@ -233,11 +242,12 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_createPlan( stream

Re: [I] Add example to spark-expr crate [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove commented on issue #1365: URL: https://github.com/apache/datafusion-comet/issues/1365#issuecomment-2643838235 Thanks for looking at this @viczsaurav. Here is some sample code that may be useful while exploring this. ```rust #[tokio::test] async fn test()

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
kevinjqliu commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643889554 > Is it possible there's an issue with your submodule that contains the data? It looks like CI unit tests are passing also. I think thats the issue, i didnt see that `

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
kevinjqliu commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643887392 I double checked by running that test with v44.1.0 and the plans are the same -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
timsaucer merged PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
kevinjqliu commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643950080 @timsaucer im still seeing this issue on my local setup. Following the same steps as the CI test https://github.com/apache/datafusion-python/blob/93ac6a820353b3ddea014be1

Re: [I] Create UNION plan node with correct schema [datafusion]

2025-02-07 Thread via GitHub
rkrishn7 commented on issue #14380: URL: https://github.com/apache/datafusion/issues/14380#issuecomment-2644143759 +1 on moving `ExpandWildcardRule`! > We might need to perform union coercion twice: once in the builder, and once in TypeCoercion(after coercion of exprs). Hmm, my

Re: [PR] DataFusion Ray rewrite to connect stages with Arrow Flight Streaming [datafusion-ray]

2025-02-07 Thread via GitHub
robtandy commented on code in PR #60: URL: https://github.com/apache/datafusion-ray/pull/60#discussion_r1947205698 ## datafusion_ray/core.py: ## @@ -0,0 +1,327 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NO

Re: [PR] Support WITHIN GROUP syntax to standardize certain existing aggregate functions [datafusion]

2025-02-07 Thread via GitHub
Garamda commented on PR #13511: URL: https://github.com/apache/datafusion/pull/13511#issuecomment-2644417031 Whenever the main branch is merged, conflicts can arise in certain test files that I also modified in this pull request. I have resolved these conflicts immediately after they occurr

Re: [PR] feat: metadata columns [datafusion]

2025-02-07 Thread via GitHub
chenkovsky commented on PR #14057: URL: https://github.com/apache/datafusion/pull/14057#issuecomment-2644417122 > > > > for stopping system column propagation, have you tested other logical plans e.g. union intersect? > > > > > > > > > I have not tried constructing logical plans d

Re: [PR] Relax physical schema validation [datafusion]

2025-02-07 Thread via GitHub
findepi commented on code in PR #14519: URL: https://github.com/apache/datafusion/pull/14519#discussion_r1946883460 ## datafusion/core/src/physical_planner.rs: ## @@ -689,7 +693,7 @@ impl DefaultPhysicalPlanner { if physical_field.data_type() != logical

Re: [I] [DISCUSS] Lower Friction / Lower Ceremony (faster?) releases [datafusion]

2025-02-07 Thread via GitHub
findepi commented on issue #14428: URL: https://github.com/apache/datafusion/issues/14428#issuecomment-2643541485 can "deciding whether to release" be "automated" by marking mandatory issues and PRs as release-blockers? The more external testing we want to do, the harder is to release. Ye

Re: [PR] Allow constructing ScalarUDF from shared implementation [datafusion]

2025-02-07 Thread via GitHub
findepi commented on code in PR #14541: URL: https://github.com/apache/datafusion/pull/14541#discussion_r1946891887 ## datafusion/expr/src/udaf.rs: ## @@ -119,9 +119,12 @@ impl AggregateUDF { where F: AggregateUDFImpl + 'static, { -Self { -

Re: [I] Proper NULL handling in array functions [datafusion]

2025-02-07 Thread via GitHub
alan910127 commented on issue #14451: URL: https://github.com/apache/datafusion/issues/14451#issuecomment-2643568090 @jkosh44 Thanks for all this information! I’ll take a look at the PR later and likely review it. If I have any thoughts or suggestions, I’ll leave comments there. -- This

Re: [PR] Chore/upgrade datafusion 45 [datafusion-python]

2025-02-07 Thread via GitHub
timsaucer commented on PR #1010: URL: https://github.com/apache/datafusion-python/pull/1010#issuecomment-2643548740 @kevinjqliu I just pushed to your branch an update to the return type. As we've been updating we've seen a lot of these small changes from string -> string_view. -- This

[PR] Add support for MS Varbinary(MAX) (#1714) [datafusion-sqlparser-rs]

2025-02-07 Thread via GitHub
TylerBrinks opened a new pull request, #1715: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1715 Hi, I'm the maintainer of https://github.com/TylerBrinks/SqlParser-cs, a C# port of this library. I'm finally getting around to submitting my first PR for this library based on an

[PR] feat: instrument spawned tasks with current tracing span when `tracing` feature is enabled [datafusion]

2025-02-07 Thread via GitHub
geoffreyclaude opened a new pull request, #14547: URL: https://github.com/apache/datafusion/pull/14547 ## Which issue does this PR close? Relates to #9415. Does not fully close the issue, but moves forward with a pre-requisite. ## Rationale for this change This allows Da

Re: [PR] 14044/enhancement/add xxhash algorithms in expression api [datafusion]

2025-02-07 Thread via GitHub
Spaarsh commented on PR #14367: URL: https://github.com/apache/datafusion/pull/14367#issuecomment-2643627690 The tests are failing due to code in the main itself at the time of branch creation. Is this PR ready for review? -- This is an automated message from the Apache Git Service. To re

Re: [PR] fix: disable checking for uint_8 and uint_16 if complex type readers are enabled [datafusion-comet]

2025-02-07 Thread via GitHub
andygrove commented on code in PR #1376: URL: https://github.com/apache/datafusion-comet/pull/1376#discussion_r1946952144 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -125,6 +125,26 @@ class CometExpressionSuite extends CometTestBase with AdaptiveS

Re: [PR] Parse Snowflake COPY INTO [datafusion-sqlparser-rs]

2025-02-07 Thread via GitHub
alamb commented on PR #1669: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1669#issuecomment-2643650028 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] datafusion-substrait API docs on docs.rs are broken [datafusion]

2025-02-07 Thread via GitHub
alamb commented on issue #13853: URL: https://github.com/apache/datafusion/issues/13853#issuecomment-2643646150 After releasing version 45 the docs are back ❤ 👓 https://docs.rs/datafusion-substrait/latest/datafusion_substrait/ ![Image](https://github.com/user-attachments/asset

  1   2   3   >