Re: [PR] Resolved bug in `parse_function_arg` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
LucaCappelletti94 commented on code in PR #1826: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1826#discussion_r2071126270 ## src/parser/mod.rs: ## @@ -5199,13 +5199,20 @@ impl<'a> Parser<'a> { // parse: [ argname ] argtype let mut name = None;

Re: [PR] Unparse `UNNEST` projection with the table column alias [datafusion]

2025-05-01 Thread via GitHub
goldmedal commented on PR #15879: URL: https://github.com/apache/datafusion/pull/15879#issuecomment-2846244745 Thanks @blaginin @xudong963 @blaginin @alamb 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2071047425 ## src/parser/mod.rs: ## @@ -716,20 +734,36 @@ impl<'a> Parser<'a> { let condition = match &start_token.token { Token::Word(w) i

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio merged PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Added support for `DROP DOMAIN` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio merged PR #1828: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1828 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Support some of pipe operators [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio merged PR #1759: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Resolved bug in `parse_function_arg` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio commented on code in PR #1826: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1826#discussion_r2071042324 ## src/parser/mod.rs: ## @@ -5199,13 +5199,20 @@ impl<'a> Parser<'a> { // parse: [ argname ] argtype let mut name = None; +l

Re: [PR] [experiment] Generate event log in Chrome tracing format [datafusion-comet]

2025-05-01 Thread via GitHub
codecov-commenter commented on PR #1706: URL: https://github.com/apache/datafusion-comet/pull/1706#issuecomment-2846219034 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1706?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2071038220 ## src/dialect/mssql.rs: ## @@ -215,6 +225,59 @@ impl MsSqlDialect { })) } +/// Parse `CREATE TRIGGER` for [MsSql] +/// +///

[PR] Fix CI in main [datafusion]

2025-05-01 Thread via GitHub
blaginin opened a new pull request, #15917: URL: https://github.com/apache/datafusion/pull/15917 ## Which issue does this PR close? Fixes CI after https://github.com/apache/datafusion/pull/15168 ## Rationale for this change ## What changes are included in this PR?

Re: [I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on issue #15915: URL: https://github.com/apache/datafusion/issues/15915#issuecomment-2846193611 hi @alamb . it sounds interesting , i would love to work in this . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on issue #15915: URL: https://github.com/apache/datafusion/issues/15915#issuecomment-2846193703 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Update extending-operators.md [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2846154397 i Think ww need @alamb help now . could you help ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] support simple/cross lateral joins [datafusion]

2025-05-01 Thread via GitHub
github-actions[bot] closed pull request #14595: support simple/cross lateral joins URL: https://github.com/apache/datafusion/pull/14595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] feat: implement contextualized ObjectStore [datafusion]

2025-05-01 Thread via GitHub
github-actions[bot] closed pull request #14805: feat: implement contextualized ObjectStore URL: https://github.com/apache/datafusion/pull/14805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Add support for event tracing for visualizing where time is spent during execution [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on issue #1705: URL: https://github.com/apache/datafusion-comet/issues/1705#issuecomment-2846126081 @alamb This may also be interesting to explore for DataFusion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[PR] [experiment] Generate event log in Chrome tracing format [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove opened a new pull request, #1706: URL: https://github.com/apache/datafusion-comet/pull/1706 ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/1705 ## Rationale for this change This is a quick POC of gene

[I] Add support for event tracing for visualizing where time is spent during execution [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove opened a new issue, #1705: URL: https://github.com/apache/datafusion-comet/issues/1705 ### What is the problem the feature request solves? TIL I learned about Chrome's tracing log format. I built a very quick POC with Comet. It would be nice to have the ability to produce th

Re: [I] [datafusion-spark] Implement `ceil` function [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15916: URL: https://github.com/apache/datafusion/issues/15916#issuecomment-2846091416 @shehabgamin and @andygrove -- here is a ticket for another spark function. I am hoping we can do one or two of these functions to set a pattern, and then we will be able to basi

[I] [datafusion-spark] Implement `ceil` function [datafusion]

2025-05-01 Thread via GitHub
alamb opened a new issue, #15916: URL: https://github.com/apache/datafusion/issues/15916 ### Is your feature request related to a problem or challenge? Part of https://github.com/apache/datafusion/issues/15914 Given the importantance of spark functions in general, we are consoli

Re: [PR] chore: regenerate builtin functions coverage [datafusion-comet]

2025-05-01 Thread via GitHub
comphead closed pull request #1698: chore: regenerate builtin functions coverage URL: https://github.com/apache/datafusion-comet/pull/1698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Fix typo in introduction.md [datafusion]

2025-05-01 Thread via GitHub
xudong963 merged PR #15910: URL: https://github.com/apache/datafusion/pull/15910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
xudong963 commented on PR #15168: URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846060139 Fyi, the main CI has failed since the PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [I] Add memory profiling / logging [datafusion-comet]

2025-05-01 Thread via GitHub
alamb commented on issue #1701: URL: https://github.com/apache/datafusion-comet/issues/1701#issuecomment-2846047457 It would be amazing to have memory monitoring of native code in datafusion too -- it is an important feature that is currently hard for downstream crates -- This is an auto

[I] [datafusion-spark] Test integrating datafusion-spark code into comet [datafusion-comet]

2025-05-01 Thread via GitHub
alamb opened a new issue, #1704: URL: https://github.com/apache/datafusion-comet/issues/1704 ### What is the problem the feature request solves? - Part of https://github.com/apache/datafusion/issues/15914 @shehabgamin added the `datafusion-spark` crate in https://github.com/apa

Re: [I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15915: URL: https://github.com/apache/datafusion/issues/15915#issuecomment-2846046740 I think this is a good first issue as there is a clear request of what is desired and examples to follow -- This is an automated message from the Apache Git Service. To respond

Re: [I] Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
phillipleblanc commented on issue #15173: URL: https://github.com/apache/datafusion/issues/15173#issuecomment-2846043563 Yeah that makes sense. Part of the complexity here is that several of the features needed to make the ListingTableProvider work today (i.e. partition columns) are actuall

[I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
alamb opened a new issue, #15915: URL: https://github.com/apache/datafusion/issues/15915 ### Is your feature request related to a problem or challenge? - Part of https://github.com/apache/datafusion/issues/15914 @shehabgamin added the `datafusion-spark` crate in https://github.

Re: [PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-05-01 Thread via GitHub
xudong963 commented on code in PR #15841: URL: https://github.com/apache/datafusion/pull/15841#discussion_r2070946442 ## benchmarks/src/util/options.rs: ## @@ -72,16 +72,11 @@ impl CommonOpt { /// Modify the existing config appropriately pub fn update_config(&self, mut

Re: [I] [DISCUSSION] Add separate crate to cover spark builtin functions [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #5600: URL: https://github.com/apache/datafusion/issues/5600#issuecomment-2846025987 We are tracking follow on work in - https://github.com/apache/datafusion/issues/15914 -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
alamb merged PR #15168: URL: https://github.com/apache/datafusion/pull/15168 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15168: URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846025123 I have filed an epic to track filling out the datafusion-spark crate: - https://github.com/apache/datafusion/issues/15914 I will file some subtickets for follow on work as we

Re: [I] [DISCUSSION] Add separate crate to cover spark builtin functions [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #5600: [DISCUSSION] Add separate crate to cover spark builtin functions URL: https://github.com/apache/datafusion/issues/5600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15168: URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846025618 Onward! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

[I] [EPIC] Complete `datafusion-spark` Spark Compatible Functions [datafusion]

2025-05-01 Thread via GitHub
alamb opened a new issue, #15914: URL: https://github.com/apache/datafusion/issues/15914 ### Is your feature request related to a problem or challenge? Many DataFusion users are using DataFusion to execution workloads originally developed for Apache Spark. Examples include - [DataF

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070868393 ## datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt: ## @@ -81,11 +81,15 @@ EXPLAIN select a from t_pushdown where b > 2 ORDER BY a; logical_

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070868393 ## datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt: ## @@ -81,11 +81,15 @@ EXPLAIN select a from t_pushdown where b > 2 ORDER BY a; logical_

[PR] Add support for `DENY` statements [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
aharpervc opened a new pull request, #1836: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1836 This is another statement for SQL Server: https://learn.microsoft.com/en-us/sql/t-sql/statements/deny-transact-sql, but implemented in a common way. Similar to GRANT & REVOKE, so usi

Re: [I] Support file row index / row id for each file in a `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
acking-you commented on issue #15892: URL: https://github.com/apache/datafusion/issues/15892#issuecomment-2845864345 nice feature👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Support file row index / row id for each file in a `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15892: URL: https://github.com/apache/datafusion/issues/15892#issuecomment-2845830552 Related discussion: - https://github.com/apache/datafusion/issues/15173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
Omega359 commented on PR #15793: URL: https://github.com/apache/datafusion/pull/15793#issuecomment-2845818047 Beyond the config try_into looking like it's case sensitive for duration_format I think this is good. There is a breaking change with the rename of FormatOptions to OutputFormat tha

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
blaginin commented on code in PR #15793: URL: https://github.com/apache/datafusion/pull/15793#discussion_r2070830060 ## datafusion/sqllogictest/test_files/information_schema.slt: ## @@ -372,6 +381,15 @@ datafusion.explain.physical_plan_only false When set to true, the explain s

Re: [PR] Unparse `UNNEST` projection with the table column alias [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15879: URL: https://github.com/apache/datafusion/pull/15879#issuecomment-2845811697 Thank you @xudong963 and @goldmedal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Generate the common SQL for the unparsing result of the unnest [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #15233: Generate the common SQL for the unparsing result of the unnest URL: https://github.com/apache/datafusion/issues/15233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Unparse `UNNEST` projection with the table column alias [datafusion]

2025-05-01 Thread via GitHub
alamb merged PR #15879: URL: https://github.com/apache/datafusion/pull/15879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
Omega359 commented on code in PR #15793: URL: https://github.com/apache/datafusion/pull/15793#discussion_r2070825920 ## datafusion/sqllogictest/test_files/information_schema.slt: ## @@ -372,6 +381,15 @@ datafusion.explain.physical_plan_only false When set to true, the explain s

Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #14362: URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2845809583 See discussion on https://github.com/apache/datafusion/issues/15173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] metadata column support [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #13975: metadata column support URL: https://github.com/apache/datafusion/issues/13975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [I] metadata column support [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #13975: URL: https://github.com/apache/datafusion/issues/13975#issuecomment-2845808404 I think this is duplicated by https://github.com/apache/datafusion/issues/15173 so let's continue the discussion there -- This is an automated message from the Apache Git Servi

Re: [I] Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15173: URL: https://github.com/apache/datafusion/issues/15173#issuecomment-2845807026 There is another request here that is related I think (basically "row number within the file"): - https://github.com/apache/datafusion/issues/15892 It seems to me what ha

Re: [PR] feat: metadata columns [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #14057: URL: https://github.com/apache/datafusion/pull/14057#issuecomment-2845795660 Thank you @chenkovsky and @adriangb and everyone else who worked on this PR. I think the idea of additional metadata columns for LIstingTable provider is super valuable but it s

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
blaginin commented on PR #15793: URL: https://github.com/apache/datafusion/pull/15793#issuecomment-2845789235 Asked Bruce to re-review, and then I'm ready to merge 🤗 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Factor out Substrait consumers into separate files [datafusion]

2025-05-01 Thread via GitHub
alamb merged PR #15794: URL: https://github.com/apache/datafusion/pull/15794 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] [substrait] refactor consumer.rs [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #13864: [substrait] refactor consumer.rs URL: https://github.com/apache/datafusion/issues/13864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15793: URL: https://github.com/apache/datafusion/pull/15793#issuecomment-2845786865 Are we happy with this PR? Shall we merge it in? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Factor out Substrait consumers into separate files [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15794: URL: https://github.com/apache/datafusion/pull/15794#issuecomment-2845786048 Thanks again @vbarua and @gabotechs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
alamb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070810103 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { Ok(None) } -

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2845779215 > Hi @alamb. I've a WIP commit for this PR. I'm trying to both address @adriangb concerns and needs, and at the same time trying to keep complexity at minimum and trying to make things

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
alamb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070806936 ## datafusion/sqllogictest/test_files/push_down_filter.slt: ## @@ -218,43 +219,57 @@ LOCATION 'test_files/scratch/push_down_filter/t.parquet'; query TT explain sel

Re: [PR] chore: regenerate builtin functions coverage [datafusion-comet]

2025-05-01 Thread via GitHub
comphead commented on PR #1698: URL: https://github.com/apache/datafusion-comet/pull/1698#issuecomment-2845671771 > Interesting we getting regressions for some string functions like > > * initcap > * lower > * upper > * abs > ... > > @mbutrovich ^^ Might be

Re: [PR] chore: Remove fast encoding option [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on PR #1703: URL: https://github.com/apache/datafusion-comet/pull/1703#issuecomment-2845639468 > I love a good negative line count. Thanks @andygrove! Do we anticipate any performance changes? Based on local testing, TPC-H performance with fast encoding took 276s

Re: [PR] Improve sqllogictest error reporting [datafusion]

2025-05-01 Thread via GitHub
berkaysynnada commented on code in PR #15905: URL: https://github.com/apache/datafusion/pull/15905#discussion_r2070694736 ## datafusion/sqllogictest/bin/sqllogictests.rs: ## @@ -235,14 +235,38 @@ async fn run_test_file( runner.with_normalizer(value_normalizer); runner.

Re: [PR] chore: Remove fast encoding option [datafusion-comet]

2025-05-01 Thread via GitHub
codecov-commenter commented on PR #1703: URL: https://github.com/apache/datafusion-comet/pull/1703#issuecomment-2845566547 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1703?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [I] [DISCUSSION] DataFusion Road Map: Q3-Q4 2025 [datafusion]

2025-05-01 Thread via GitHub
Dandandan commented on issue #15878: URL: https://github.com/apache/datafusion/issues/15878#issuecomment-2845332274 I am currently interested in the following subjects where I'll probably experiment with some things or help out others. - [ ] Window Functions (profiling, implementing i

[PR] chore: Remove fast encoding option [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove opened a new pull request, #1703: URL: https://github.com/apache/datafusion-comet/pull/1703 ## Which issue does this PR close? N/A ## Rationale for this change Now that we are using a version of Arrow that lets us skip validation on IPC reads, t

Re: [I] Spark SQL test failures in native_iceberg_compat mode [datafusion-comet]

2025-05-01 Thread via GitHub
parthchandra commented on issue #1542: URL: https://github.com/apache/datafusion-comet/issues/1542#issuecomment-2845276570 Failure count : ``` Core 1: Tests: succeeded 9113, failed 25, canceled 6, ignored 292, pending 0 Core 2: Tests: succeeded 2636, failed 19, canceled 0, ignored

Re: [PR] Added SQL Example for `Aggregate Functions` [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on PR #15778: URL: https://github.com/apache/datafusion/pull/15778#issuecomment-2844743481 > This PR appears to hve no changes https://private-user-images.githubusercontent.com/490673/438510565-a8b12e9d-739c-4b18-9df2-69d64572667e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
kazuyukitanimura commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2070493011 ## spark/src/main/scala/org/apache/comet/CometExecIterator.scala: ## @@ -130,6 +134,21 @@ class CometExecIterator( def getNextBatch(): Option[Colum

Re: [PR] feat: decode() expression when using 'utf-8' encoding [datafusion-comet]

2025-05-01 Thread via GitHub
mbutrovich commented on PR #1697: URL: https://github.com/apache/datafusion-comet/pull/1697#issuecomment-2845174358 It's interesting to me that `native_comet` (CometScan) fails, but `native_datafusion` (DataSourceExec) succeeds. I wonder if DataSourceExec is already unpacking dictionaries

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2070469650 ## native/core/src/execution/jni_api.rs: ## @@ -359,6 +365,21 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan( // Retriev

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
comphead commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2070464759 ## native/core/src/execution/jni_api.rs: ## @@ -359,6 +365,21 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan( // Retrieve

Re: [PR] feat: regexp_replace() expression with no starting offset [datafusion-comet]

2025-05-01 Thread via GitHub
mbutrovich commented on code in PR #1700: URL: https://github.com/apache/datafusion-comet/pull/1700#discussion_r2070460938 ## spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala: ## @@ -188,6 +188,22 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSpar

Re: [PR] ci: require approving review [datafusion-python]

2025-05-01 Thread via GitHub
timsaucer merged PR #1122: URL: https://github.com/apache/datafusion-python/pull/1122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [PR] feat: regexp_replace() expression with no starting offset [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on code in PR #1700: URL: https://github.com/apache/datafusion-comet/pull/1700#discussion_r2070420243 ## spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala: ## @@ -188,6 +188,22 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSpark

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2052746932 ## datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt: ## @@ -81,11 +81,15 @@ EXPLAIN select a from t_pushdown where b > 2 ORDER BY a; logical_

Re: [PR] feat: regexp_replace() expression with no starting offset [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on code in PR #1700: URL: https://github.com/apache/datafusion-comet/pull/1700#discussion_r2070408978 ## spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala: ## @@ -188,6 +188,22 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSpark

Re: [PR] feat: More warning info for users [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove merged PR #1667: URL: https://github.com/apache/datafusion-comet/pull/1667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
codecov-commenter commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2845028742 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1702?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

[PR] add benchmark code for `Reuse rows in row cursor stream` [datafusion]

2025-05-01 Thread via GitHub
acking-you opened a new pull request, #15913: URL: https://github.com/apache/datafusion/pull/15913 ## Which issue does this PR close? - Closes #. ## Rationale for this change You can see:https://github.com/apache/datafusion/issues/15720#issuecomment-284453160

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070340727 ## datafusion/core/tests/physical_optimizer/push_down_filter.rs: ## @@ -154,29 +153,25 @@ impl FileSource for TestSource { fn try_pushdown_filters(

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
aharpervc commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2070337010 ## src/dialect/mssql.rs: ## @@ -215,6 +225,59 @@ impl MsSqlDialect { })) } +/// Parse `CREATE TRIGGER` for [MsSql] +/// +

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070335984 ## datafusion/datasource-parquet/src/source.rs: ## @@ -559,25 +549,8 @@ impl FileSource for ParquetSource { .predicate()

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2844943324 @mbutrovich I'd like to get your feedback on this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-05-01 Thread via GitHub
NevroHelios commented on PR #15841: URL: https://github.com/apache/datafusion/pull/15841#issuecomment-2844942814 I pushed the updates. Could you please run the ci again? @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Feat: support bit_count function [datafusion-comet]

2025-05-01 Thread via GitHub
kazantsev-maksim commented on PR #1602: URL: https://github.com/apache/datafusion-comet/pull/1602#issuecomment-2844940964 @mbutrovich I couldn't find a built-in implementation of bit_count in the DataFusion project, but i rewrote it using scalarFunc without adding a new proto expr. -- T

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
berkaysynnada commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2844916340 Hi @alamb. I've a WIP commit for this PR. I'm trying to both address @adriangb concerns and needs, and at the same time trying to keep complexity at minimum and trying to make

[PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove opened a new pull request, #1702: URL: https://github.com/apache/datafusion-comet/pull/1702 ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/1701 ## Rationale for this change We need a way to profile mem

[I] Wrong query results for filters that involve partition columns and data file columns [datafusion]

2025-05-01 Thread via GitHub
adriangb opened a new issue, #15912: URL: https://github.com/apache/datafusion/issues/15912 ### Describe the bug Filters such as `partition_col = col_from_file` are never applied if `datafusion.execution.parquet.pushdown_filters = true` ### To Reproduce With `datafusion-

[PR] feat: metadata handling for aggregates and window functions [datafusion]

2025-05-01 Thread via GitHub
timsaucer opened a new pull request, #15911: URL: https://github.com/apache/datafusion/pull/15911 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/15902 ## Rationale for this change This change is a follow on to https://github.com/apa

[PR] ci: require approving review [datafusion-python]

2025-05-01 Thread via GitHub
timsaucer opened a new pull request, #1122: URL: https://github.com/apache/datafusion-python/pull/1122 This is a change to the `.asf.yaml` file that requires an approving review to merge to `main`. I copied the configuration from our upstream `datafusion` repository. -- This is an automa

[PR] Fix typo in introduction.md [datafusion]

2025-05-01 Thread via GitHub
tom-mont opened a new pull request, #15910: URL: https://github.com/apache/datafusion/pull/15910 - Fix typo in introduction.md - Remove period from end of bullet point to maintain consistency with other bullet points ## Which issue does this PR close? - No issue: sim

Re: [PR] Allow stored procedures to be defined without `BEGIN`/`END` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
aharpervc commented on code in PR #1834: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1834#discussion_r2070278308 ## src/ast/mod.rs: ## @@ -3744,7 +3750,7 @@ pub enum Statement { or_alter: bool, name: ObjectName, params: Option>, -

Re: [PR] Update extending-operators.md [datafusion]

2025-05-01 Thread via GitHub
xudong963 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2844855985 > > > > You can rebase with main > > > > > > > > > doe this solve the issue ? > > > > > > You can open the failed CI and see what's wrong: > > ``` > > err

Re: [PR] chore(deps): bump assert_cmd from 2.0.16 to 2.0.17 [datafusion]

2025-05-01 Thread via GitHub
xudong963 merged PR #15909: URL: https://github.com/apache/datafusion/pull/15909 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
alamb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070243952 ## datafusion/core/src/datasource/listing/table.rs: ## @@ -982,18 +980,6 @@ impl TableProvider for ListingTable { return Ok(TableProviderFilterP

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070247089 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { Ok(None) }

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070247089 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { Ok(None) }

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070234428 ## datafusion/core/tests/physical_optimizer/push_down_filter.rs: ## @@ -154,29 +153,25 @@ impl FileSource for TestSource { fn try_pushdown_filters(

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
alamb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070220686 ## datafusion/core/tests/physical_optimizer/push_down_filter.rs: ## @@ -154,29 +153,25 @@ impl FileSource for TestSource { fn try_pushdown_filters( &

Re: [PR] Update extending-operators.md [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2844762089 > > > You can rebase with main > > > > > > doe this solve the issue ? > > You can open the failed CI and see what's wrong: > > ``` > error[E0599]: no method

  1   2   >