Re: [I] Main is broken [datafusion]

2025-04-29 Thread via GitHub
gstvg commented on issue #15896: URL: https://github.com/apache/datafusion/issues/15896#issuecomment-2840980768 After https://github.com/apache/datafusion/pull/15149, flatten stopped working for `List(FixedSizeList)` because it expected the inner fixed size list to be casted to list, which

Re: [PR] Add all missing table options to be handled in any order [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
tomershaniii commented on code in PR #1747: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1747#discussion_r2067961343 ## src/ast/helpers/stmt_create_table.rs: ## @@ -76,27 +78,20 @@ pub struct CreateTableBuilder { pub constraints: Vec, pub hive_distribut

[PR] Fix `flatten` scalar function when inner list is `FixedSizeList` [datafusion]

2025-04-29 Thread via GitHub
gstvg opened a new pull request, #15898: URL: https://github.com/apache/datafusion/pull/15898 ## Which issue does this PR close? ## Rationale for this change After #15149, `flatten` stopped working for `List(FixedSizeList)` because it expected the inner fixed size list to be ca

[I] Register schema table, failed to resolve schema [datafusion]

2025-04-29 Thread via GitHub
shencangsheng opened a new issue, #15897: URL: https://github.com/apache/datafusion/issues/15897 ### Describe the bug I registered a table named report.user using ctx.register_csv, but encountered a "failed to resolve schema: report" error in ctx.sql. ```rust pub async fn re

Re: [PR] feat: More warning info for users [datafusion-comet]

2025-04-29 Thread via GitHub
hsiang-c commented on code in PR #1667: URL: https://github.com/apache/datafusion-comet/pull/1667#discussion_r2067930692 ## spark/src/main/scala/org/apache/comet/DataTypeSupport.scala: ## @@ -33,20 +37,25 @@ trait DataTypeSupport { * @return * true if the datatype is s

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-04-29 Thread via GitHub
joroKr21 commented on PR #15149: URL: https://github.com/apache/datafusion/pull/15149#issuecomment-2840909664 Looks like a semantic merge conflict with #15160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067910538 ## src/test_utils.rs: ## @@ -166,6 +168,30 @@ impl TestedDialects { only_statement } +/// The same as [`one_statement_parses_to`] bu

Re: [I] Question: why is the Visitor trait limited to statements, relations & expressions? [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
ramnes commented on issue #934: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/934#issuecomment-2840901054 > Two options for a generalised Visitor trait come to mind: > > 1. expose pre + post trait method variants for every AST node type, or > 2. expose only two trai

Re: [PR] feat: More warning info for users [datafusion-comet]

2025-04-29 Thread via GitHub
hsiang-c commented on code in PR #1667: URL: https://github.com/apache/datafusion-comet/pull/1667#discussion_r2067903455 ## spark/src/main/scala/org/apache/comet/DataTypeSupport.scala: ## @@ -33,20 +37,25 @@ trait DataTypeSupport { * @return * true if the datatype is s

Re: [PR] Add all missing table options to be handled in any order [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1747: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1747#discussion_r2067864705 ## src/ast/helpers/stmt_create_table.rs: ## @@ -76,27 +78,20 @@ pub struct CreateTableBuilder { pub constraints: Vec, pub hive_distribution: H

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2067861834 ## src/dialect/mssql.rs: ## @@ -215,6 +218,78 @@ impl MsSqlDialect { })) } +/// Parse a SQL CREATE statement +fn parse_create(&s

Re: [PR] Add all missing table options to be handled in any order [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1747: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1747#discussion_r2067857586 ## src/parser/mod.rs: ## @@ -7081,18 +7029,243 @@ impl<'a> Parser<'a> { if let Token::Word(word) = self.peek_token().token {

Re: [PR] fix: fold cast null to substrait typed null [datafusion]

2025-04-29 Thread via GitHub
discord9 commented on code in PR #15854: URL: https://github.com/apache/datafusion/pull/15854#discussion_r2067801065 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -1590,6 +1590,21 @@ pub fn from_cast( schema: &DFSchemaRef, ) -> Result { let Cast { expr,

[I] Main is broken [datafusion]

2025-04-29 Thread via GitHub
xudong963 opened a new issue, #15896: URL: https://github.com/apache/datafusion/issues/15896 https://github.com/user-attachments/assets/777fe494-87e5-4698-835b-180bc793dff6"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Standardize CREATE TABLE options equals signs [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
github-actions[bot] commented on PR #1751: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1751#issuecomment-2840628297 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or

Re: [PR] Simple Functions Preview [datafusion]

2025-04-29 Thread via GitHub
github-actions[bot] commented on PR #14668: URL: https://github.com/apache/datafusion/pull/14668#issuecomment-2840626218 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] chore: Return NativeType instead of DataType for get_example_types [datafusion]

2025-04-29 Thread via GitHub
github-actions[bot] closed pull request #14778: chore: Return NativeType instead of DataType for get_example_types URL: https://github.com/apache/datafusion/pull/14778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Draft: LogicalScalar [datafusion]

2025-04-29 Thread via GitHub
github-actions[bot] closed pull request #14609: Draft: LogicalScalar URL: https://github.com/apache/datafusion/pull/14609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Make Expr::alias and alias_qualified smarter by calling unalias [datafusion]

2025-04-29 Thread via GitHub
github-actions[bot] commented on PR #14749: URL: https://github.com/apache/datafusion/pull/14749#issuecomment-2840626173 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] [wip] attach diagnostic to duplicate table name error [datafusion]

2025-04-29 Thread via GitHub
github-actions[bot] commented on PR #14767: URL: https://github.com/apache/datafusion/pull/14767#issuecomment-2840626137 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] chore : migrated all the UDFS to invoke_with_args [datafusion]

2025-04-29 Thread via GitHub
github-actions[bot] closed pull request #14779: chore : migrated all the UDFS to invoke_with_args URL: https://github.com/apache/datafusion/pull/14779 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Implement intermeidate result blocked approach sketch [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #15591: URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2840614135 This is very much on my list to review, but I am backed up and likely won't have a chance for another day or two -- This is an automated message from the Apache Git Service. To respo

Re: [PR] fix: Allow ORDER BY aggregates not present in SELECT list [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #15876: URL: https://github.com/apache/datafusion/pull/15876#issuecomment-2840609788 FYI @jonahgao -- would you possible have time to review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] doc: Update known users [datafusion]

2025-04-29 Thread via GitHub
alamb commented on code in PR #15895: URL: https://github.com/apache/datafusion/pull/15895#discussion_r2067724398 ## docs/source/user-guide/introduction.md: ## @@ -120,11 +120,11 @@ Here are some active projects using DataFusion: - [Polygon.io](https://polygon.io/) Stock Market

Re: [PR] Keeping pull request in sync with the base branch [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #15894: URL: https://github.com/apache/datafusion/pull/15894#issuecomment-2840606083 Thanks #15603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Keeping pull request in sync with the base branch [datafusion]

2025-04-29 Thread via GitHub
alamb merged PR #15894: URL: https://github.com/apache/datafusion/pull/15894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Keeping pull request in sync with the base branch [datafusion]

2025-04-29 Thread via GitHub
alamb closed issue #15877: Keeping pull request in sync with the base branch URL: https://github.com/apache/datafusion/issues/15877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] [substrait] Build basic test suite to validate produced Substrait plans [datafusion]

2025-04-29 Thread via GitHub
alamb commented on issue #15069: URL: https://github.com/apache/datafusion/issues/15069#issuecomment-2840601952 It is a good idea -- another potential issue is that it would effectively "tax" other features in the sense that writing tests for unrelated features might trigger a substrait bug

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #15168: URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2840589691 This looks great to me -- I plan to merge it tomorrow and start collecting next steps in a new `EPIC` ticket unless someone beats me to it -- This is an automated message from the Ap

Re: [PR] Support `GroupsAccumulator` for Avg duration [datafusion]

2025-04-29 Thread via GitHub
shruti2522 commented on PR #15748: URL: https://github.com/apache/datafusion/pull/15748#issuecomment-2840577253 Got it @alamb @goldmedal, will test and share results soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] chore: Prepare 0.8.1 release [branch-0.8] [datafusion-comet]

2025-04-29 Thread via GitHub
codecov-commenter commented on PR #1699: URL: https://github.com/apache/datafusion-comet/pull/1699#issuecomment-2840556859 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1699?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [I] Question: why is the Visitor trait limited to statements, relations & expressions? [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
freshtonic commented on issue #934: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/934#issuecomment-2840551720 > I can open a PR for this commit, but is this the direction we want to go in? I thought folks here wanted option 2 (which I didn't have the time to work on so far.

Re: [PR] Factor out Substrait consumers into separate files [datafusion]

2025-04-29 Thread via GitHub
vbarua commented on code in PR #15794: URL: https://github.com/apache/datafusion/pull/15794#discussion_r2067610862 ## datafusion/substrait/src/logical_plan/consumer/rex/extended_expression.rs: ## @@ -0,0 +1,109 @@ +// Licensed to the Apache Software Foundation (ASF) under one +/

Re: [PR] Factor out Substrait consumers into separate files [datafusion]

2025-04-29 Thread via GitHub
vbarua commented on code in PR #15794: URL: https://github.com/apache/datafusion/pull/15794#discussion_r2067574175 ## datafusion/substrait/src/logical_plan/consumer/mod.rs: ## @@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] Factor out Substrait consumers into separate files [datafusion]

2025-04-29 Thread via GitHub
vbarua commented on code in PR #15794: URL: https://github.com/apache/datafusion/pull/15794#discussion_r2067574175 ## datafusion/substrait/src/logical_plan/consumer/mod.rs: ## @@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

[PR] chore: Prepare 0.8.1 release [datafusion-comet]

2025-04-29 Thread via GitHub
andygrove opened a new pull request, #1699: URL: https://github.com/apache/datafusion-comet/pull/1699 ## Which issue does this PR close? N/A ## Rationale for this change We want to create a 0.8.1 release with a fix needed by Iceberg. ## What changes

Re: [I] Set up Comet + Iceberg integration tests in CI [datafusion-comet]

2025-04-29 Thread via GitHub
hsiang-c commented on issue #1685: URL: https://github.com/apache/datafusion-comet/issues/1685#issuecomment-2840445298 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] fix: fold cast null to substrait typed null [datafusion]

2025-04-29 Thread via GitHub
vbarua commented on code in PR #15854: URL: https://github.com/apache/datafusion/pull/15854#discussion_r2067003785 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -1590,6 +1590,21 @@ pub fn from_cast( schema: &DFSchemaRef, ) -> Result { let Cast { expr, da

Re: [PR] fix: fold cast null to substrait typed null [datafusion]

2025-04-29 Thread via GitHub
vbarua commented on code in PR #15854: URL: https://github.com/apache/datafusion/pull/15854#discussion_r2067003785 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -1590,6 +1590,21 @@ pub fn from_cast( schema: &DFSchemaRef, ) -> Result { let Cast { expr, da

Re: [I] Make ClickBench Q23 Go Faster [datafusion]

2025-04-29 Thread via GitHub
EmeraldShift commented on issue #15177: URL: https://github.com/apache/datafusion/issues/15177#issuecomment-2840412850 > They are currently exploring the possibility of using it alongside projections (a feature in ClickHouse akin to materialized views) to create secondary indexes and simila

Re: [PR] Add `union_tag` scalar function [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #14687: URL: https://github.com/apache/datafusion/pull/14687#issuecomment-2840392253 > LGTM. I think I'd like to see a test with multiple columns but the logic looks solid to me. I believe the use of unsafe is indeed ok given the conditions outlined. Thanks agai

Re: [PR] Add `union_tag` scalar function [datafusion]

2025-04-29 Thread via GitHub
alamb commented on code in PR #14687: URL: https://github.com/apache/datafusion/pull/14687#discussion_r2067521368 ## datafusion/sqllogictest/test_files/union_function.slt: ## @@ -45,3 +49,19 @@ select union_extract(union_column, 1) from union_table; query error DataFusion err

Re: [PR] Add `union_tag` scalar function [datafusion]

2025-04-29 Thread via GitHub
alamb commented on code in PR #14687: URL: https://github.com/apache/datafusion/pull/14687#discussion_r2067520961 ## datafusion/sqllogictest/test_files/union_function.slt: ## @@ -23,7 +26,8 @@ query ?I select union_column, union_extract(union_column, 'int') from union_table; -

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #15149: URL: https://github.com/apache/datafusion/pull/15149#issuecomment-2840387613 It appears the tests started failing on main after this PR was merged: - https://github.com/apache/datafusion/actions/runs/14740728702/job/41378119017 -- This is an automated mes

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067496065 ## tests/sqlparser_mssql.rs: ## @@ -1393,6 +1394,85 @@ fn parse_mssql_declare() { let _ = ms().verified_stmt(declare_cursor_for_select); } +#

Re: [PR] Add support for `GO` batch delimiter in SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on PR #1809: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1809#issuecomment-2840347205 @iffyio anything else on this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067493757 ## src/ast/mod.rs: ## @@ -4225,11 +4267,10 @@ impl fmt::Display for Statement { Statement::Fetch { name,

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2067492348 ## src/dialect/mssql.rs: ## @@ -215,6 +218,78 @@ impl MsSqlDialect { })) } +/// Parse a SQL CREATE statement +fn parse_create

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067470750 ## src/parser/mod.rs: ## @@ -4453,6 +4487,9 @@ impl<'a> Parser<'a> { break; } } +if le

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067467383 ## src/parser/mod.rs: ## @@ -8735,6 +8779,14 @@ impl<'a> Parser<'a> { }) } +/// Parse [Statement::Open] Review Comment: Done

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067462307 ## src/parser/mod.rs: ## @@ -8735,6 +8779,14 @@ impl<'a> Parser<'a> { }) } +/// Parse [Statement::Open] Review Comment: It's

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067461233 ## src/test_utils.rs: ## @@ -166,6 +168,30 @@ impl TestedDialects { only_statement } +/// The same as [`one_statement_parses_to`]

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067459659 ## tests/sqlparser_mssql.rs: ## @@ -1393,6 +1393,52 @@ fn parse_mssql_declare() { let _ = ms().verified_stmt(declare_cursor_for_select); } +#

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067442898 ## tests/sqlparser_mssql.rs: ## @@ -1393,6 +1393,52 @@ fn parse_mssql_declare() { let _ = ms().verified_stmt(declare_cursor_for_select); } +#

Re: [PR] Add `union_tag` scalar function [datafusion]

2025-04-29 Thread via GitHub
Omega359 commented on PR #14687: URL: https://github.com/apache/datafusion/pull/14687#issuecomment-2840240489 LGTM. I think I'd like to see a test with multiple columns but the logic looks solid to me. I believe the use of unsafe is indeed ok given the conditions outlined. -- This is an

Re: [I] [DISCUSSION] JOIN "task force" / project team [datafusion]

2025-04-29 Thread via GitHub
alamb commented on issue #15885: URL: https://github.com/apache/datafusion/issues/15885#issuecomment-2840233753 > not sure if it will help direction, cost nothing to share :) [Debunking the Myth of Join Ordering: Toward Robust SQL Analytics](https://arxiv.org/abs/2502.15181) I have t

Re: [I] Add diagrams for relationship between `FileSource`, `DataSource` and `DataSourceExec` [datafusion]

2025-04-29 Thread via GitHub
alamb commented on issue #15887: URL: https://github.com/apache/datafusion/issues/15887#issuecomment-2840224042 Thank you @onlyjackfrost -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Add Extension Type / Metadata support for Scalar UDFs [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #15646: URL: https://github.com/apache/datafusion/pull/15646#issuecomment-2840222682 The march to supporting user defined types has begun! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] docs: Add documentation for accelerating Iceberg Parquet scans with Comet [branch-0.8] [datafusion-comet]

2025-04-29 Thread via GitHub
andygrove merged PR #1696: URL: https://github.com/apache/datafusion-comet/pull/1696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067410508 ## src/ast/mod.rs: ## @@ -3032,6 +3068,14 @@ pub enum Statement { partition: Option>, }, /// ```sql +/// OPEN cursor_name +

Re: [I] Release sqlparser-rs version `0.56.0` around 2024-04-20 [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
alamb commented on issue #1756: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1756#issuecomment-2840210026 I have made a release candidate and started voting! - https://lists.apache.org/thread/d4y7jdwbyo4nk7zogp2qzxn7sj6nt3d0 -- This is an automated message from the Apa

Re: [PR] Prepare for 0.56.0 release: Version and CHANGELOG [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
alamb commented on PR #1822: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1822#issuecomment-2840208694 - Follow up PR is https://github.com/apache/datafusion-sqlparser-rs/pull/1832 -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] docs: Add documentation for accelerating Iceberg Parquet scans with Comet [branch-0.8] [datafusion-comet]

2025-04-29 Thread via GitHub
hsiang-c commented on code in PR #1696: URL: https://github.com/apache/datafusion-comet/pull/1696#discussion_r2067407711 ## docs/source/user-guide/datasources.md: ## @@ -19,29 +19,36 @@ # Supported Spark Data Sources -## Parquet +## File Formats + +### Parquet When `spar

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067391534 ## src/ast/mod.rs: ## @@ -3403,6 +3447,10 @@ pub enum Statement { /// Cursor name name: Ident, direction: FetchDirection,

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2067388595 ## src/dialect/mssql.rs: ## @@ -215,6 +218,78 @@ impl MsSqlDialect { })) } +/// Parse a SQL CREATE statement +fn parse_create(&s

Re: [PR] Feat: introduce `ExecutionPlan::partition_statistics` API [datafusion]

2025-04-29 Thread via GitHub
alamb merged PR #15852: URL: https://github.com/apache/datafusion/pull/15852 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Add `statistics_by_partition` API to `ExecutionPlan` [datafusion]

2025-04-29 Thread via GitHub
alamb closed issue #15495: Add `statistics_by_partition` API to `ExecutionPlan` URL: https://github.com/apache/datafusion/issues/15495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Feat: introduce `ExecutionPlan::partition_statistics` API [datafusion]

2025-04-29 Thread via GitHub
alamb commented on PR #15852: URL: https://github.com/apache/datafusion/pull/15852#issuecomment-284017 Let's get this one in -- it has been outstanding for too long. Thanks again @xudong963 and @berkaysynnada -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Saner handling of nulls inside arrays [datafusion]

2025-04-29 Thread via GitHub
thinkharderdev merged PR #15149: URL: https://github.com/apache/datafusion/pull/15149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
aharpervc commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2067364403 ## src/parser/mod.rs: ## @@ -5256,14 +5256,19 @@ impl<'a> Parser<'a> { pub fn parse_create_trigger( &mut self, +or_alter: boo

Re: [PR] Update `56.0.0` Changelog with latest commits [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
alamb merged PR #1832: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] chore: regenerate builtin functions coverage [datafusion-comet]

2025-04-29 Thread via GitHub
comphead commented on PR #1698: URL: https://github.com/apache/datafusion-comet/pull/1698#issuecomment-2840082085 Interesting we getting regressions for some string functions like - initcap - lower - upper - abs ... -- This is an automated message from the Apache Git Servi

Re: [PR] Update `56.0.0` Changelog with latest commits [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
alamb commented on PR #1832: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1832#issuecomment-2840054822 @iffyio FYI -- I think once this is merged, I'll make the RC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Prepare for 0.56.0 release: Version and CHANGELOG [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
alamb commented on PR #1822: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1822#issuecomment-2840044478 > Aah @alamb I got confused and merged the wrong PR here my bad 😞 realised now after the fact the ask was to merge the other PR into this one No worries -- I'll retar

Re: [PR] Prepare for 0.56.0 release: Version and CHANGELOG [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on PR #1822: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1822#issuecomment-2840029729 Aah @alamb I got confused and merged the wrong PR here my bad :disappointed: realised now after the fact the ask was to merge the other PR into this one -- This is an au

[PR] decode(col, 'UTF-8') support using cast [datafusion-comet]

2025-04-29 Thread via GitHub
mbutrovich opened a new pull request, #1697: URL: https://github.com/apache/datafusion-comet/pull/1697 ## Which issue does this PR close? Closes #. ## Rationale for this change Spark SQL's `decode` with two arguments converts a binary column to a string g

Re: [PR] Prepare for 0.56.0 release: Version and CHANGELOG [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio merged PR #1822: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2067197344 ## src/parser/mod.rs: ## @@ -5256,14 +5256,19 @@ impl<'a> Parser<'a> { pub fn parse_create_trigger( &mut self, +or_alter: bool,

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2067198693 ## src/parser/mod.rs: ## @@ -5256,14 +5256,19 @@ impl<'a> Parser<'a> { pub fn parse_create_trigger( &mut self, +or_alter: bool,

Re: [PR] support OR operator in binary `evaluate_bounds` [datafusion]

2025-04-29 Thread via GitHub
davidhewitt commented on code in PR #15716: URL: https://github.com/apache/datafusion/pull/15716#discussion_r2067195794 ## datafusion/physical-expr/src/intervals/cp_solver.rs: ## @@ -645,6 +645,17 @@ impl ExprIntervalGraph { .map(|child| self.graph[*child].inter

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2067197344 ## src/parser/mod.rs: ## @@ -5256,14 +5256,19 @@ impl<'a> Parser<'a> { pub fn parse_create_trigger( &mut self, +or_alter: bool,

[PR] chore: regenerate builtin functions coverage [datafusion-comet]

2025-04-29 Thread via GitHub
comphead opened a new pull request, #1698: URL: https://github.com/apache/datafusion-comet/pull/1698 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] Improve support for cursors for SQL Server [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1831: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1831#discussion_r2067185362 ## src/test_utils.rs: ## @@ -166,6 +168,30 @@ impl TestedDialects { only_statement } +/// The same as [`one_statement_parses_to`] bu

Re: [PR] feat: More warning info for users [datafusion-comet]

2025-04-29 Thread via GitHub
andygrove commented on code in PR #1667: URL: https://github.com/apache/datafusion-comet/pull/1667#discussion_r2067187303 ## spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala: ## @@ -1253,9 +1254,8 @@ object CometSparkSessionExtensions extends Logging {

Re: [PR] Prepare for 0.56.0 release: Version and CHANGELOG [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
alamb commented on PR #1822: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1822#issuecomment-2839962839 @iffyio -- I made a small PR to this one to update the change log with a few more commits (I can't push a commit to the validio-io repo) - https://github.com/validio

Re: [PR] feat: More warning info for users [datafusion-comet]

2025-04-29 Thread via GitHub
andygrove commented on code in PR #1667: URL: https://github.com/apache/datafusion-comet/pull/1667#discussion_r2067175853 ## spark/src/main/scala/org/apache/comet/DataTypeSupport.scala: ## @@ -33,20 +37,25 @@ trait DataTypeSupport { * @return * true if the datatype is

Re: [PR] docs: Add documentation for accelerating Iceberg Parquet scans with Comet [branch-0.8] [datafusion-comet]

2025-04-29 Thread via GitHub
codecov-commenter commented on PR #1696: URL: https://github.com/apache/datafusion-comet/pull/1696#issuecomment-2839941535 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1696?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] doc: Update known users [datafusion]

2025-04-29 Thread via GitHub
Copilot commented on code in PR #15895: URL: https://github.com/apache/datafusion/pull/15895#discussion_r2067157191 ## docs/source/user-guide/introduction.md: ## @@ -120,11 +120,11 @@ Here are some active projects using DataFusion: - [Polygon.io](https://polygon.io/) Stock Mark

Re: [PR] Added support for `DROP DOMAIN` [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
LucaCappelletti94 commented on code in PR #1828: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1828#discussion_r2067152516 ## src/ast/mod.rs: ## @@ -3319,6 +3319,18 @@ pub enum Statement { drop_behavior: Option, }, /// ```sql +/// DROP DOMAI

Re: [PR] Resolved bug in `parse_function_arg` [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
LucaCappelletti94 commented on code in PR #1826: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1826#discussion_r2067151721 ## src/parser/mod.rs: ## @@ -5199,12 +5199,22 @@ impl<'a> Parser<'a> { // parse: [ argname ] argtype let mut name = None;

Re: [PR] Added support for `DROP DOMAIN` [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1828: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1828#discussion_r2067148582 ## src/ast/mod.rs: ## @@ -3319,6 +3319,18 @@ pub enum Statement { drop_behavior: Option, }, /// ```sql +/// DROP DOMAIN +///

Re: [PR] Resolved bug in `parse_function_arg` [datafusion-sqlparser-rs]

2025-04-29 Thread via GitHub
iffyio commented on code in PR #1826: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1826#discussion_r2067143571 ## src/parser/mod.rs: ## @@ -5199,12 +5199,22 @@ impl<'a> Parser<'a> { // parse: [ argname ] argtype let mut name = None; +l

[PR] doc: Update known users [datafusion]

2025-04-29 Thread via GitHub
comphead opened a new pull request, #15895: URL: https://github.com/apache/datafusion/pull/15895 ## Which issue does this PR close? - Closes #. Update users description where missing and some punctuation ## Rationale for this change ## What changes

Re: [I] [DISCUSSION] Sorts being removed from subqueries [datafusion]

2025-04-29 Thread via GitHub
comphead commented on issue #15886: URL: https://github.com/apache/datafusion/issues/15886#issuecomment-2839773088 Agree this is confusing, DuckDb treats queries below as equal, I believe it just unnests outer query preserving whatever is in the inner query whereas DataFusion optimizes in d

Re: [PR] chore: update dev/release/rat_exclude_files.txt [datafusion-comet]

2025-04-29 Thread via GitHub
andygrove merged PR #1689: URL: https://github.com/apache/datafusion-comet/pull/1689 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

[PR] [branch-0.8] docs: Add documentation for accelerating Iceberg Parquet scans with Comet [datafusion-comet]

2025-04-29 Thread via GitHub
andygrove opened a new pull request, #1696: URL: https://github.com/apache/datafusion-comet/pull/1696 …omet (#1683) ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ##

Re: [PR] fix: fold cast null to substrait typed null [datafusion]

2025-04-29 Thread via GitHub
vbarua commented on code in PR #15854: URL: https://github.com/apache/datafusion/pull/15854#discussion_r2067014140 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -1590,6 +1590,21 @@ pub fn from_cast( schema: &DFSchemaRef, ) -> Result { let Cast { expr, da

Re: [PR] fix: fold cast null to substrait typed null [datafusion]

2025-04-29 Thread via GitHub
vbarua commented on code in PR #15854: URL: https://github.com/apache/datafusion/pull/15854#discussion_r2067003785 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -1590,6 +1590,21 @@ pub fn from_cast( schema: &DFSchemaRef, ) -> Result { let Cast { expr, da

Re: [PR] chore: update dev/release/rat_exclude_files.txt [datafusion-comet]

2025-04-29 Thread via GitHub
codecov-commenter commented on PR #1689: URL: https://github.com/apache/datafusion-comet/pull/1689#issuecomment-2839511977 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1689?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] Substrait: Handle inner map fields in schema renaming [datafusion]

2025-04-29 Thread via GitHub
gabotechs commented on code in PR #15869: URL: https://github.com/apache/datafusion/pull/15869#discussion_r2066875639 ## datafusion/substrait/tests/cases/substrait_validations.rs: ## @@ -61,16 +61,41 @@ mod tests { let proto_plan = read_json("tests

  1   2   >