[PR] feat: support enable_url_table config [datafusion-python]

2024-12-21 Thread via GitHub
chenkovsky opened a new pull request, #980: URL: https://github.com/apache/datafusion-python/pull/980 # Which issue does this PR close? Closes #. # Rationale for this change without url table support, we have to create table, if we want to query a file. # What ch

Re: [PR] Support 1 or 3 arg in generate_series() UDTF [datafusion]

2024-12-21 Thread via GitHub
goldmedal commented on code in PR #13856: URL: https://github.com/apache/datafusion/pull/13856#discussion_r1894761801 ## datafusion/functions-table/src/generate_series.rs: ## @@ -22,22 +22,25 @@ use async_trait::async_trait; use datafusion_catalog::Session; use datafusion_cata

Re: [I] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-21 Thread via GitHub
zhuqi-lucas commented on issue #13873: URL: https://github.com/apache/datafusion/issues/13873#issuecomment-2558320184 cc @alamb @Dandandan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Re-support `Signature::Any(0)`, and add tests for zero argument udfs [datafusion]

2024-12-21 Thread via GitHub
jayzhan211 commented on PR #13871: URL: https://github.com/apache/datafusion/pull/13871#issuecomment-2558319784 I update the fix here https://github.com/alamb/datafusion/pull/22 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[PR] Consolidate Example: dataframe_output.rs into dataframe.rs [datafusion]

2024-12-21 Thread via GitHub
zhuqi-lucas opened a new pull request, #13877: URL: https://github.com/apache/datafusion/pull/13877 ## Which issue does this PR close? Closes [#13844)](https://github.com/apache/datafusion/issues/13844) ## Rationale for this change Part of https://github.com/apache/datafu

Re: [I] Consolidate Example: dataframe_output.rs into dataframe.rs [datafusion]

2024-12-21 Thread via GitHub
zhuqi-lucas commented on issue #13844: URL: https://github.com/apache/datafusion/issues/13844#issuecomment-2558314282 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Generate GroupByHash output in multiple RecordBatches [datafusion]

2024-12-21 Thread via GitHub
github-actions[bot] closed pull request #11758: Generate GroupByHash output in multiple RecordBatches URL: https://github.com/apache/datafusion/pull/11758 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Fix DISTINCT ON expressions match ORDER BY expressions check - normal… [datafusion]

2024-12-21 Thread via GitHub
github-actions[bot] commented on PR #13039: URL: https://github.com/apache/datafusion/pull/13039#issuecomment-2558299054 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] feat: support inner iejoin [datafusion]

2024-12-21 Thread via GitHub
github-actions[bot] commented on PR #12754: URL: https://github.com/apache/datafusion/pull/12754#issuecomment-2558299074 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] feat: Add a `spark.comet.exec.memoryPool` configuration for experimenting with various datafusion memory pool setups. [datafusion-comet]

2024-12-21 Thread via GitHub
andygrove commented on PR #1021: URL: https://github.com/apache/datafusion-comet/pull/1021#issuecomment-2558294675 > I have noticed that for now Comet biases towards always using the unified memory manager, so this patch for native memory management may not be relevant, so I prefer closing

Re: [PR] Re-support `Signature::Any(0)`, and add tests for zero argument udfs [datafusion]

2024-12-21 Thread via GitHub
jayzhan211 commented on PR #13871: URL: https://github.com/apache/datafusion/pull/13871#issuecomment-2558295033 We need to keep `TypeSignature::Any(0)` instead of switching to `TypeSignature::Nullary`? Isn't it better to force to use `Nullary` -- This is an automated message from the Apac

Re: [I] Functionality of `array_repeat` udf [datafusion]

2024-12-21 Thread via GitHub
jayzhan211 commented on issue #13872: URL: https://github.com/apache/datafusion/issues/13872#issuecomment-2558290456 empty array `[]` and array of null with len = 1 `[]` looks the same but they are different. > SELECT array_repeat(ARRAY[1, 2], null) I think we can return null

Re: [PR] minor: move shuffle classes from common to spark [datafusion-comet]

2024-12-21 Thread via GitHub
andygrove merged PR #1193: URL: https://github.com/apache/datafusion-comet/pull/1193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Support `Ident` length limits in dialects [datafusion-sqlparser-rs]

2024-12-21 Thread via GitHub
tobyhede commented on issue #1491: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1491#issuecomment-2558286190 @iajoiner Ident is currently a "universal" struct for capturing any identifier. I think adding limits will require capturing the different subtypes of Ident,

Re: [I] could not compile `sqlparser` [datafusion-sqlparser-rs]

2024-12-21 Thread via GitHub
tobyhede commented on issue #1612: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1612#issuecomment-2558284568 It might be worth either removing or updating .tool_versions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Handle empty projection in Postgres SELECT statements [datafusion-sqlparser-rs]

2024-12-21 Thread via GitHub
tobyhede commented on PR #1613: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1613#issuecomment-2558284318 Hadn't used the dialect flags before, is much cleaner. Thanks @iffyio -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] ci improvements [datafusion]

2024-12-21 Thread via GitHub
Omega359 commented on PR #13876: URL: https://github.com/apache/datafusion/pull/13876#issuecomment-2558283740 an uncached build: https://github.com/apache/datafusion/actions/runs/12449465274 a cached build: https://github.com/apache/datafusion/actions/runs/12449630002 -- This is an aut

Re: [PR] ci improvements [datafusion]

2024-12-21 Thread via GitHub
Omega359 commented on PR #13876: URL: https://github.com/apache/datafusion/pull/13876#issuecomment-2558282900 I've reverted the nextest changes for now as the github action to install the binary of it is not approved and building nextest from source would take longer than the time it saves

[PR] minor: move shuffle classes from common to spark [datafusion-comet]

2024-12-21 Thread via GitHub
andygrove opened a new pull request, #1193: URL: https://github.com/apache/datafusion-comet/pull/1193 ## Which issue does this PR close? N/A but this was necessary in https://github.com/apache/datafusion-comet/pull/1192 so am breaking this out into a separate PR to keep code

Re: [PR] Parse Postgres's LOCK TABLE statement [datafusion-sqlparser-rs]

2024-12-21 Thread via GitHub
freshtonic commented on code in PR #1614: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1614#discussion_r1894703918 ## src/ast/mod.rs: ## @@ -3336,6 +3336,27 @@ pub enum Statement { is_eq: bool, }, /// ```sql +/// LOCK [ TABLE ] [ ONLY ] nam

Re: [PR] Parse Postgres's LOCK TABLE statement [datafusion-sqlparser-rs]

2024-12-21 Thread via GitHub
freshtonic commented on code in PR #1614: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1614#discussion_r1894703311 ## src/parser/mod.rs: ## @@ -9604,7 +9604,13 @@ impl<'a> Parser<'a> { top = Some(self.parse_top()?); } -let projecti

Re: [PR] feat: Move shuffle block decompression and decoding to native code [datafusion-comet]

2024-12-21 Thread via GitHub
codecov-commenter commented on PR #1192: URL: https://github.com/apache/datafusion-comet/pull/1192#issuecomment-2558263848 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1192?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] ci improvements [datafusion]

2024-12-21 Thread via GitHub
Omega359 commented on PR #13876: URL: https://github.com/apache/datafusion/pull/13876#issuecomment-2558259239 > Thanks @Omega359 is it a draft? I dont see tests in checks No, but it should be. I can't seem to make it draft but apparently taiki-e/install-action@nextest is not an allowe

Re: [PR] ci improvements [datafusion]

2024-12-21 Thread via GitHub
comphead commented on PR #13876: URL: https://github.com/apache/datafusion/pull/13876#issuecomment-2558251132 is it a draft? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add documentation for `SHOW FUNCTIONS` [datafusion]

2024-12-21 Thread via GitHub
comphead merged PR #13868: URL: https://github.com/apache/datafusion/pull/13868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Minor: improve error message when ARRAY literals can not be planned [datafusion]

2024-12-21 Thread via GitHub
comphead merged PR #13859: URL: https://github.com/apache/datafusion/pull/13859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] feat: Move shuffle block decompression and decoding to native code [datafusion-comet]

2024-12-21 Thread via GitHub
andygrove commented on code in PR #1192: URL: https://github.com/apache/datafusion-comet/pull/1192#discussion_r1894693226 ## spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala: ## @@ -462,37 +461,6 @@ class CometExecSuite extends CometTestBase { } } - tes

[PR] ci improvements [datafusion]

2024-12-21 Thread via GitHub
Omega359 opened a new pull request, #13876: URL: https://github.com/apache/datafusion/pull/13876 ## Which issue does this PR close? Closes #13846, part of #13845 ## Rationale for this change Improve ci run time ## What changes are included in this PR?

Re: [I] Test DataFusion 44.0.0 with delta.rs [datafusion]

2024-12-21 Thread via GitHub
alamb commented on issue #13834: URL: https://github.com/apache/datafusion/issues/13834#issuecomment-2558201899 @rtyler has made a PR here: - https://github.com/delta-io/delta-rs/pull/3073 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Complete / integrate sqlite sqllogictest test scripts integrattion [datafusion]

2024-12-21 Thread via GitHub
alamb commented on issue #13812: URL: https://github.com/apache/datafusion/issues/13812#issuecomment-2558201563 > separate repo and use git modules for attaching to df repo (my preferred solution - datafusion-testing) yes this is my preferred solution too -- This is an auto

Re: [I] Improve efficiency of CI checks (so we can add MORE!) [datafusion]

2024-12-21 Thread via GitHub
Omega359 commented on issue #13845: URL: https://github.com/apache/datafusion/issues/13845#issuecomment-2558187140 Maintaining an extended workflow shouldn't be too bad tbh. I think having a workflow that runs outside of PR's (iow runs nightly) could be useful as well for expensive tests th

Re: [PR] Add sum statistics and PhysicalExpr::column_statistics [datafusion]

2024-12-21 Thread via GitHub
alamb commented on PR #13736: URL: https://github.com/apache/datafusion/pull/13736#issuecomment-2558186777 I have been thinking a lot about this PR and I don't want to let it die because we are stuck in trying to figure out a broader staistics question. I would like to find an incremental w

Re: [I] Complete / integrate sqlite sqllogictest test scripts integrattion [datafusion]

2024-12-21 Thread via GitHub
Omega359 commented on issue #13812: URL: https://github.com/apache/datafusion/issues/13812#issuecomment-2558185238 Ok, I believe I'm ready to push a PR for the source code since the sqllogictest-rs folks pushed a release with my changes. I just need a decision on what to do with the actual

Re: [I] Rename / simplify `BuiltInWindowExpr` / `BuiltInWindowFunctionExpr` [datafusion]

2024-12-21 Thread via GitHub
alamb closed issue #13473: Rename / simplify `BuiltInWindowExpr` / `BuiltInWindowFunctionExpr` URL: https://github.com/apache/datafusion/issues/13473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Rename / simplify `BuiltInWindowExpr` / `BuiltInWindowFunctionExpr` [datafusion]

2024-12-21 Thread via GitHub
alamb commented on issue #13473: URL: https://github.com/apache/datafusion/issues/13473#issuecomment-2558177455 Thanks to @irenjj I think this one is more or less done, so closing the ticket -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[PR] Minor: remove unused async-compression feature [datafusion]

2024-12-21 Thread via GitHub
jonahgao opened a new pull request, #13875: URL: https://github.com/apache/datafusion/pull/13875 ## Which issue does this PR close? N/A ## Rationale for this change async-compression provides two [IO implementations](https://docs.rs/async-compression/0.4.18/async_com

[PR] feat: Move shuffle block decompression and decoding to native code [datafusion-comet]

2024-12-21 Thread via GitHub
andygrove opened a new pull request, #1192: URL: https://github.com/apache/datafusion-comet/pull/1192 ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/1123 This helps with the preparation for https://github.com/apache/datafusi

Re: [PR] Minor: fix: Include FetchRel when producing LogicalPlan from Sort [datafusion]

2024-12-21 Thread via GitHub
alamb merged PR #13862: URL: https://github.com/apache/datafusion/pull/13862 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Substrait roundtrip fails for Sort with a fetch [datafusion]

2024-12-21 Thread via GitHub
alamb closed issue #13860: Substrait roundtrip fails for Sort with a fetch URL: https://github.com/apache/datafusion/issues/13860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Improve header size on summary page [datafusion-site]

2024-12-21 Thread via GitHub
alamb closed pull request #50: Improve header size on summary page URL: https://github.com/apache/datafusion-site/pull/50 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Improve header size on summary page [datafusion-site]

2024-12-21 Thread via GitHub
alamb commented on PR #50: URL: https://github.com/apache/datafusion-site/pull/50#issuecomment-2558152603 I just checked and I think it looks pretty good to me now. Thank you @timsaucer I am not a visual designer so maybe someone else with a better eye could finagle it more. It does

[PR] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-21 Thread via GitHub
zhuqi-lucas opened a new pull request, #13874: URL: https://github.com/apache/datafusion/pull/13874 ## Which issue does this PR close? Closes [#13873](https://github.com/apache/datafusion/issues/13873) ## Rationale for this change ```rust DataFrameWriteOptions is missing a

Re: [PR] Improve header size on summary page [datafusion-site]

2024-12-21 Thread via GitHub
timsaucer commented on PR #50: URL: https://github.com/apache/datafusion-site/pull/50#issuecomment-2558141265 The change I proposed just merged in and is live. Is that good or do you want it even smaller? Here is what it looks like on mobile devices ![image](https://github.co

Re: [PR] Minor: fix: Include FetchRel when producing LogicalPlan from Sort [datafusion]

2024-12-21 Thread via GitHub
robtandy commented on code in PR #13862: URL: https://github.com/apache/datafusion/pull/13862#discussion_r1894640008 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -368,14 +368,45 @@ pub fn to_substrait_rel( .iter() .map(|e| substra

Re: [PR] Minor: fix: Include FetchRel when producing LogicalPlan from Sort [datafusion]

2024-12-21 Thread via GitHub
robtandy commented on code in PR #13862: URL: https://github.com/apache/datafusion/pull/13862#discussion_r1894639389 ## datafusion/substrait/src/logical_plan/producer.rs: ## @@ -368,14 +368,45 @@ pub fn to_substrait_rel( .iter() .map(|e| substra

Re: [PR] feat(substrait): modular substrait consumer [datafusion]

2024-12-21 Thread via GitHub
alamb merged PR #13803: URL: https://github.com/apache/datafusion/pull/13803 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Implement `SHOW FUNCTIONS` [datafusion]

2024-12-21 Thread via GitHub
matthewmturner commented on PR #13799: URL: https://github.com/apache/datafusion/pull/13799#issuecomment-2558122828 Indeed, I am looking to incorporate something to improve the experience of looking at function help in dft. I have some ideas but haven't gotten around to it yet -- This is

Re: [PR] Parse Postgres's LOCK TABLE statement [datafusion-sqlparser-rs]

2024-12-21 Thread via GitHub
iffyio commented on code in PR #1614: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1614#discussion_r1894627136 ## src/parser/mod.rs: ## @@ -9604,7 +9604,13 @@ impl<'a> Parser<'a> { top = Some(self.parse_top()?); } -let projection =

Re: [PR] Handle empty projection in Postgres SELECT statements [datafusion-sqlparser-rs]

2024-12-21 Thread via GitHub
iffyio commented on code in PR #1613: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1613#discussion_r1894626201 ## src/parser/mod.rs: ## @@ -9604,7 +9604,13 @@ impl<'a> Parser<'a> { top = Some(self.parse_top()?); } -let projection =

Re: [I] 2gb parquet file takes 100s to process, even on second attempt (on main) [datafusion]

2024-12-21 Thread via GitHub
zhuqi-lucas commented on issue #13785: URL: https://github.com/apache/datafusion/issues/13785#issuecomment-2558067624 I create a issue for the improvement, i will create a PR soon, thanks. https://github.com/apache/datafusion/issues/13873 -- This is an automated message from the Apache

Re: [I] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-21 Thread via GitHub
zhuqi-lucas commented on issue #13873: URL: https://github.com/apache/datafusion/issues/13873#issuecomment-2558067485 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[I] Support (order by / sort) for DataFrameWriteOptions [datafusion]

2024-12-21 Thread via GitHub
zhuqi-lucas opened a new issue, #13873: URL: https://github.com/apache/datafusion/issues/13873 ### Is your feature request related to a problem or challenge? DataFrameWriteOptions is missing an order by / sort by like available in SQL. ### Describe the solution you'd like

Re: [PR] Support unparsing implicit lateral `UNNEST` plan to SQL text [datafusion]

2024-12-21 Thread via GitHub
goldmedal commented on code in PR #13824: URL: https://github.com/apache/datafusion/pull/13824#discussion_r1894582202 ## datafusion/sql/src/unparser/plan.rs: ## @@ -723,19 +733,48 @@ impl Unparser<'_> { internal_err!("Unnest input is not a Projection: {unne

Re: [PR] Support unparsing implicit lateral `UNNEST` plan to SQL text [datafusion]

2024-12-21 Thread via GitHub
goldmedal commented on code in PR #13824: URL: https://github.com/apache/datafusion/pull/13824#discussion_r1894582093 ## datafusion/sql/src/unparser/rewrite.rs: ## @@ -257,6 +267,43 @@ pub(super) fn subquery_alias_inner_query_and_columns( (outer_projections.input.as_ref(),