Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
ozankabak commented on PR #14271: URL: https://github.com/apache/datafusion/pull/14271#issuecomment-2623728047 Thanks for reviewing carefully, as always, much appreciated 🚀 > ```select a, agg(b) FROM ... GROUP BY a ORDER BY a, agg(b)``` You are right that all queries of this fo

Re: [PR] Fix `CREATE FUNCTION` round trip for Hive dialect [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
iffyio commented on PR #1693: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1693#issuecomment-2623763546 My bad, I somehow managed to miss that the test was failing before merging -- This is an automated message from the Apache Git Service. To respond to the message, please

[PR] Fix `CREATE FUNCTION` round trip for Hive dialect [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
iffyio opened a new pull request, #1693: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1693 Fixes the test failure in #1690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Add `enable_url_table` as a argument to SessionStateBuilder [datafusion]

2025-01-29 Thread via GitHub
alamb commented on issue #12394: URL: https://github.com/apache/datafusion/issues/12394#issuecomment-2622768363 Let's close this one for now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Add `enable_url_table` as a argument to SessionStateBuilder [datafusion]

2025-01-29 Thread via GitHub
alamb closed issue #12394: Add `enable_url_table` as a argument to SessionStateBuilder URL: https://github.com/apache/datafusion/issues/12394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] chore: Move all array_* serde to new framework, use correct INCOMPAT config [datafusion-comet]

2025-01-29 Thread via GitHub
andygrove commented on code in PR #1349: URL: https://github.com/apache/datafusion-comet/pull/1349#discussion_r1934596188 ## docs/source/user-guide/configs.md: ## @@ -64,6 +64,7 @@ Comet provides the following configuration settings. | spark.comet.explain.native.enabled | When

Re: [PR] Reduce size of `Expr` struct [datafusion]

2025-01-29 Thread via GitHub
findepi commented on code in PR #14366: URL: https://github.com/apache/datafusion/pull/14366#discussion_r1934670024 ## datafusion/expr/src/logical_plan/plan.rs: ## @@ -2420,19 +2420,24 @@ impl Window { .iter() .enumerate() .filter_map(|(idx

Re: [I] Simple Functions [datafusion]

2025-01-29 Thread via GitHub
adriangb commented on issue #12635: URL: https://github.com/apache/datafusion/issues/12635#issuecomment-2622869956 I'll let @davidhewitt chime in but we've experience a lot of generic bloat from having to implement functions that operate on scalars, arrays, dictionary arrays and take multip

Re: [I] Improve Parallel Reading (CSV, JSON) / Help Wanted [datafusion]

2025-01-29 Thread via GitHub
alamb commented on issue #8723: URL: https://github.com/apache/datafusion/issues/8723#issuecomment-2622766890 If anyone wants a fun exercise, getting the CSV reader to read in parallel from local files owuld greatly speed up the h2o benchmarks -- This is an automated message from the Apac

Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]

2025-01-29 Thread via GitHub
adriangb commented on PR #14362: URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2622993100 @chenkovsky could you maybe translate that into a test that we can add? I'm having trouble imagining in what sorts of situations this would apply. Generally in SQL if you have two c

[I] Create a wrapper class to access org.apache.arrow.c.SchemaImporter [datafusion-comet]

2025-01-29 Thread via GitHub
huaxingao opened a new issue, #1352: URL: https://github.com/apache/datafusion-comet/issues/1352 ### What is the problem the feature request solves? CometSchemaImporter is a Comet class but is in the org.apache.arrow.c package to overcome access restrictions (Arrow's SchemaImporter is

Re: [PR] Extend lambda support for ClickHouse, DuckDB and Generic dialects [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
iffyio commented on code in PR #1686: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1686#discussion_r1935083383 ## src/dialect/mod.rs: ## @@ -340,12 +340,21 @@ pub trait Dialect: Debug + Any { /// Returns true if the dialect supports lambda functions, for exa

Re: [PR] start refactoring process by setting up base + init [datafusion]

2025-01-29 Thread via GitHub
logan-keede commented on PR #14306: URL: https://github.com/apache/datafusion/pull/14306#issuecomment-2623485431 @Rachelint I have added the test to CI, Please review it whenever you can find some time. Thanks -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Fix DDL generation in case of an empty arguments function. [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
iffyio merged PR #1690: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
berkaysynnada commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1935090911 ## datafusion/core/tests/physical_optimizer/enforce_sorting.rs: ## @@ -238,6 +241,338 @@ async fn test_remove_unnecessary_sort5() -> Result<()> { Ok(())

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
berkaysynnada commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1935098072 ## datafusion/core/tests/physical_optimizer/enforce_sorting.rs: ## @@ -238,6 +241,338 @@ async fn test_remove_unnecessary_sort5() -> Result<()> { Ok(())

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
berkaysynnada commented on PR #14271: URL: https://github.com/apache/datafusion/pull/14271#issuecomment-2623769912 @alamb could you take a final look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Extend lambda support for ClickHouse, DuckDB and Generic dialects [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
samuelcolvin commented on code in PR #1686: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1686#discussion_r1935157823 ## src/dialect/mod.rs: ## @@ -340,12 +340,21 @@ pub trait Dialect: Debug + Any { /// Returns true if the dialect supports lambda functions, f

Re: [I] [Epic] Extract catalog functionality from the core to make it more modular [datafusion]

2025-01-29 Thread via GitHub
alamb commented on issue #10782: URL: https://github.com/apache/datafusion/issues/10782#issuecomment-2622779885 > Hi, I am working on moving `InformationSchema` into the `datafusion-catalog`. This would require moving `core/src/datasource/streaming.rs` (`StreaminTable`) to some place out of

[PR] Deprecate the use of `datafusion_sql::ResolvedTableReference and TableReference` [datafusion]

2025-01-29 Thread via GitHub
alamb opened a new pull request, #14365: URL: https://github.com/apache/datafusion/pull/14365 ## Which issue does this PR close? ## Rationale for this change Noticed while working on https://github.com/apache/datafusion/pull/14364 with @logan-keede `datafusion-sq

Re: [PR] Reduce size of `Expr` struct [datafusion]

2025-01-29 Thread via GitHub
alamb commented on code in PR #14366: URL: https://github.com/apache/datafusion/pull/14366#discussion_r1934699879 ## datafusion/expr/src/expr.rs: ## @@ -297,7 +298,7 @@ pub enum Expr { /// [`ExprFunctionExt`]: crate::expr_fn::ExprFunctionExt AggregateFunction(Aggregate

Re: [I] Build time regression [datafusion]

2025-01-29 Thread via GitHub
alamb commented on issue #14256: URL: https://github.com/apache/datafusion/issues/14256#issuecomment-2622937724 > Let me see if I can find ways to make Expr smaller though I can make Expr less than half the size in this PR: - https://github.com/apache/datafusion/pull/14366 I

Re: [PR] Support arrays_overlap function (alias of `array_has_any`) [datafusion]

2025-01-29 Thread via GitHub
alamb merged PR #14217: URL: https://github.com/apache/datafusion/pull/14217 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Improve deprecation message for MemoryExec [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14322: URL: https://github.com/apache/datafusion/pull/14322#issuecomment-2622945474 Thanks @xudong963 and @shehabgamin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Support arrays_overlap function [datafusion]

2025-01-29 Thread via GitHub
alamb closed issue #14216: Support arrays_overlap function URL: https://github.com/apache/datafusion/issues/14216 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] Minor: include the number of files run in sqllogictest display [datafusion]

2025-01-29 Thread via GitHub
alamb commented on code in PR #14359: URL: https://github.com/apache/datafusion/pull/14359#discussion_r1934703443 ## datafusion/sqllogictest/bin/sqllogictests.rs: ## @@ -184,7 +186,11 @@ async fn run_tests() -> Result<()> { .collect() .await; -m.println(f

Re: [PR] chore(deps): bump rustyline from 14.0.0 to 15.0.0 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14265: URL: https://github.com/apache/datafusion/pull/14265#issuecomment-2622947328 I am working to keep the dependencies updated and the PR queue lower -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] chore(deps): bump rustyline from 14.0.0 to 15.0.0 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb merged PR #14265: URL: https://github.com/apache/datafusion/pull/14265 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] multiply overflow in stats.rs [datafusion]

2025-01-29 Thread via GitHub
LindaSummer commented on issue #13775: URL: https://github.com/apache/datafusion/issues/13775#issuecomment-2623179970 Hi, Sorry for delay on this issue. I will try to work on it now. 😊 Best Regards, Edward -- This is an automated message from the Apache Git Service.

Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]

2025-01-29 Thread via GitHub
adriangb commented on PR #14362: URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2623202562 Was that working in #14057? I didn't see a test for it. Hypothetically speaking we could do something in DFSchema to deduplicate but I worry that won't make it work e.g. we'll

Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]

2025-01-29 Thread via GitHub
adriangb commented on PR #14362: URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2623125556 Is this that important to support? The example seems a bit contrived, I think it'd be more reasonable if it occurred naturally as part of a join or something where a user could une

Re: [I] Jan 18, 2025: This week(s) in DataFusion [datafusion]

2025-01-29 Thread via GitHub
alamb commented on issue #14179: URL: https://github.com/apache/datafusion/issues/14179#issuecomment-2622790016 Thanks to some great work from @buraksenn @berkaysynnada and @logan-keede we have completed extracting physical optimizer rules: - https://github.com/apache/datafusion/issues/1

Re: [I] Build time regression [datafusion]

2025-01-29 Thread via GitHub
alamb commented on issue #14256: URL: https://github.com/apache/datafusion/issues/14256#issuecomment-2622793969 Thanks to some great work from @buraksenn @berkaysynnada and @logan-keede we have completed extracting physical optimizer rules: - https://github.com/apache/datafusion/issues/1

Re: [I] Build time regression [datafusion]

2025-01-29 Thread via GitHub
alamb commented on issue #14256: URL: https://github.com/apache/datafusion/issues/14256#issuecomment-2622793735 > After removing the WildcardOptions (by replacing it with an empty structure) I can see the build time drops. Removing the rule itself and the change in core doesn't help. It loo

Re: [I] LimitPushdown rule uncorrect remove some GlobalLimitExec [datafusion]

2025-01-29 Thread via GitHub
alamb closed issue #14204: LimitPushdown rule uncorrect remove some GlobalLimitExec URL: https://github.com/apache/datafusion/issues/14204 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] fix: LimitPushdown rule uncorrect remove some GlobalLimitExec [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14245: URL: https://github.com/apache/datafusion/pull/14245#issuecomment-2622814836 Thanks again @zhuqi-lucas and @xudong963 -- this PR took a while but I think things are good in the end -- This is an automated message from the Apache Git Service. To respond to t

Re: [PR] Deprecate the use of `datafusion_sql::ResolvedTableReference and TableReference` [datafusion]

2025-01-29 Thread via GitHub
alamb commented on code in PR #14365: URL: https://github.com/apache/datafusion/pull/14365#discussion_r1934560993 ## datafusion/sql/src/lib.rs: ## @@ -50,5 +50,9 @@ pub mod unparser; pub mod utils; mod values; +#[deprecated( +since = "45.0.0", +note = "use datafusion

Re: [PR] fix: LimitPushdown rule uncorrect remove some GlobalLimitExec [datafusion]

2025-01-29 Thread via GitHub
alamb merged PR #14245: URL: https://github.com/apache/datafusion/pull/14245 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] refactor: switch `BooleanBufferBuilder` to `NullBufferBuilder` in single_group_by [datafusion]

2025-01-29 Thread via GitHub
alamb merged PR #14360: URL: https://github.com/apache/datafusion/pull/14360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] move information_schema to datafusion-catalog [datafusion]

2025-01-29 Thread via GitHub
logan-keede commented on code in PR #14364: URL: https://github.com/apache/datafusion/pull/14364#discussion_r1934568525 ## datafusion/catalog/src/lib.rs: ## @@ -18,23 +18,264 @@ //! Interfaces and default implementations of catalogs and schemas. //! //! Implementations +//! *

Re: [PR] [substrait] Add support for ExtensionTable [datafusion]

2025-01-29 Thread via GitHub
vbarua commented on PR #13772: URL: https://github.com/apache/datafusion/pull/13772#issuecomment-2623341026 Apologies for the delay, I haven't had the bandwidth to follow up on this (and I still don't tbh). At this point I'm ambivalent about this capability, but I wouldn't vote again

Re: [I] Create a wrapper class to access org.apache.arrow.c.SchemaImporter [datafusion-comet]

2025-01-29 Thread via GitHub
parthchandra commented on issue #1352: URL: https://github.com/apache/datafusion-comet/issues/1352#issuecomment-2623341094 +1. CometSchemaImporter need never be exposed to Iceberg. -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] Replace is_sorted helper with standard one. [datafusion]

2025-01-29 Thread via GitHub
github-actions[bot] commented on PR #13608: URL: https://github.com/apache/datafusion/pull/13608#issuecomment-2623347375 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Add related source code locations to errors [datafusion]

2025-01-29 Thread via GitHub
alamb commented on code in PR #13664: URL: https://github.com/apache/datafusion/pull/13664#discussion_r1933729644 ## datafusion/sql/src/utils.rs: ## @@ -90,12 +90,35 @@ pub(crate) fn rebase_expr( .data() } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] Review Comment

Re: [PR] Add related source code locations to errors [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #13664: URL: https://github.com/apache/datafusion/pull/13664#issuecomment-2621422574 I merged up from main for the PR to get a clean CI test run -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Fix Float and Decimal coercion [datafusion]

2025-01-29 Thread via GitHub
ozankabak commented on PR #14273: URL: https://github.com/apache/datafusion/pull/14273#issuecomment-2621628206 As I said, I am OK with this change so merging is fine from my perspective -- but I don't think it is a great idea to pull the trigger before achieving consensus with people who le

[I] Remove `recompute_schema` usage from optimizer [datafusion]

2025-01-29 Thread via GitHub
findepi opened a new issue, #14357: URL: https://github.com/apache/datafusion/issues/14357 The basic assumption that for a given operator we can recompute its schema from inputs' schema is unsound. - metadata: for plans constructed from SQL metadata will usually be empty, but an appl

Re: [I] Incorrect result for IS NOT NULL predicate over UNION ALL query [datafusion]

2025-01-29 Thread via GitHub
findepi commented on issue #14352: URL: https://github.com/apache/datafusion/issues/14352#issuecomment-2621637814 This is actually two separate bugs: - initial plan for the `Union` already has the field marked as not-nullable, because the plan builder simply takes first input schema a

Re: [PR] Make TypedString contain Value instead of String to support and preserve other quote styles [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
MartinSahlen commented on code in PR #1679: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1679#discussion_r1933869079 ## tests/sqlparser_bigquery.rs: ## @@ -2214,6 +2214,30 @@ fn test_select_as_value() { assert_eq!(Some(ValueTableMode::AsValue), select.value_

Re: [PR] perf(array-agg): add fast path for array agg for `merge_batch` [datafusion]

2025-01-29 Thread via GitHub
alamb merged PR #14299: URL: https://github.com/apache/datafusion/pull/14299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Update version in `datafusion-cli/Dockerfile` to 1.81 [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14344: URL: https://github.com/apache/datafusion/pull/14344#issuecomment-2621905518 I tested that this builds locally via ```shell $ docker build -f datafusion-cli/Dockerfile . ``` It seems to work great Thanks @comphead and @findepi --

Re: [PR] chore: Move all array_* serde to new framework, use correct INCOMPAT config [datafusion-comet]

2025-01-29 Thread via GitHub
codecov-commenter commented on PR #1349: URL: https://github.com/apache/datafusion-comet/pull/1349#issuecomment-2621918258 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1349?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] chore(deps): bump aws-sdk-sso from 1.50.0 to 1.56.0 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14259: URL: https://github.com/apache/datafusion/pull/14259#issuecomment-2621924170 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] refactor: switch `BooleanBufferBuilder` to `NullBufferBuilder` in binary_map [datafusion]

2025-01-29 Thread via GitHub
Chen-Yuan-Lai commented on code in PR #14341: URL: https://github.com/apache/datafusion/pull/14341#discussion_r1934067753 ## datafusion/physical-expr-common/src/binary_map.rs: ## @@ -553,10 +553,11 @@ where /// Returns a `NullBuffer` with a single null value at the given inde

Re: [PR] chore(deps): bump aws-sdk-ssooidc from 1.51.0 to 1.57.1 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14314: URL: https://github.com/apache/datafusion/pull/14314#issuecomment-2621922553 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] chore(deps): bump home from 0.5.9 to 0.5.11 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14257: URL: https://github.com/apache/datafusion/pull/14257#issuecomment-2621923758 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] chore(deps): bump aws-sdk-sts from 1.51.0 to 1.57.0 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14263: URL: https://github.com/apache/datafusion/pull/14263#issuecomment-2621924416 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] chore(deps): bump rustyline from 14.0.0 to 15.0.0 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14265: URL: https://github.com/apache/datafusion/pull/14265#issuecomment-2621925843 MSRV is updated in - https://github.com/apache/datafusion/pull/14330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Update version in `datafusion-cli/Dockerfile` to 1.81 [datafusion]

2025-01-29 Thread via GitHub
alamb merged PR #14344: URL: https://github.com/apache/datafusion/pull/14344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] chore(deps): bump rustyline from 14.0.0 to 15.0.0 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14265: URL: https://github.com/apache/datafusion/pull/14265#issuecomment-2621926063 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] Feature: Monotonic Sets [datafusion]

2025-01-29 Thread via GitHub
berkaysynnada commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1934073481 ## datafusion/sqllogictest/test_files/aggregate.slt: ## @@ -4963,6 +4963,9 @@ false true NULL +statement ok Review Comment: done -- This is an au

[PR] Remove redundant processing from exprToProtoInternal [datafusion-comet]

2025-01-29 Thread via GitHub
andygrove opened a new pull request, #1351: URL: https://github.com/apache/datafusion-comet/pull/1351 ## Which issue does this PR close? N/A ## Rationale for this change We have two very similar methods, `exprToProto` and `exprToProtoInternal`. `exp

Re: [PR] Add Common Subexpression Elimination for `PhysicalExpr` trees [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #13046: URL: https://github.com/apache/datafusion/pull/13046#issuecomment-2621931000 Marking as draft as I am not sure what is happeing with this PR and I am trying to work down the review queue -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Feature: Monotonic Sets [datafusion]

2025-01-29 Thread via GitHub
berkaysynnada commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1934074103 ## datafusion/sqllogictest/test_files/aggregate.slt: ## @@ -4978,17 +4981,19 @@ logical_plan physical_plan 01)GlobalLimitExec: skip=0, fetch=5 02)--Coales

Re: [PR] [substrait] Add support for ExtensionTable [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #13772: URL: https://github.com/apache/datafusion/pull/13772#issuecomment-2621935202 Hi @ccciudatu and @vbarua -- I am trying to work down the backlog of PRs in the review queue -- what is the status of this PR? Is it ready for review / inclusion? Are you still worki

Re: [PR] add manual trigger for extended tests in pull requests [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14331: URL: https://github.com/apache/datafusion/pull/14331#issuecomment-2621909853 > I wonder if we can create a check in the PR to show that the running the tests is in progress, @edmondop I may be misunderstanding you, but I think once the `extended` job is

Re: [I] External Error prefix is repeated multiple times [datafusion]

2025-01-29 Thread via GitHub
getChan commented on issue #14080: URL: https://github.com/apache/datafusion/issues/14080#issuecomment-2621943510 Take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Add Common Subexpression Elimination for `PhysicalExpr` trees [datafusion]

2025-01-29 Thread via GitHub
andygrove commented on PR #13046: URL: https://github.com/apache/datafusion/pull/13046#issuecomment-2621949545 > @andygrove are you still interested in this feature / reviewing this PR? @peter-toth @alamb Apologies, I am still interested in this but other priorities came up and I have

Re: [PR] Extend lambda support for ClickHouse, DuckDB and Generic dialects [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
iffyio commented on code in PR #1686: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1686#discussion_r1934222955 ## src/dialect/mod.rs: ## @@ -340,12 +340,21 @@ pub trait Dialect: Debug + Any { /// Returns true if the dialect supports lambda functions, for exa

[PR] Unpin aws sdk dependencies in `datafusion-cli` [datafusion]

2025-01-29 Thread via GitHub
alamb opened a new pull request, #14361: URL: https://github.com/apache/datafusion/pull/14361 ## Which issue does this PR close? Now that we have updated MSRV, we can unpin some of the dependencies - #14330 ## Rationale for this change Keep up with dependencies

Re: [PR] Restore ability to run single SLT file [datafusion]

2025-01-29 Thread via GitHub
findepi commented on PR #14355: URL: https://github.com/apache/datafusion/pull/14355#issuecomment-2622167347 thank you @Omega359 @alamb ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] chore(deps): bump aws-sdk-ssooidc from 1.51.0 to 1.57.1 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb merged PR #14314: URL: https://github.com/apache/datafusion/pull/14314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] chore(deps): bump aws-sdk-ssooidc from 1.51.0 to 1.57.1 in /datafusion-cli [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14314: URL: https://github.com/apache/datafusion/pull/14314#issuecomment-2622170551 Trying to clear out the review queue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] [WIP] Introduce the "parser" feature to gate the SQL text processing and leaving only AST and other support types [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
felipecrv opened a new pull request, #1691: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1691 This crate can become a very lightweight dependency to `datafusion` (and other projects) that have their own SQL parser but need to use `datafusion-sqlparser-rs` AST types to interac

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
ozankabak commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1934350058 ## datafusion/sqllogictest/test_files/aggregate.slt: ## @@ -6203,3 +6203,97 @@ physical_plan 14)--PlaceholderRowExec 15)ProjectionExec:

Re: [PR] chore: Move all array_* serde to new framework, use correct INCOMPAT config [datafusion-comet]

2025-01-29 Thread via GitHub
andygrove commented on code in PR #1349: URL: https://github.com/apache/datafusion-comet/pull/1349#discussion_r1934352035 ## common/src/main/scala/org/apache/comet/CometConf.scala: ## @@ -605,6 +605,15 @@ object CometConf extends ShimCometConf { .booleanConf .creat

Re: [PR] chore: Move all array_* serde to new framework, use correct INCOMPAT config [datafusion-comet]

2025-01-29 Thread via GitHub
andygrove commented on code in PR #1349: URL: https://github.com/apache/datafusion-comet/pull/1349#discussion_r1934351236 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -929,6 +929,19 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde w

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
ozankabak commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1934352834 ## datafusion/expr/src/udaf.rs: ## @@ -818,6 +826,26 @@ pub mod aggregate_doc_sections { }; } +/// Status of an Aggregate Expression's Set Monotonicity +

Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]

2025-01-29 Thread via GitHub
adriangb commented on code in PR #14362: URL: https://github.com/apache/datafusion/pull/14362#discussion_r1934364024 ## datafusion/expr/src/utils.rs: ## @@ -736,11 +802,18 @@ pub fn exprlist_to_fields<'a>( .into_iter() .map(|c| c.flat_na

Re: [PR] expose write options [datafusion-python]

2025-01-29 Thread via GitHub
kylebarron commented on code in PR #1006: URL: https://github.com/apache/datafusion-python/pull/1006#discussion_r1934368047 ## src/options.rs: ## @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the

Re: [PR] chore: Prepare for DataFusion 45 (bump to DataFusion rev 5592834 + Arrow 54.0.0) [datafusion-comet]

2025-01-29 Thread via GitHub
andygrove commented on code in PR #1332: URL: https://github.com/apache/datafusion-comet/pull/1332#discussion_r1934474112 ## native/spark-expr/src/conversion_funcs/cast.rs: ## @@ -988,6 +988,9 @@ fn is_datafusion_spark_compatible( return true; } match from_typ

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
ozankabak commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1934480800 ## datafusion/core/tests/physical_optimizer/enforce_sorting.rs: ## @@ -238,6 +241,338 @@ async fn test_remove_unnecessary_sort5() -> Result<()> { Ok(()) }

Re: [PR] chore: Prepare for DataFusion 45 (bump to DataFusion rev 5592834 + Arrow 54.0.0) [datafusion-comet]

2025-01-29 Thread via GitHub
andygrove commented on code in PR #1332: URL: https://github.com/apache/datafusion-comet/pull/1332#discussion_r1934475131 ## native/core/src/execution/operators/scan.rs: ## @@ -304,11 +304,7 @@ fn scan_schema(input_batch: &InputBatch, data_types: &[DataType]) -> SchemaRef {

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
alamb commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r1934471695 ## datafusion/core/tests/physical_optimizer/enforce_sorting.rs: ## @@ -238,6 +241,338 @@ async fn test_remove_unnecessary_sort5() -> Result<()> { Ok(()) } +#

Re: [PR] Feature: AggregateMonotonicity [datafusion]

2025-01-29 Thread via GitHub
alamb commented on code in PR #14271: URL: https://github.com/apache/datafusion/pull/14271#discussion_r193447 ## datafusion/core/tests/physical_optimizer/enforce_sorting.rs: ## @@ -238,6 +241,338 @@ async fn test_remove_unnecessary_sort5() -> Result<()> { Ok(()) } +#

Re: [I] Add support for function chaining and the dot syntax for function calls [datafusion]

2025-01-29 Thread via GitHub
gstvg commented on issue #12206: URL: https://github.com/apache/datafusion/issues/12206#issuecomment-2621108439 Hi @doupache, are you working on it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Example for using a separate threadpool for CPU bound work (try 2) [datafusion]

2025-01-29 Thread via GitHub
JanKaul commented on PR #14286: URL: https://github.com/apache/datafusion/pull/14286#issuecomment-2621226726 I tried the dedicated executor in my [test example](https://github.com/JanKaul/cpu-io-executor/blob/main/src/two_runtimes.rs). I'm not entirely sure if I'm using it wrong or maybe my

[PR] Restore ability to run single SLT file [datafusion]

2025-01-29 Thread via GitHub
findepi opened a new pull request, #14355: URL: https://github.com/apache/datafusion/pull/14355 Make it possible again to run single SLT file, even if it's name is a substring of other file(s). For example, after the change, this command: cargo test --test sqllogictests -- t

[PR] Fix bug when parsing a Snowflake stage name that has a semicolon righ… [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
yoavcloud opened a new pull request, #1688: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1688 …t after -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [I] Implement xxhash algorithms as part of the expression API [datafusion]

2025-01-29 Thread via GitHub
HectorPascual commented on issue #14044: URL: https://github.com/apache/datafusion/issues/14044#issuecomment-2621513736 Hi @Spaarsh, The hashes match to the python module : ![Image](https://github.com/user-attachments/assets/d8f1f9e9-7a33-4e84-ad71-b07769a962cd) In r

[PR] Add parsing for GRANT ROLE and GRANT DATABASE ROLE in Snowflake dialect [datafusion-sqlparser-rs]

2025-01-29 Thread via GitHub
yoavcloud opened a new pull request, #1689: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1689 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] fix: Fall back to Spark when hashing decimals with precision > 18 [datafusion-comet]

2025-01-29 Thread via GitHub
andygrove commented on code in PR #1325: URL: https://github.com/apache/datafusion-comet/pull/1325#discussion_r1933815284 ## spark/src/main/scala/org/apache/comet/serde/hash.scala: ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more

Re: [I] External Error prefix is repeated multiple times [datafusion]

2025-01-29 Thread via GitHub
Omega359 commented on issue #14080: URL: https://github.com/apache/datafusion/issues/14080#issuecomment-2622008215 Note that when this is fixed the sqlite test files will need to be regenerated. See https://github.com/apache/datafusion/pull/14290 for a PR to do that. -- This is an automa

Re: [PR] Fix UNION field nullability tracking [datafusion]

2025-01-29 Thread via GitHub
Omega359 commented on code in PR #14356: URL: https://github.com/apache/datafusion/pull/14356#discussion_r1934134314 ## datafusion/expr/src/logical_plan/plan.rs: ## @@ -2645,6 +2643,106 @@ pub struct Union { pub schema: DFSchemaRef, } +impl Union { +/// Constructs ne

Re: [PR] Fix UNION field nullability tracking [datafusion]

2025-01-29 Thread via GitHub
Omega359 commented on code in PR #14356: URL: https://github.com/apache/datafusion/pull/14356#discussion_r1934137664 ## datafusion/expr/src/logical_plan/plan.rs: ## @@ -2645,6 +2643,106 @@ pub struct Union { pub schema: DFSchemaRef, } +impl Union { +/// Constructs ne

Re: [PR] Script and documentation for regenerating sqlite test files [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14290: URL: https://github.com/apache/datafusion/pull/14290#issuecomment-2622029735 I plan to test this (by way of review) later today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] perf(array-agg): add fast path for array agg for `merge_batch` [datafusion]

2025-01-29 Thread via GitHub
alamb commented on PR #14299: URL: https://github.com/apache/datafusion/pull/14299#issuecomment-2621915540 Thanks again @rluvaton -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Script and documentation for regenerating sqlite test files [datafusion]

2025-01-29 Thread via GitHub
Omega359 commented on code in PR #14290: URL: https://github.com/apache/datafusion/pull/14290#discussion_r1934302894 ## datafusion/sqllogictest/regenerate_sqlite_files.sh: ## @@ -0,0 +1,179 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or mo

Re: [I] User Defined Coercion Rules [datafusion]

2025-01-29 Thread via GitHub
findepi commented on issue #14296: URL: https://github.com/apache/datafusion/issues/14296#issuecomment-2622375940 I am OK having user-defined coercion rules as long as they are applied reasonably in the DataFusion core. In particular, the coercion rules should be applied by the analyzer,

Re: [PR] Script and documentation for regenerating sqlite test files [datafusion]

2025-01-29 Thread via GitHub
Omega359 commented on code in PR #14290: URL: https://github.com/apache/datafusion/pull/14290#discussion_r1934307461 ## datafusion/sqllogictest/regenerate_sqlite_files.sh: ## @@ -0,0 +1,179 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or mo

<    1   2   3   >