Re: [I] Support file row index / row id for each file in a `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15892: URL: https://github.com/apache/datafusion/issues/15892#issuecomment-2845830552 Related discussion: - https://github.com/apache/datafusion/issues/15173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
phillipleblanc commented on issue #15173: URL: https://github.com/apache/datafusion/issues/15173#issuecomment-2846043563 Yeah that makes sense. Part of the complexity here is that several of the features needed to make the ListingTableProvider work today (i.e. partition columns) are actuall

Re: [I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15915: URL: https://github.com/apache/datafusion/issues/15915#issuecomment-2846046740 I think this is a good first issue as there is a clear request of what is desired and examples to follow -- This is an automated message from the Apache Git Service. To respond

[I] [datafusion-spark] Test integrating datafusion-spark code into comet [datafusion-comet]

2025-05-01 Thread via GitHub
alamb opened a new issue, #1704: URL: https://github.com/apache/datafusion-comet/issues/1704 ### What is the problem the feature request solves? - Part of https://github.com/apache/datafusion/issues/15914 @shehabgamin added the `datafusion-spark` crate in https://github.com/apa

Re: [I] Add memory profiling / logging [datafusion-comet]

2025-05-01 Thread via GitHub
alamb commented on issue #1701: URL: https://github.com/apache/datafusion-comet/issues/1701#issuecomment-2846047457 It would be amazing to have memory monitoring of native code in datafusion too -- it is an important feature that is currently hard for downstream crates -- This is an auto

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
xudong963 commented on PR #15168: URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846060139 Fyi, the main CI has failed since the PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] Fix typo in introduction.md [datafusion]

2025-05-01 Thread via GitHub
xudong963 merged PR #15910: URL: https://github.com/apache/datafusion/pull/15910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] chore: regenerate builtin functions coverage [datafusion-comet]

2025-05-01 Thread via GitHub
comphead closed pull request #1698: chore: regenerate builtin functions coverage URL: https://github.com/apache/datafusion-comet/pull/1698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[I] [datafusion-spark] Implement `ceil` function [datafusion]

2025-05-01 Thread via GitHub
alamb opened a new issue, #15916: URL: https://github.com/apache/datafusion/issues/15916 ### Is your feature request related to a problem or challenge? Part of https://github.com/apache/datafusion/issues/15914 Given the importantance of spark functions in general, we are consoli

Re: [I] [datafusion-spark] Implement `ceil` function [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15916: URL: https://github.com/apache/datafusion/issues/15916#issuecomment-2846091416 @shehabgamin and @andygrove -- here is a ticket for another spark function. I am hoping we can do one or two of these functions to set a pattern, and then we will be able to basi

[PR] [experiment] Generate event log in Chrome tracing format [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove opened a new pull request, #1706: URL: https://github.com/apache/datafusion-comet/pull/1706 ## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/1705 ## Rationale for this change This is a quick POC of gene

Re: [I] Add support for event tracing for visualizing where time is spent during execution [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on issue #1705: URL: https://github.com/apache/datafusion-comet/issues/1705#issuecomment-2846126081 @alamb This may also be interesting to explore for DataFusion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] support simple/cross lateral joins [datafusion]

2025-05-01 Thread via GitHub
github-actions[bot] closed pull request #14595: support simple/cross lateral joins URL: https://github.com/apache/datafusion/pull/14595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] feat: implement contextualized ObjectStore [datafusion]

2025-05-01 Thread via GitHub
github-actions[bot] closed pull request #14805: feat: implement contextualized ObjectStore URL: https://github.com/apache/datafusion/pull/14805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [I] Reuse Rows allocation in SortPreservingMergeStream / `RowCursorStream` [datafusion]

2025-05-01 Thread via GitHub
acking-you commented on issue #15720: URL: https://github.com/apache/datafusion/issues/15720#issuecomment-2844531606 I think I need to submit two PRs to complete this issue: 1. Add benchmark code for SortPreservingMergeStream 2. Implement the reuse of Rows -- This is an automated

[PR] chore(deps): bump assert_cmd from 2.0.16 to 2.0.17 [datafusion]

2025-05-01 Thread via GitHub
dependabot[bot] opened a new pull request, #15909: URL: https://github.com/apache/datafusion/pull/15909 Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd) from 2.0.16 to 2.0.17. Changelog Sourced from https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md";>asser

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
alamb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070243952 ## datafusion/core/src/datasource/listing/table.rs: ## @@ -982,18 +980,6 @@ impl TableProvider for ListingTable { return Ok(TableProviderFilterP

Re: [PR] feat: decode() expression when using 'utf-8' encoding [datafusion-comet]

2025-05-01 Thread via GitHub
mbutrovich commented on PR #1697: URL: https://github.com/apache/datafusion-comet/pull/1697#issuecomment-2844679657 Linux is failing in CometFuzzTestSuite due to failing to cast a Dictionary to utf-8. RNG there must be generating different values. I'll look into that failure. -- This is

Re: [I] Reuse Rows allocation in SortPreservingMergeStream / `RowCursorStream` [datafusion]

2025-05-01 Thread via GitHub
acking-you commented on issue #15720: URL: https://github.com/apache/datafusion/issues/15720#issuecomment-2844672108 > # Overall Implementation > Adjust `RowCursorStream` to become the owner of `Rows` with continuous reuse, requiring each partition to maintain two `Rows` instances (subseq

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2070469650 ## native/core/src/execution/jni_api.rs: ## @@ -359,6 +365,21 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan( // Retriev

Re: [PR] feat: decode() expression when using 'utf-8' encoding [datafusion-comet]

2025-05-01 Thread via GitHub
mbutrovich commented on PR #1697: URL: https://github.com/apache/datafusion-comet/pull/1697#issuecomment-2845174358 It's interesting to me that `native_comet` (CometScan) fails, but `native_datafusion` (DataSourceExec) succeeds. I wonder if DataSourceExec is already unpacking dictionaries

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
kazuyukitanimura commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2070493011 ## spark/src/main/scala/org/apache/comet/CometExecIterator.scala: ## @@ -130,6 +134,21 @@ class CometExecIterator( def getNextBatch(): Option[Colum

Re: [PR] Added SQL Example for `Aggregate Functions` [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on PR #15778: URL: https://github.com/apache/datafusion/pull/15778#issuecomment-2844743481 > This PR appears to hve no changes https://private-user-images.githubusercontent.com/490673/438510565-a8b12e9d-739c-4b18-9df2-69d64572667e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9

Re: [PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-05-01 Thread via GitHub
NevroHelios commented on code in PR #15841: URL: https://github.com/apache/datafusion/pull/15841#discussion_r2069898066 ## datafusion/common/src/column.rs: ## @@ -130,8 +130,8 @@ impl Column { /// where `"foo.BAR"` would be parsed to a reference to column named `foo.BAR`

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
codecov-commenter commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2845028742 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1702?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] feat: More warning info for users [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove merged PR #1667: URL: https://github.com/apache/datafusion-comet/pull/1667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] feat: regexp_replace() expression with no starting offset [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on code in PR #1700: URL: https://github.com/apache/datafusion-comet/pull/1700#discussion_r2070408978 ## spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala: ## @@ -188,6 +188,22 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSpark

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2052746932 ## datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt: ## @@ -81,11 +81,15 @@ EXPLAIN select a from t_pushdown where b > 2 ORDER BY a; logical_

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070247089 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { Ok(None) }

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070247089 ## datafusion/physical-plan/src/execution_plan.rs: ## @@ -471,39 +471,53 @@ pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync { Ok(None) }

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#issuecomment-2844943324 @mbutrovich I'd like to get your feedback on this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-05-01 Thread via GitHub
NevroHelios commented on PR #15841: URL: https://github.com/apache/datafusion/pull/15841#issuecomment-2844942814 I pushed the updates. Could you please run the ci again? @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070335984 ## datafusion/datasource-parquet/src/source.rs: ## @@ -559,25 +549,8 @@ impl FileSource for ParquetSource { .predicate()

Re: [PR] Feat: support bit_count function [datafusion-comet]

2025-05-01 Thread via GitHub
kazantsev-maksim commented on PR #1602: URL: https://github.com/apache/datafusion-comet/pull/1602#issuecomment-2844940964 @mbutrovich I couldn't find a built-in implementation of bit_count in the DataFusion project, but i rewrote it using scalarFunc without adding a new proto expr. -- T

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
aharpervc commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2070337010 ## src/dialect/mssql.rs: ## @@ -215,6 +225,59 @@ impl MsSqlDialect { })) } +/// Parse `CREATE TRIGGER` for [MsSql] +/// +

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070340727 ## datafusion/core/tests/physical_optimizer/push_down_filter.rs: ## @@ -154,29 +153,25 @@ impl FileSource for TestSource { fn try_pushdown_filters(

Re: [PR] feat: ORDER BY ALL [datafusion]

2025-05-01 Thread via GitHub
PokIsemaine commented on code in PR #15772: URL: https://github.com/apache/datafusion/pull/15772#discussion_r2070204781 ## datafusion/expr/src/expr.rs: ## @@ -701,6 +701,24 @@ impl TryCast { } } +/// OrderBy Expressions +pub enum OrderByExprs { +OrderByExprVec(Vec),

Re: [PR] chore(deps): bump assert_cmd from 2.0.16 to 2.0.17 [datafusion]

2025-05-01 Thread via GitHub
xudong963 merged PR #15909: URL: https://github.com/apache/datafusion/pull/15909 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] feat: regexp_replace() expression with no starting offset [datafusion-comet]

2025-05-01 Thread via GitHub
mbutrovich commented on code in PR #1700: URL: https://github.com/apache/datafusion-comet/pull/1700#discussion_r2070460938 ## spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala: ## @@ -188,6 +188,22 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSpar

Re: [PR] ci: require approving review [datafusion-python]

2025-05-01 Thread via GitHub
timsaucer merged PR #1122: URL: https://github.com/apache/datafusion-python/pull/1122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
comphead commented on code in PR #1702: URL: https://github.com/apache/datafusion-comet/pull/1702#discussion_r2070464759 ## native/core/src/execution/jni_api.rs: ## @@ -359,6 +365,21 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_executePlan( // Retrieve

Re: [PR] Update extending-operators.md [datafusion]

2025-05-01 Thread via GitHub
xudong963 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2844855985 > > > > You can rebase with main > > > > > > > > > doe this solve the issue ? > > > > > > You can open the failed CI and see what's wrong: > > ``` > > err

Re: [PR] Allow stored procedures to be defined without `BEGIN`/`END` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
aharpervc commented on code in PR #1834: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1834#discussion_r2070278308 ## src/ast/mod.rs: ## @@ -3744,7 +3750,7 @@ pub enum Statement { or_alter: bool, name: ObjectName, params: Option>, -

[PR] Fix typo in introduction.md [datafusion]

2025-05-01 Thread via GitHub
tom-mont opened a new pull request, #15910: URL: https://github.com/apache/datafusion/pull/15910 - Fix typo in introduction.md - Remove period from end of bullet point to maintain consistency with other bullet points ## Which issue does this PR close? - No issue: sim

[PR] ci: require approving review [datafusion-python]

2025-05-01 Thread via GitHub
timsaucer opened a new pull request, #1122: URL: https://github.com/apache/datafusion-python/pull/1122 This is a change to the `.asf.yaml` file that requires an approving review to merge to `main`. I copied the configuration from our upstream `datafusion` repository. -- This is an automa

[PR] feat: metadata handling for aggregates and window functions [datafusion]

2025-05-01 Thread via GitHub
timsaucer opened a new pull request, #15911: URL: https://github.com/apache/datafusion/pull/15911 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/15902 ## Rationale for this change This change is a follow on to https://github.com/apa

[I] Wrong query results for filters that involve partition columns and data file columns [datafusion]

2025-05-01 Thread via GitHub
adriangb opened a new issue, #15912: URL: https://github.com/apache/datafusion/issues/15912 ### Describe the bug Filters such as `partition_col = col_from_file` are never applied if `datafusion.execution.parquet.pushdown_filters = true` ### To Reproduce With `datafusion-

[PR] feat: Add memory profiling [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove opened a new pull request, #1702: URL: https://github.com/apache/datafusion-comet/pull/1702 ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/1701 ## Rationale for this change We need a way to profile mem

Re: [I] Spark SQL test failures in native_iceberg_compat mode [datafusion-comet]

2025-05-01 Thread via GitHub
parthchandra commented on issue #1542: URL: https://github.com/apache/datafusion-comet/issues/1542#issuecomment-2845276570 Failure count : ``` Core 1: Tests: succeeded 9113, failed 25, canceled 6, ignored 292, pending 0 Core 2: Tests: succeeded 2636, failed 19, canceled 0, ignored

[PR] chore: Remove fast encoding option [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove opened a new pull request, #1703: URL: https://github.com/apache/datafusion-comet/pull/1703 ## Which issue does this PR close? N/A ## Rationale for this change Now that we are using a version of Arrow that lets us skip validation on IPC reads, t

Re: [I] [DISCUSSION] DataFusion Road Map: Q3-Q4 2025 [datafusion]

2025-05-01 Thread via GitHub
Dandandan commented on issue #15878: URL: https://github.com/apache/datafusion/issues/15878#issuecomment-2845332274 I am currently interested in the following subjects where I'll probably experiment with some things or help out others. - [ ] Window Functions (profiling, implementing i

Re: [PR] Improve sqllogictest error reporting [datafusion]

2025-05-01 Thread via GitHub
gabotechs commented on PR #15905: URL: https://github.com/apache/datafusion/pull/15905#issuecomment-2844394504 👍 Done! also added some failure indexes to help our eyes parse the different errors quickly -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] feat: regexp_replace() expression with no starting offset [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on code in PR #1700: URL: https://github.com/apache/datafusion-comet/pull/1700#discussion_r2070420243 ## spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala: ## @@ -188,6 +188,22 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSpark

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
alamb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070220686 ## datafusion/core/tests/physical_optimizer/push_down_filter.rs: ## @@ -154,29 +153,25 @@ impl FileSource for TestSource { fn try_pushdown_filters( &

Re: [PR] Update extending-operators.md [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2844762089 > > > You can rebase with main > > > > > > doe this solve the issue ? > > You can open the failed CI and see what's wrong: > > ``` > error[E0599]: no method

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15801: URL: https://github.com/apache/datafusion/pull/15801#discussion_r2070234428 ## datafusion/core/tests/physical_optimizer/push_down_filter.rs: ## @@ -154,29 +153,25 @@ impl FileSource for TestSource { fn try_pushdown_filters(

Re: [PR] refactor filter pushdown apis [datafusion]

2025-05-01 Thread via GitHub
berkaysynnada commented on PR #15801: URL: https://github.com/apache/datafusion/pull/15801#issuecomment-2844916340 Hi @alamb. I've a WIP commit for this PR. I'm trying to both address @adriangb concerns and needs, and at the same time trying to keep complexity at minimum and trying to make

[PR] add benchmark code for `Reuse rows in row cursor stream` [datafusion]

2025-05-01 Thread via GitHub
acking-you opened a new pull request, #15913: URL: https://github.com/apache/datafusion/pull/15913 ## Which issue does this PR close? - Closes #. ## Rationale for this change You can see:https://github.com/apache/datafusion/issues/15720#issuecomment-284453160

Re: [PR] chore: Remove fast encoding option [datafusion-comet]

2025-05-01 Thread via GitHub
codecov-commenter commented on PR #1703: URL: https://github.com/apache/datafusion-comet/pull/1703#issuecomment-2845566547 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1703?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] Improve sqllogictest error reporting [datafusion]

2025-05-01 Thread via GitHub
berkaysynnada commented on code in PR #15905: URL: https://github.com/apache/datafusion/pull/15905#discussion_r2070694736 ## datafusion/sqllogictest/bin/sqllogictests.rs: ## @@ -235,14 +235,38 @@ async fn run_test_file( runner.with_normalizer(value_normalizer); runner.

Re: [PR] chore: Remove fast encoding option [datafusion-comet]

2025-05-01 Thread via GitHub
andygrove commented on PR #1703: URL: https://github.com/apache/datafusion-comet/pull/1703#issuecomment-2845639468 > I love a good negative line count. Thanks @andygrove! Do we anticipate any performance changes? Based on local testing, TPC-H performance with fast encoding took 276s

Re: [PR] chore: regenerate builtin functions coverage [datafusion-comet]

2025-05-01 Thread via GitHub
comphead commented on PR #1698: URL: https://github.com/apache/datafusion-comet/pull/1698#issuecomment-2845671771 > Interesting we getting regressions for some string functions like > > * initcap > * lower > * upper > * abs > ... > > @mbutrovich ^^ Might be

Re: [PR] feat: metadata columns [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #14057: URL: https://github.com/apache/datafusion/pull/14057#issuecomment-2845795660 Thank you @chenkovsky and @adriangb and everyone else who worked on this PR. I think the idea of additional metadata columns for LIstingTable provider is super valuable but it s

Re: [PR] Support marking columns as system columns via Field's metadata [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #14362: URL: https://github.com/apache/datafusion/pull/14362#issuecomment-2845809583 See discussion on https://github.com/apache/datafusion/issues/15173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
Omega359 commented on code in PR #15793: URL: https://github.com/apache/datafusion/pull/15793#discussion_r2070825920 ## datafusion/sqllogictest/test_files/information_schema.slt: ## @@ -372,6 +381,15 @@ datafusion.explain.physical_plan_only false When set to true, the explain s

Re: [PR] Unparse `UNNEST` projection with the table column alias [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15879: URL: https://github.com/apache/datafusion/pull/15879#issuecomment-2845811697 Thank you @xudong963 and @goldmedal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Unparse `UNNEST` projection with the table column alias [datafusion]

2025-05-01 Thread via GitHub
alamb merged PR #15879: URL: https://github.com/apache/datafusion/pull/15879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Generate the common SQL for the unparsing result of the unnest [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #15233: Generate the common SQL for the unparsing result of the unnest URL: https://github.com/apache/datafusion/issues/15233 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] refactor: replace `unwrap_or` with `unwrap_or_else` for improved lazy… [datafusion]

2025-05-01 Thread via GitHub
xudong963 commented on code in PR #15841: URL: https://github.com/apache/datafusion/pull/15841#discussion_r2070946442 ## benchmarks/src/util/options.rs: ## @@ -72,16 +72,11 @@ impl CommonOpt { /// Modify the existing config appropriately pub fn update_config(&self, mut

[I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
alamb opened a new issue, #15915: URL: https://github.com/apache/datafusion/issues/15915 ### Is your feature request related to a problem or challenge? - Part of https://github.com/apache/datafusion/issues/15914 @shehabgamin added the `datafusion-spark` crate in https://github.

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15793: URL: https://github.com/apache/datafusion/pull/15793#issuecomment-2845786865 Are we happy with this PR? Shall we merge it in? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Factor out Substrait consumers into separate files [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15794: URL: https://github.com/apache/datafusion/pull/15794#issuecomment-2845786048 Thanks again @vbarua and @gabotechs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Factor out Substrait consumers into separate files [datafusion]

2025-05-01 Thread via GitHub
alamb merged PR #15794: URL: https://github.com/apache/datafusion/pull/15794 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] [substrait] refactor consumer.rs [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #13864: [substrait] refactor consumer.rs URL: https://github.com/apache/datafusion/issues/13864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
blaginin commented on PR #15793: URL: https://github.com/apache/datafusion/pull/15793#issuecomment-2845789235 Asked Bruce to re-review, and then I'm ready to merge 🤗 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Support file row index / row id for each file in a `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
acking-you commented on issue #15892: URL: https://github.com/apache/datafusion/issues/15892#issuecomment-2845864345 nice feature👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070868393 ## datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt: ## @@ -81,11 +81,15 @@ EXPLAIN select a from t_pushdown where b > 2 ORDER BY a; logical_

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15168: URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846025123 I have filed an epic to track filling out the datafusion-spark crate: - https://github.com/apache/datafusion/issues/15914 I will file some subtickets for follow on work as we

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
alamb merged PR #15168: URL: https://github.com/apache/datafusion/pull/15168 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

[I] [EPIC] Complete `datafusion-spark` Spark Compatible Functions [datafusion]

2025-05-01 Thread via GitHub
alamb opened a new issue, #15914: URL: https://github.com/apache/datafusion/issues/15914 ### Is your feature request related to a problem or challenge? Many DataFusion users are using DataFusion to execution workloads originally developed for Apache Spark. Examples include - [DataF

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-05-01 Thread via GitHub
alamb commented on PR #15168: URL: https://github.com/apache/datafusion/pull/15168#issuecomment-2846025618 Onward! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [I] [DISCUSSION] Add separate crate to cover spark builtin functions [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #5600: [DISCUSSION] Add separate crate to cover spark builtin functions URL: https://github.com/apache/datafusion/issues/5600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] [DISCUSSION] Add separate crate to cover spark builtin functions [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #5600: URL: https://github.com/apache/datafusion/issues/5600#issuecomment-2846025987 We are tracking follow on work in - https://github.com/apache/datafusion/issues/15914 -- This is an automated message from the Apache Git Service. To respond to the message, p

[PR] Fix CI in main [datafusion]

2025-05-01 Thread via GitHub
blaginin opened a new pull request, #15917: URL: https://github.com/apache/datafusion/pull/15917 ## Which issue does this PR close? Fixes CI after https://github.com/apache/datafusion/pull/15168 ## Rationale for this change ## What changes are included in this PR?

Re: [PR] Add `CREATE TRIGGER` support for SQL Server [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio commented on code in PR #1810: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1810#discussion_r2071038220 ## src/dialect/mssql.rs: ## @@ -215,6 +225,59 @@ impl MsSqlDialect { })) } +/// Parse `CREATE TRIGGER` for [MsSql] +/// +///

Re: [I] Support metadata columns (`location`, `size`, `last_modified`) in `ListingTableProvider` [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #15173: URL: https://github.com/apache/datafusion/issues/15173#issuecomment-2845807026 There is another request here that is related I think (basically "row number within the file"): - https://github.com/apache/datafusion/issues/15892 It seems to me what ha

Re: [I] metadata column support [datafusion]

2025-05-01 Thread via GitHub
alamb commented on issue #13975: URL: https://github.com/apache/datafusion/issues/13975#issuecomment-2845808404 I think this is duplicated by https://github.com/apache/datafusion/issues/15173 so let's continue the discussion there -- This is an automated message from the Apache Git Servi

Re: [I] metadata column support [datafusion]

2025-05-01 Thread via GitHub
alamb closed issue #13975: metadata column support URL: https://github.com/apache/datafusion/issues/13975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
blaginin commented on code in PR #15793: URL: https://github.com/apache/datafusion/pull/15793#discussion_r2070830060 ## datafusion/sqllogictest/test_files/information_schema.slt: ## @@ -372,6 +381,15 @@ datafusion.explain.physical_plan_only false When set to true, the explain s

Re: [PR] Add `FormatOptions` to Config [datafusion]

2025-05-01 Thread via GitHub
Omega359 commented on PR #15793: URL: https://github.com/apache/datafusion/pull/15793#issuecomment-2845818047 Beyond the config try_into looking like it's case sensitive for duration_format I think this is good. There is a breaking change with the rename of FormatOptions to OutputFormat tha

[PR] Add support for `DENY` statements [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
aharpervc opened a new pull request, #1836: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1836 This is another statement for SQL Server: https://learn.microsoft.com/en-us/sql/t-sql/statements/deny-transact-sql, but implemented in a common way. Similar to GRANT & REVOKE, so usi

Re: [PR] Implement Parquet filter pushdown via new filter pushdown APIs [datafusion]

2025-05-01 Thread via GitHub
adriangb commented on code in PR #15769: URL: https://github.com/apache/datafusion/pull/15769#discussion_r2070868393 ## datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt: ## @@ -81,11 +81,15 @@ EXPLAIN select a from t_pushdown where b > 2 ORDER BY a; logical_

Re: [PR] Resolved bug in `parse_function_arg` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
LucaCappelletti94 commented on code in PR #1826: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1826#discussion_r2071126270 ## src/parser/mod.rs: ## @@ -5199,13 +5199,20 @@ impl<'a> Parser<'a> { // parse: [ argname ] argtype let mut name = None;

Re: [PR] Update extending-operators.md [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on PR #15832: URL: https://github.com/apache/datafusion/pull/15832#issuecomment-2846154397 i Think ww need @alamb help now . could you help ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on issue #15915: URL: https://github.com/apache/datafusion/issues/15915#issuecomment-2846193611 hi @alamb . it sounds interesting , i would love to work in this . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [I] [datafusion-spark] Example of using Spark compatible function library [datafusion]

2025-05-01 Thread via GitHub
Adez017 commented on issue #15915: URL: https://github.com/apache/datafusion/issues/15915#issuecomment-2846193703 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Resolved bug in `parse_function_arg` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio commented on code in PR #1826: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1826#discussion_r2071042324 ## src/parser/mod.rs: ## @@ -5199,13 +5199,20 @@ impl<'a> Parser<'a> { // parse: [ argname ] argtype let mut name = None; +l

Re: [PR] Support some of pipe operators [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio merged PR #1759: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] [experiment] Generate event log in Chrome tracing format [datafusion-comet]

2025-05-01 Thread via GitHub
codecov-commenter commented on PR #1706: URL: https://github.com/apache/datafusion-comet/pull/1706#issuecomment-2846219034 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1706?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] Added support for `DROP DOMAIN` [datafusion-sqlparser-rs]

2025-05-01 Thread via GitHub
iffyio merged PR #1828: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1828 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

  1   2   >