Re: [PR] feat: add read array support [datafusion-comet]

2025-03-18 Thread via GitHub
kazuyukitanimura commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r2001465890 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -61,13 +61,15 @@ object QueryPlanSerde extends Logging with ShimQueryPlanS

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-18 Thread via GitHub
andygrove commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r2002089283 ## native/core/src/execution/planner.rs: ## @@ -3004,4 +3006,130 @@ mod tests { type_info: None, } } + +#[test] +fn test_c

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-18 Thread via GitHub
comphead merged PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@d

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-18 Thread via GitHub
comphead commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2734862629 Thanks everyone -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-18 Thread via GitHub
andygrove commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r2002090347 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -763,7 +766,8 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde wi

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-18 Thread via GitHub
comphead commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r2001739415 ## native/core/Cargo.toml: ## @@ -77,6 +77,7 @@ jni = { version = "0.21", features = ["invocation"] } lazy_static = "1.4" assertables = "7" hex = "0.4.3" +

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-18 Thread via GitHub
comphead commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r2001744891 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -61,13 +61,15 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde wit

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-17 Thread via GitHub
comphead commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2731468304 @andygrove @kazuyukitanimura please have a second look Nested arrays and Iceberg compat support will be added in follow up PR -- This is an automated message from the Apach

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-07 Thread via GitHub
andygrove commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2707696640 I'd quite like to merge https://github.com/apache/datafusion-comet/pull/1479 before this one so that we can be sure that no additional test failures are introduced for `native

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-07 Thread via GitHub
comphead commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2706871828 I think the last merge brought up a new test which fails now on schema mismatch, checking this ``` - array_compact *** FAILED *** (204 milliseconds) org.apache.sp

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-05 Thread via GitHub
comphead commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r1982427638 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -60,14 +60,16 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde wit

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-05 Thread via GitHub
kazuyukitanimura commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r1982384947 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -60,14 +60,16 @@ object QueryPlanSerde extends Logging with ShimQueryPlanS

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-04 Thread via GitHub
comphead commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2698592405 > In `CometExecRule` we check to see if we support the partitioning types for the shuffle but do not check that we support the types of other columns. > > @comphead Do yo

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-04 Thread via GitHub
andygrove commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2698094548 In `CometExecRule` we check to see if we support the partitioning types for the shuffle but do not check that we support the types of other columns. @comphead Do you wan

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-04 Thread via GitHub
andygrove commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2698077082 Some tests are failing due to https://github.com/apache/datafusion-comet/issues/1289 I think the root cause is that we are trying to shuffle with arrays and Comet shuff

Re: [PR] feat: add read array support [datafusion-comet]

2025-03-03 Thread via GitHub
andygrove commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r1977996365 ## spark/src/test/scala/org/apache/comet/exec/CometNativeReaderSuite.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

Re: [PR] feat: add read array support [datafusion-comet]

2025-02-28 Thread via GitHub
codecov-commenter commented on PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#issuecomment-2691785564 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1456?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] feat: add read array support [datafusion-comet]

2025-02-28 Thread via GitHub
comphead commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r1976123048 ## spark/src/test/scala/org/apache/comet/exec/CometNativeReaderSuite.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] feat: add read array support [datafusion-comet]

2025-02-27 Thread via GitHub
andygrove commented on code in PR #1456: URL: https://github.com/apache/datafusion-comet/pull/1456#discussion_r1974062103 ## spark/src/test/scala/org/apache/comet/exec/CometNativeReaderSuite.scala: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[PR] feat: add read array support [datafusion-comet]

2025-02-27 Thread via GitHub
comphead opened a new pull request, #1456: URL: https://github.com/apache/datafusion-comet/pull/1456 ## Which issue does this PR close? Closes #1454 . ## Rationale for this change ## What changes are included in this PR? ## How are these cha