Re: [PR] [SPARK-51157][SQL][3.5] Add missing @VarArgs Scala annotation for Scala function APIs [spark]

2025-02-12 Thread via GitHub
yaooqinn commented on PR #49906: URL: https://github.com/apache/spark/pull/49906#issuecomment-2652991574 cc @dongjoon-hyun, and thanks for reminding -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-51146][INFRA][FOLLOWUP] Use awk to update release scripts [spark]

2025-02-12 Thread via GitHub
cloud-fan closed pull request #49891: [SPARK-51146][INFRA][FOLLOWUP] Use awk to update release scripts URL: https://github.com/apache/spark/pull/49891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
HyukjinKwon commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952479407 ## sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala: ## @@ -21,4 +21,10 @@ private[sql] object ConnectCommon { va

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
HyukjinKwon commented on PR #49880: URL: https://github.com/apache/spark/pull/49880#issuecomment-2653503883 Some tests might fail ... need to go sleep .. but should be able to review this. I addressed most of major comments. -- This is an automated message from the Apache Git Service. To

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
vladimirg-db commented on code in PR #49835: URL: https://github.com/apache/spark/pull/49835#discussion_r1952289253 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -5554,6 +5554,18 @@ object SQLConf { .booleanConf .createWithDefault(tr

[PR] [SPARK-51179][SQL] Refactor SupportsOrderingWithinGroup so that centralized check [spark]

2025-02-12 Thread via GitHub
beliefer opened a new pull request, #49908: URL: https://github.com/apache/spark/pull/49908 ### What changes were proposed in this pull request? This PR propose to refactor SupportsOrderingWithinGroup so that centralized check. ### Why are the changes needed? Currently, the

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952693866 ## sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala: ## @@ -16,9 +16,23 @@ */ package org.apache.spark.sql.connect

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952689474 ## sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala: ## @@ -366,10 +367,14 @@ object SparkConnectService extends Logg

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952698222 ## sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/LocalAuthInterceptor.scala: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952708046 ## sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala: ## @@ -422,7 +422,13 @@ object SparkConnectClient { * port or

Re: [PR] [SPARK-51160][SQL] Refactor literal function resolution [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on PR #49887: URL: https://github.com/apache/spark/pull/49887#issuecomment-2653657593 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-51160][SQL] Refactor literal function resolution [spark]

2025-02-12 Thread via GitHub
cloud-fan closed pull request #49887: [SPARK-51160][SQL] Refactor literal function resolution URL: https://github.com/apache/spark/pull/49887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952726409 ## python/pyspark/sql/connect/session.py: ## @@ -1072,6 +1083,13 @@ def _start_connect_server(master: str, opts: Dict[str, Any]) -> None: conf.set

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952728970 ## sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala: ## @@ -422,7 +422,13 @@ object SparkConnectClient { * port or

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
miland-db commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952801745 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -329,6 +376,32 @@ class AstBuilder extends DataTypeAstBuilder

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1952799496 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -266,22 +275,41 @@ trait ColumnResolutionHelper extends Logg

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
miland-db commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952815202 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala: ## @@ -405,3 +406,34 @@ case class ExceptionHandler(

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
miland-db commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952812919 ## sql/catalyst/src/main/scala/org/apache/spark/sql/exceptions/SqlScriptingRuntimeException.scala: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
miland-db commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952833092 ## sql/catalyst/src/main/scala/org/apache/spark/sql/exceptions/SqlScriptingRuntimeException.scala: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] [SPARK-51067][SQL] Revert session level collation for DML queries and apply object level collation for DDL queries [spark]

2025-02-12 Thread via GitHub
dejankrak-db commented on code in PR #49772: URL: https://github.com/apache/spark/pull/49772#discussion_r1952626714 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala: ## @@ -155,22 +123,22 @@ object ResolveDefaultStringTypes

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952684605 ## sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala: ## @@ -16,9 +16,23 @@ */ package org.apache.spark.sql.connect

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-02-12 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1952747169 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala: ## @@ -73,28 +95,49 @@ class ResolveCatalogs(val catalogManager: Catal

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
hvanhovell commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952747630 ## python/pyspark/sql/connect/session.py: ## @@ -1072,6 +1083,13 @@ def _start_connect_server(master: str, opts: Dict[str, Any]) -> None: conf.set

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-02-12 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1952747979 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala: ## @@ -35,10 +38,29 @@ class ResolveCatalogs(val catalogManager: Catal

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
vladimirg-db commented on code in PR #49835: URL: https://github.com/apache/spark/pull/49835#discussion_r1952753811 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -5554,6 +5554,18 @@ object SQLConf { .booleanConf .createWithDefault(tr

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-02-12 Thread via GitHub
dusantism-db commented on PR #49445: URL: https://github.com/apache/spark/pull/49445#issuecomment-2653856575 @cloud-fan I've made changes according to your comments, forbade use of DROP TEMPORARY VARIABLE in scripts per offline discussion, updated API of VariableManager and updated FOR to u

Re: [PR] [SPARK-47995][INFRA][PYTHON] Refresh testing image for pyarrow 17 [spark]

2025-02-12 Thread via GitHub
aimtsou commented on PR #47965: URL: https://github.com/apache/spark/pull/47965#issuecomment-2653860396 @zhengruifeng and @dongjoon-hyun: Do you think we could upgrade this to pyarrow 18.0.0? The latest MLFlow supports until pyarrow < 19, which is a pity. -- This is an automated messag

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952770947 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala: ## @@ -405,3 +406,34 @@ case class ExceptionHandler(

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952769403 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -329,6 +376,32 @@ class AstBuilder extends DataTypeAstBuilder

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952766998 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -288,6 +288,53 @@ class AstBuilder extends DataTypeAstBuilder Excepti

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952774459 ## sql/catalyst/src/main/scala/org/apache/spark/sql/exceptions/SqlScriptingRuntimeException.scala: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952772285 ## sql/catalyst/src/main/scala/org/apache/spark/sql/exceptions/SqlScriptingRuntimeException.scala: ## @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundat

Re: [PR] [WIP][SPARK-51180][BUILD] Upgrade Arrow to 19.0.0 [spark]

2025-02-12 Thread via GitHub
aimtsou commented on PR #49909: URL: https://github.com/apache/spark/pull/49909#issuecomment-2653887583 For some reason, the workflow gets forbidden (403) when uploading the image. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1952787519 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala: ## @@ -73,28 +109,34 @@ class ResolveCatalogs(val catalogManager: Catalog

Re: [PR] [SPARK-51163][BUILD] Exclude duplicated jars from connect-repl [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49892: URL: https://github.com/apache/spark/pull/49892#issuecomment-2654001989 Thank you for the confirmation, @pan3793 ! Merged to master/4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-51163][BUILD] Exclude duplicated jars from connect-repl [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun closed pull request #49892: [SPARK-51163][BUILD] Exclude duplicated jars from connect-repl URL: https://github.com/apache/spark/pull/49892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-51157][SQL][3.5] Add missing @VarArgs Scala annotation for Scala function APIs [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun closed pull request #49906: [SPARK-51157][SQL][3.5] Add missing @VarArgs Scala annotation for Scala function APIs URL: https://github.com/apache/spark/pull/49906 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Test arrow 18.2.0 [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49904: URL: https://github.com/apache/spark/pull/49904#issuecomment-2654031020 Is this ready, @LuciferYang ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] [SPARK-51157][SQL][3.5] Add missing @VarArgs Scala annotation for Scala function APIs [spark]

2025-02-12 Thread via GitHub
yaooqinn opened a new pull request, #49906: URL: https://github.com/apache/spark/pull/49906 ### What changes were proposed in this pull request? This PR adds missing `@varargs` Scala annotation for Scala function APIs ### Why are the changes needed? To instruct the compil

Re: [PR] [SPARK-51146][INFRA] Use awk to update release scripts [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on PR #49891: URL: https://github.com/apache/spark/pull/49891#issuecomment-2653019099 thanks for the review, merging to master/4.0! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [SPARK-51008][SQL] Add ResultStage for AQE [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on PR #49715: URL: https://github.com/apache/spark/pull/49715#issuecomment-2653107083 thanks, merging to master/4.0! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[PR] [SPARK-42746][SQL][FOLLOWUP] Improve the code for SupportsOrderingWithinGroup and Mode [spark]

2025-02-12 Thread via GitHub
beliefer opened a new pull request, #49907: URL: https://github.com/apache/spark/pull/49907 ### What changes were proposed in this pull request? This PR propose to improve the code for `SupportsOrderingWithinGroup` and Mode ### Why are the changes needed? First, we can simpl

Re: [PR] [SPARK-45891][SQL][FOLLOWUP] Disable `spark.sql.variant.allowReadingShredded` by default [spark]

2025-02-12 Thread via GitHub
pan3793 commented on PR #49874: URL: https://github.com/apache/spark/pull/49874#issuecomment-2653036367 @gene-db @cloud-fan I read the following docs to learn the new added variant feature, and haven't deep dive the code. - https://github.com/apache/spark/blob/master/common/varian

Re: [PR] [SPARK-51008][SQL] Add ResultStage for AQE [spark]

2025-02-12 Thread via GitHub
cloud-fan closed pull request #49715: [SPARK-51008][SQL] Add ResultStage for AQE URL: https://github.com/apache/spark/pull/49715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
Kimahriman commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952894869 ## sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/LocalAuthInterceptor.scala: ## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on code in PR #49835: URL: https://github.com/apache/spark/pull/49835#discussion_r1952917545 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -5554,6 +5554,18 @@ object SQLConf { .booleanConf .createWithDefault(t

Re: [PR] [SPARK-51156][CONNECT] Provide a basic authentication token when running Spark Connect server locally [spark]

2025-02-12 Thread via GitHub
Kimahriman commented on code in PR #49880: URL: https://github.com/apache/spark/pull/49880#discussion_r1952889051 ## sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/config/ConnectCommon.scala: ## @@ -21,4 +21,10 @@ private[sql] object ConnectCommon { val

Re: [PR] [SPARK-51067][SQL] Revert session level collation for DML queries and apply object level collation for DDL queries [spark]

2025-02-12 Thread via GitHub
dejankrak-db commented on code in PR #49772: URL: https://github.com/apache/spark/pull/49772#discussion_r1952897952 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala: ## @@ -18,80 +18,48 @@ package org.apache.spark.sql.cata

Re: [PR] [SPARK-51067][SQL] Revert session level collation for DML queries and apply object level collation for DDL queries [spark]

2025-02-12 Thread via GitHub
dejankrak-db commented on code in PR #49772: URL: https://github.com/apache/spark/pull/49772#discussion_r1952921268 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala: ## @@ -155,22 +123,22 @@ object ResolveDefaultStringTypes

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
vladimirg-db commented on PR #49835: URL: https://github.com/apache/spark/pull/49835#issuecomment-2654149118 @dongjoon-hyun I cannot repro it. Let me rebase to the latest master. ![image](https://github.com/user-attachments/assets/f16913ee-74ac-417d-abac-9dffb5eefa31) -- This is an

Re: [PR] [SPARK-50855][SS][CONNECT] Spark Connect Support for TransformWithState In Scala [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49488: URL: https://github.com/apache/spark/pull/49488#issuecomment-2654154481 Just a question. Is this ready for 4.0.0, @HeartSaVioR ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49835: URL: https://github.com/apache/spark/pull/49835#issuecomment-2654156160 Thank you, @vladimirg-db . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [WIP][SPARK-51180][BUILD] Upgrade Arrow to 19.0.0 [spark]

2025-02-12 Thread via GitHub
aimtsou commented on PR #49909: URL: https://github.com/apache/spark/pull/49909#issuecomment-2654187290 @dongjoon-hyun: I will look the settings for the fork. I will update the PR and the ticket on JIRA, and probably hold off until Arrow 19.0.1 since 19.0.0 is not on central maven, an

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on PR #49835: URL: https://github.com/apache/spark/pull/49835#issuecomment-2654199207 thanks, merging to master/4.0! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
cloud-fan closed pull request #49835: [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE URL: https://github.com/apache/spark/pull/49835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952992999 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -329,6 +376,32 @@ class AstBuilder extends DataTypeAstBuilder

Re: [PR] [SPARK-51067][SQL] Revert session level collation for DML queries and apply object level collation for DDL queries [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49772: URL: https://github.com/apache/spark/pull/49772#discussion_r1952990564 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala: ## @@ -155,22 +123,22 @@ object ResolveDefaultStringTypes ex

Re: [PR] [SPARK-48375][SQL] Add support for SIGNAL statement [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49726: URL: https://github.com/apache/spark/pull/49726#discussion_r1952994703 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala: ## @@ -405,3 +406,34 @@ case class ExceptionHandler(

[PR] [SPARK-51059][ML][CONNECT][DOCS] Document how ALLOWED_ATTRIBUTES works [spark]

2025-02-12 Thread via GitHub
zhengruifeng opened a new pull request, #49918: URL: https://github.com/apache/spark/pull/49918 ### What changes were proposed in this pull request? Document how ALLOWED_ATTRIBUTES works ### Why are the changes needed? to explain how the reflection is checked

[PR] [SPARK-51176][PYTHON][CONNECT] Meet consistency for unexpected errors PySpark Connect <> Classic [spark]

2025-02-12 Thread via GitHub
itholic opened a new pull request, #49926: URL: https://github.com/apache/spark/pull/49926 ### What changes were proposed in this pull request? This PR proposes to add `UnknownException` for Spark Connect Python client to meet consistency for unexpected errors PySpark Connect <> Class

Re: [PR] [SPARK-51197][ML][PYTHON][CONNECT][TESTS] Unit test clean up [spark]

2025-02-12 Thread via GitHub
zhengruifeng closed pull request #49927: [SPARK-51197][ML][PYTHON][CONNECT][TESTS] Unit test clean up URL: https://github.com/apache/spark/pull/49927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] [SPARK-51197][ML][PYTHON][CONNECT][TESTS] Unit test clean up [spark]

2025-02-12 Thread via GitHub
zhengruifeng opened a new pull request, #49927: URL: https://github.com/apache/spark/pull/49927 ### What changes were proposed in this pull request? Unit test clean up ### Why are the changes needed? test code clean up ### Does this PR introduce _any_ user-

Re: [PR] [SPARK-51197][ML][PYTHON][CONNECT][TESTS] Unit test clean up [spark]

2025-02-12 Thread via GitHub
zhengruifeng commented on PR #49927: URL: https://github.com/apache/spark/pull/49927#issuecomment-2655648621 all tests passed. merged to master/4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-51183][SQL] Link to Parquet spec in Variant docs [spark]

2025-02-12 Thread via GitHub
pan3793 commented on PR #49910: URL: https://github.com/apache/spark/pull/49910#issuecomment-2655346077 > point to the (not finalized) Variant spec in Parquet (what this PR is doing) @gene-db I agree with this option. To avoid confusion, the link should point to a git-hash instead of

Re: [PR] [SPARK-51190][ML][PYTHON][CONNECT] Fix TreeEnsembleModel.treeWeights [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49919: URL: https://github.com/apache/spark/pull/49919#issuecomment-2655436397 All tests passed. Merged to master/4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-51189][CORE] Promote `JobFailed` to `DeveloperApi` [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun closed pull request #49920: [SPARK-51189][CORE] Promote `JobFailed` to `DeveloperApi` URL: https://github.com/apache/spark/pull/49920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] [SPARK-51189][CORE] Promote `JobFailed` to `DeveloperApi` [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49920: URL: https://github.com/apache/spark/pull/49920#issuecomment-2655465674 Merged to master/4.0. Thank you again, @pan3793 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] [SPARK-51193][CORE] Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun closed pull request #49923: [SPARK-51193][CORE] Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final URL: https://github.com/apache/spark/pull/49923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] [SPARK-51193][CORE] Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49923: URL: https://github.com/apache/spark/pull/49923#issuecomment-2655656659 Merged to master/4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[PR] [SPARK-51182][SQL] DataFrameWriter should throw dataPathNotSpecifiedError when path is not specified [spark]

2025-02-12 Thread via GitHub
vrozov opened a new pull request, #49928: URL: https://github.com/apache/spark/pull/49928 ### What changes were proposed in this pull request? Change error message from `Expected exactly one path to be specified, but got: .` to `'path' is not specified.` when path is not specified in the

Re: [PR] [SPARK-51182][SQL] DataFrameWriter should throw dataPathNotSpecifiedError when path is not specified [spark]

2025-02-12 Thread via GitHub
vrozov commented on PR #49928: URL: https://github.com/apache/spark/pull/49928#issuecomment-2655681572 @HeartSaVioR Please review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [SPARK-51182][SQL] Throw a `dataPathNotSpecifiedError` instead of `multiplePathsSpecifiedError` when saving a `DataFrame` to a path without specifying the path [spark]

2025-02-12 Thread via GitHub
wayneguow closed pull request #49922: [SPARK-51182][SQL] Throw a `dataPathNotSpecifiedError` instead of `multiplePathsSpecifiedError` when saving a `DataFrame` to a path without specifying the path URL: https://github.com/apache/spark/pull/49922 -- This is an automated message from the Apach

Re: [PR] [SPARK-51194][BUILD] Upgrade `scalafmt` to 3.8.6 [spark]

2025-02-12 Thread via GitHub
wayneguow commented on PR #49924: URL: https://github.com/apache/spark/pull/49924#issuecomment-2655684705 cc @LuciferYang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] [SPARK-42746][SQL][FOLLOWUP] Improve the code for SupportsOrderingWithinGroup and Mode [spark]

2025-02-12 Thread via GitHub
beliefer commented on code in PR #49907: URL: https://github.com/apache/spark/pull/49907#discussion_r1953921153 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala: ## @@ -390,10 +390,6 @@ case class PercentileCont(left: Expressio

Re: [PR] [WIP][SPARK-51180][BUILD] Upgrade Arrow to 19.0.0 [spark]

2025-02-12 Thread via GitHub
zhengruifeng commented on PR #49909: URL: https://github.com/apache/spark/pull/49909#issuecomment-2655704038 cc @LuciferYang who is also working on arrow upgrade -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-51152][SQL] Add richer examples for the get_json_object function [spark]

2025-02-12 Thread via GitHub
fusheng9399 commented on code in PR #49875: URL: https://github.com/apache/spark/pull/49875#discussion_r1952078550 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala: ## @@ -42,6 +42,10 @@ import org.apache.spark.unsafe.types.UTF8Strin

Re: [PR] [SPARK-51172][SS][3.5] Rename to spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan [spark]

2025-02-12 Thread via GitHub
HeartSaVioR commented on PR #49905: URL: https://github.com/apache/spark/pull/49905#issuecomment-2655303035 https://issues.apache.org/jira/browse/SPARK-51187 Here is it. I can't guarantee that this can be done before the first RC. If we want to make graceful fix, this ticket is definitely a

Re: [PR] [SPARK-51189][CORE] Promote `JobFailed` to `DeveloperApi` [spark]

2025-02-12 Thread via GitHub
pan3793 commented on PR #49920: URL: https://github.com/apache/spark/pull/49920#issuecomment-2655372674 cc @dongjoon-hyun @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-51188][BUILD] Upgrade Arrow to 18.2.0 [spark]

2025-02-12 Thread via GitHub
LuciferYang commented on PR #49904: URL: https://github.com/apache/spark/pull/49904#issuecomment-2655375596 > Is this ready, @LuciferYang ? It's ready now @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-51193][CORE] Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49923: URL: https://github.com/apache/spark/pull/49923#issuecomment-2655622140 Thank you, @yaooqinn ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-51195][BUILD][K8S] Upgrade `kubernetes-client` to 7.1.0 [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun closed pull request #49925: [SPARK-51195][BUILD][K8S] Upgrade `kubernetes-client` to 7.1.0 URL: https://github.com/apache/spark/pull/49925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-51195][BUILD][K8S] Upgrade `kubernetes-client` to 7.1.0 [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49925: URL: https://github.com/apache/spark/pull/49925#issuecomment-2655756945 All K8s tests passed. Merged to master for Apache Spark 4.1.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] [SPARK-50856][SS][PYTHON][CONNECT] Spark Connect Support for TransformWithStateInPandas In Python [spark]

2025-02-12 Thread via GitHub
HeartSaVioR commented on code in PR #49560: URL: https://github.com/apache/spark/pull/49560#discussion_r1953971557 ## python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py: ## @@ -270,14 +272,10 @@ def check_results(batch_df, _): # test list state with ttl has

Re: [PR] [SPARK-51182][SQL] DataFrameWriter should throw dataPathNotSpecifiedError when path is not specified [spark]

2025-02-12 Thread via GitHub
HeartSaVioR commented on PR #49928: URL: https://github.com/apache/spark/pull/49928#issuecomment-2655767922 I'll defer the review to SQL area committers, cc. @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-51194][BUILD] Upgrade `scalafmt` to 3.8.6 [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun closed pull request #49924: [SPARK-51194][BUILD] Upgrade `scalafmt` to 3.8.6 URL: https://github.com/apache/spark/pull/49924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[PR] [SPARK-51193][CORE] Upgrade Netty to 4.1.118.Final [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun opened a new pull request, #49923: URL: https://github.com/apache/spark/pull/49923 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [SPARK-51188][BUILD] Upgrade Arrow to 18.2.0 [spark]

2025-02-12 Thread via GitHub
LuciferYang commented on PR #49904: URL: https://github.com/apache/spark/pull/49904#issuecomment-2655483713 Thank you @dongjoon-hyun ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on PR #49835: URL: https://github.com/apache/spark/pull/49835#issuecomment-2655485295 oh sorry somehow I thought the CI passed and merge it... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-51193][CORE] Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun commented on PR #49923: URL: https://github.com/apache/spark/pull/49923#issuecomment-2655487389 Could you review this PR, @LuciferYang ? Technically, there are two CVE patches here. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[PR] [SPARK-51194][BUILD] Upgrade `scalafmt` to 3.8.6 [spark]

2025-02-12 Thread via GitHub
wayneguow opened a new pull request, #49924: URL: https://github.com/apache/spark/pull/49924 ### What changes were proposed in this pull request? This PR aims to upgrade `scalafmt` from 3.8.5 to 3.8.6. ### Why are the changes needed? Update to the latest version,

Re: [PR] [SPARK-51067][SQL] Revert session level collation for DML queries and apply object level collation for DDL queries [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on code in PR #49772: URL: https://github.com/apache/spark/pull/49772#discussion_r195388 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala: ## @@ -155,22 +123,22 @@ object ResolveDefaultStringTypes ex

[PR] [SPARK-51192][CONNECT] Expose `processWithoutResponseObserverForTesting` in `SparkConnectPlanner` [spark]

2025-02-12 Thread via GitHub
vicennial opened a new pull request, #49921: URL: https://github.com/apache/spark/pull/49921 ### What changes were proposed in this pull request? Adds a new `processWithoutResponseObserverForTesting` private-package method scoped to `planner`. This method is called by `Sp

Re: [PR] [SPARK-51190][ML][PYTHON][CONNECT] Fix TreeEnsembleModel.treeWeights [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun closed pull request #49919: [SPARK-51190][ML][PYTHON][CONNECT] Fix TreeEnsembleModel.treeWeights URL: https://github.com/apache/spark/pull/49919 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] [SPARK-51182][SQL] Throw a `dataPathNotSpecifiedError` instead of `multiplePathsSpecifiedError` when saving a `DataFrame` to a path without specifying the path [spark]

2025-02-12 Thread via GitHub
wayneguow opened a new pull request, #49922: URL: https://github.com/apache/spark/pull/49922 ### What changes were proposed in this pull request? This PR aims to throw a `dataPathNotSpecifiedError` instead of `multiplePathsSpecifiedError` when saving a `DataFrame` to a path wi

Re: [PR] [SPARK-51190][ML][PYTHON][CONNECT] Fix TreeEnsembleModel.treeWeights [spark]

2025-02-12 Thread via GitHub
zhengruifeng commented on PR #49919: URL: https://github.com/apache/spark/pull/49919#issuecomment-2655450259 thank you @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-51059][ML][CONNECT][DOCS] Document how ALLOWED_ATTRIBUTES works [spark]

2025-02-12 Thread via GitHub
zhengruifeng commented on PR #49918: URL: https://github.com/apache/spark/pull/49918#issuecomment-2655453745 merged to master/4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] [SPARK-51059][ML][CONNECT][DOCS] Document how ALLOWED_ATTRIBUTES works [spark]

2025-02-12 Thread via GitHub
zhengruifeng closed pull request #49918: [SPARK-51059][ML][CONNECT][DOCS] Document how ALLOWED_ATTRIBUTES works URL: https://github.com/apache/spark/pull/49918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] [SPARK-51146][SQL][FOLLOWUP] Respect system env `SPARK_CONNECT_MODE` in places that access the api mode config [spark]

2025-02-12 Thread via GitHub
cloud-fan opened a new pull request, #49930: URL: https://github.com/apache/spark/pull/49930 ### What changes were proposed in this pull request? This is a followup of the additional Spark Connect distribution work. Some places that access the api mode config do not use the co

[PR] [SPARK-51198][CORE][DOCS] Revise `defaultMinPartitions` function description [spark]

2025-02-12 Thread via GitHub
dongjoon-hyun opened a new pull request, #49929: URL: https://github.com/apache/spark/pull/49929 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

Re: [PR] [SPARK-51146][SQL][FOLLOWUP] Respect system env `SPARK_CONNECT_MODE` in places that access the api mode config [spark]

2025-02-12 Thread via GitHub
cloud-fan commented on PR #49930: URL: https://github.com/apache/spark/pull/49930#issuecomment-2655720404 cc @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

  1   2   3   >