Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on PR #49553: URL: https://github.com/apache/spark/pull/49553#issuecomment-2599528794 The python linter passed in the last run merged to master/4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng closed pull request #49553: [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect URL: https://github.com/apache/spark/pull/49553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[PR] [MINOR][DOCS] Fix miss semicolon on list file sql example [spark]

2025-01-17 Thread via GitHub
camilesing opened a new pull request, #49561: URL: https://github.com/apache/spark/pull/49561 ### What changes were proposed in this pull request? fix miss semicolon on list file SQL example ### Why are the changes needed? list file SQL example miss semicolon

Re: [PR] [SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on PR #49417: URL: https://github.com/apache/spark/pull/49417#issuecomment-2599510084 @HeartSaVioR - PTAL when u get a chance, thx ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [DO NOT MERGE][SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on PR #49417: URL: https://github.com/apache/spark/pull/49417#issuecomment-2599507319 > this depends on Spark Connect integration work. Before that, do not merge. We discussed this offline and we are not going to treat connect work as a blocker for 4.0. We have

Re: [PR] [DO NOT MERGE][SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49417: URL: https://github.com/apache/spark/pull/49417#discussion_r1920942256 ## sql/api/src/main/scala/org/apache/spark/sql/streaming/ExpiredTimerInfo.scala: ## @@ -26,7 +26,7 @@ import org.apache.spark.annotation.{Evolving, Experimental}

Re: [PR] [DO NOT MERGE][SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49417: URL: https://github.com/apache/spark/pull/49417#discussion_r1920942256 ## sql/api/src/main/scala/org/apache/spark/sql/streaming/ExpiredTimerInfo.scala: ## @@ -26,7 +26,7 @@ import org.apache.spark.annotation.{Evolving, Experimental}

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on PR #49547: URL: https://github.com/apache/spark/pull/49547#issuecomment-2599477389 Merged to master/4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
zhengruifeng closed pull request #49547: [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet URL: https://github.com/apache/spark/pull/49547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [Only Test][Don't review] Test codecov token fork transfer [spark]

2025-01-17 Thread via GitHub
panbingkun closed pull request #49542: [Only Test][Don't review] Test codecov token fork transfer URL: https://github.com/apache/spark/pull/49542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
navinvishy commented on PR #48446: URL: https://github.com/apache/spark/pull/48446#issuecomment-2599464304 > Thank you for making this PR before. It's more natural to follow the upstream `Kubernetes-client` change. Sure, yes I agree solving upstream is better. -- This is an automat

[PR] [SPARK-50856][SS][PYTHON][CONNECT] Spark Connect Support for TransformWithStateInPandas In Python [spark]

2025-01-17 Thread via GitHub
jingz-db opened a new pull request, #49560: URL: https://github.com/apache/spark/pull/49560 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How wa

Re: [PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on PR #49554: URL: https://github.com/apache/spark/pull/49554#issuecomment-2599420439 let's fix the python test failure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920896222 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on code in PR #49553: URL: https://github.com/apache/spark/pull/49553#discussion_r1920895048 ## python/pyspark/ml/tests/test_regression.py: ## @@ -0,0 +1,181 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on code in PR #49553: URL: https://github.com/apache/spark/pull/49553#discussion_r1920894533 ## mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala: ## @@ -903,7 +903,7 @@ class LinearRegressionSummary private[regression] ( val

Re: [PR] [SPARK-49768][SQL] Provide error conditions for `make_date`/`make_timestamp` errors `_LEGACY_ERROR_TEMP_2000` [spark]

2025-01-17 Thread via GitHub
github-actions[bot] closed pull request #48242: [SPARK-49768][SQL] Provide error conditions for `make_date`/`make_timestamp` errors `_LEGACY_ERROR_TEMP_2000` URL: https://github.com/apache/spark/pull/48242 -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] [SPARK-49162][SQL] Push down date_trunc function [spark]

2025-01-17 Thread via GitHub
github-actions[bot] closed pull request #47666: [SPARK-49162][SQL] Push down date_trunc function URL: https://github.com/apache/spark/pull/47666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-49893] Respect user schema nullability for file data sources when DSV2 Table is used. [spark]

2025-01-17 Thread via GitHub
github-actions[bot] closed pull request #48321: [SPARK-49893] Respect user schema nullability for file data sources when DSV2 Table is used. URL: https://github.com/apache/spark/pull/48321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [SPARK-47336][SQL][CONNECT] Provide to PySpark a functionality to get estimated size of DataFrame in bytes [spark]

2025-01-17 Thread via GitHub
github-actions[bot] commented on PR #46368: URL: https://github.com/apache/spark/pull/46368#issuecomment-2599408190 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-50882][SQL][TESTS] Skip `TPCDSCollationQueryTestSuite.q22-v2.7` test in GitHub Action CI [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #49558: URL: https://github.com/apache/spark/pull/49558#issuecomment-2599407503 Hi, @nikolamand-db , @cloud-fan . Although I know that this is a valuable test suite for Collation feature, can we exclude this long-running query from the CIs? -- This is an

Re: [PR] [SPARK-50882][SQL][TESTS] Skip `TPCDSCollationQueryTestSuite.q22-v2.7` test in GitHub Action CI [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on code in PR #49558: URL: https://github.com/apache/spark/pull/49558#discussion_r1920891391 ## sql/core/src/test/scala/org/apache/spark/sql/TPCDSCollationQueryTestSuite.scala: ## @@ -251,7 +251,10 @@ class TPCDSCollationQueryTestSuite extends QueryTest w

Re: [PR] [SPARK-47573][K8S] Support custom driver log url [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #45728: URL: https://github.com/apache/spark/pull/45728#issuecomment-2599334259 Could you fix the compilation errors? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #48446: URL: https://github.com/apache/spark/pull/48446#issuecomment-2599332272 Thank you for making this PR before. It's more natural to follow the upstream `Kubernetes-client` change. -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #48446: [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client URL: https://github.com/apache/spark/pull/48446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #48446: URL: https://github.com/apache/spark/pull/48446#issuecomment-2599331022 Let me close this PR since SPARK-37687 is resolved via - #49159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920822887 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala: ## @@ -140,6 +143,93 @@ object ParserUtils extends SparkParserUtils { } }

Re: [PR] [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility [spark]

2025-01-17 Thread via GitHub
HeartSaVioR closed pull request #49275: [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility URL: https://github.com/apache/spark/pull/49275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility [spark]

2025-01-17 Thread via GitHub
HeartSaVioR commented on PR #49275: URL: https://github.com/apache/spark/pull/49275#issuecomment-2599265195 Thanks, I'm merging on behalf of @brkyvz . Merging to master/4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] [SPARK-50883] Support altering multiple columns in the same command [spark]

2025-01-17 Thread via GitHub
ctring opened a new pull request, #49559: URL: https://github.com/apache/spark/pull/49559 ### What changes were proposed in this pull request? Current ALTER TABLE ... ALTER COLUMN syntax only allows altering one column at a time. For a large table with many columns, we nee

[PR] [SPARK-50882][SQL][TESTS] Skip `TPCDSCollationQueryTestSuite.q22-v2.7` test in GitHub Action CI [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun opened a new pull request, #49558: URL: https://github.com/apache/spark/pull/49558 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

Re: [PR] [SPARK-50633][INFRA][FOLLOWUP][4.0] Let CODECOV_TOKEN transfer to build_and_test.yml [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #49541: [SPARK-50633][INFRA][FOLLOWUP][4.0] Let CODECOV_TOKEN transfer to build_and_test.yml URL: https://github.com/apache/spark/pull/49541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] [SPARK-50855][SS][CONNECT] Spark Connect Support for TransformWithState [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49488: URL: https://github.com/apache/spark/pull/49488#discussion_r1920659644 ## connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateStreamingSuite.scala: ## @@ -0,0 +1,330 @@ +/* + * Licensed to the

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on code in PR #49553: URL: https://github.com/apache/spark/pull/49553#discussion_r1920659369 ## python/pyspark/ml/tests/test_regression.py: ## @@ -0,0 +1,181 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licens

Re: [PR] [SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0. [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #49552: URL: https://github.com/apache/spark/pull/49552#issuecomment-2599059684 Merged to master/4.0. Thank you, @LuciferYang . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] [SPARK-50855][SS][CONNECT] Spark Connect Support for TransformWithState [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49488: URL: https://github.com/apache/spark/pull/49488#discussion_r1920658871 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala: ## @@ -525,6 +557,66 @@ private class KeyValueGroupedDatasetImpl[

Re: [PR] [SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0. [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #49552: [SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0. URL: https://github.com/apache/spark/pull/49552 -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] [SPARK-50871][BUILD] Upgrade `scala-parallel-collections` to 1.2.0 [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #49548: [SPARK-50871][BUILD] Upgrade `scala-parallel-collections` to 1.2.0 URL: https://github.com/apache/spark/pull/49548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [WIP][SPARK-50838][SQL]Add checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with UnionLoop cases. [spark]

2025-01-17 Thread via GitHub
dtenedor commented on code in PR #49518: URL: https://github.com/apache/spark/pull/49518#discussion_r1920578691 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -3099,6 +3099,29 @@ ], "sqlState" : "42602" }, + "INVALID_RECURSIVE_REFERENCE" : {

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920517706 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala: ## @@ -81,12 +107,79 @@ class SqlScriptingExecutionFrame( scopes.re

Re: [PR] [SPARK-50853][CORE] Close temp shuffle file writable channel [spark]

2025-01-17 Thread via GitHub
ChenMichael commented on PR #49531: URL: https://github.com/apache/spark/pull/49531#issuecomment-2598839045 Yea, I was finding it difficult to write a unit test for this, so the way I tested was by launching spark shell, running some queries that would force shuffles and looking at the numb

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
grundprinzip commented on code in PR #49553: URL: https://github.com/apache/spark/pull/49553#discussion_r1920467658 ## mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala: ## @@ -903,7 +903,7 @@ class LinearRegressionSummary private[regression] ( val

Re: [PR] [SPARK-50739][SQL][FOLLOW] Simplify ResolveRecursiveCTESuite with dsl [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on PR #49557: URL: https://github.com/apache/spark/pull/49557#issuecomment-2598677019 cc @nemanjapetr-db @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[PR] [SPARK-50739][SQL][FOLLOW] Simplify ResolveRecursiveCTESuite with dsl [spark]

2025-01-17 Thread via GitHub
cloud-fan opened a new pull request, #49557: URL: https://github.com/apache/spark/pull/49557 ### What changes were proposed in this pull request? A followup of https://github.com/apache/spark/pull/49351 to simplify the test via dsl. ### Why are the changes needed?

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
mihailoale-db commented on code in PR #49526: URL: https://github.com/apache/spark/pull/49526#discussion_r1920400881 ## sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala: ## @@ -1174,4 +1175,22 @@ class DSV2CharVarcharTestSuite extends CharVarcharTestSuite

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920400108 ## sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4: ## @@ -79,6 +81,29 @@ setStatementWithOptionalVarKeyword LEFT_PAREN query R

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920376339 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,104 @@ class AstBuilder extends DataTypeAstBuilder script

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920376339 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,104 @@ class AstBuilder extends DataTypeAstBuilder script

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920370673 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,104 @@ class AstBuilder extends DataTypeAstBuilder script

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920365670 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -2589,6 +2607,12 @@ }, "sqlState" : "HY000" }, + "INVALID_HANDLER_DECLARATION" : {

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on code in PR #49526: URL: https://github.com/apache/spark/pull/49526#discussion_r1920361079 ## sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala: ## @@ -1174,4 +1175,22 @@ class DSV2CharVarcharTestSuite extends CharVarcharTestSuite

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920355818 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
cloud-fan closed pull request #49351: [SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. URL: https://github.com/apache/spark/pull/49351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on PR #49351: URL: https://github.com/apache/spark/pull/49351#issuecomment-2598604126 thanks, merging to master/4.0! I'll create a followup PR shortly to do some cleanup -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on code in PR #49351: URL: https://github.com/apache/spark/pull/49351#discussion_r1920348348 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -4332,4 +4332,12 @@ private[sql] object QueryCompilationErrors extends

Re: [PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
stevomitric commented on code in PR #49554: URL: https://github.com/apache/spark/pull/49554#discussion_r1920344710 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -460,6 +469,8 @@ static CollationMeta loadCollationMeta(CollationIden

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920343147 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends L

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920338631 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends L

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
nemanjapetr-db commented on code in PR #49351: URL: https://github.com/apache/spark/pull/49351#discussion_r1915799531 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala: ## @@ -539,48 +582,58 @@ case class Union( children.

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920342882 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,99 @@ class AstBuilder extends DataTypeAstBuilder script

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920342235 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" :

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920341964 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" :

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920341608 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" :

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920341122 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -3203,6 +3227,12 @@ }, "sqlState" : "42616" }, + "INVALID_SQLSTATE" : { +"mess

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920340784 ## sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4: ## @@ -79,6 +81,29 @@ setStatementWithOptionalVarKeyword LEFT_PAREN query RIG

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920340725 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -5414,6 +5444,11 @@ "Attach a comment to the namespace ." ] }, +

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920338631 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends L

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
mihailoale-db commented on code in PR #49526: URL: https://github.com/apache/spark/pull/49526#discussion_r1920334041 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4941,6 +4941,37 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920334038 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -5414,6 +5444,11 @@ "Attach a comment to the namespace ." ] }, + "

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r192023 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -3203,6 +3227,12 @@ }, "sqlState" : "42616" }, + "INVALID_SQLSTATE" : { +"messag

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920331651 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -2589,6 +2607,12 @@ }, "sqlState" : "HY000" }, + "INVALID_HANDLER_DECLARATION" : { +

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920330305 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" : {

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920329866 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" : {

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
davidm-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920315665 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends Logg

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
davidm-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920311210 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends Logg

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on PR #49553: URL: https://github.com/apache/spark/pull/49553#issuecomment-2598517345 LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920289915 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920288448 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920288783 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920290353 ## python/pyspark/ml/tests/connect/test_parity_evaluation.py: ## @@ -0,0 +1,49 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributo

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920281642 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala: ## @@ -140,6 +143,93 @@ object ParserUtils extends SparkParserUtils { } }

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on code in PR #49526: URL: https://github.com/apache/spark/pull/49526#discussion_r1920259173 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4941,6 +4941,37 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with Ada

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920216506 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite exte

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920216506 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite exte

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920185334 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920180208 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala: ## @@ -63,6 +67,79 @@ case class SqlScriptingInterpreter(session: SparkSessio

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920172727 ## python/pyspark/ml/wrapper.py: ## @@ -353,7 +353,7 @@ def copy(self: "JP", extra: Optional["ParamMap"] = None) -> "JP": if extra is None: extra

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920169052 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920167547 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920161697 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +112,23 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920161697 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +112,23 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920155719 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite exte

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
mihailoale-db commented on PR #49526: URL: https://github.com/apache/spark/pull/49526#issuecomment-2598303534 CIs finished. @MaxGekk @cloud-fan could we merge it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-50792][SQL][FOLLOWUP] Improve the push down information for binary [spark]

2025-01-17 Thread via GitHub
beliefer commented on PR #49555: URL: https://github.com/apache/spark/pull/49555#issuecomment-2598302725 ping @cloud-fan cc @sunxiaoguang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920147018 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +112,21 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
stefankandic commented on code in PR #49554: URL: https://github.com/apache/spark/pull/49554#discussion_r1920130238 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -460,6 +469,8 @@ static CollationMeta loadCollationMeta(CollationIde

[PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
stevomitric opened a new pull request, #49554: URL: https://github.com/apache/spark/pull/49554 ### What changes were proposed in this pull request? Added all RTRIM collations to TVF when using `Collations` generator. ### Why are the changes needed? As per the collation projec

Re: [PR] [WIP][SPARK-50838][SQL]Add checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with UnionLoop cases. [spark]

2025-01-17 Thread via GitHub
milanisvet commented on code in PR #49518: URL: https://github.com/apache/spark/pull/49518#discussion_r1920073718 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala: ## @@ -1042,6 +1043,75 @@ trait CheckAnalysis extends PredicateHelper with

Re: [PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
stevomitric commented on PR #49554: URL: https://github.com/apache/spark/pull/49554#issuecomment-2598164875 cc @stefankandic and @dejankrak-db to take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on PR #49553: URL: https://github.com/apache/spark/pull/49553#issuecomment-2598136642 @wbo4958 @HyukjinKwon @WeichenXu123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

  1   2   >