Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #48446: URL: https://github.com/apache/spark/pull/48446#issuecomment-2599332272 Thank you for making this PR before. It's more natural to follow the upstream `Kubernetes-client` change. -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #48446: URL: https://github.com/apache/spark/pull/48446#issuecomment-2599331022 Let me close this PR since SPARK-37687 is resolved via - #49159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #48446: [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client URL: https://github.com/apache/spark/pull/48446 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] [SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0. [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #49552: [SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0. URL: https://github.com/apache/spark/pull/49552 -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] [SPARK-50633][INFRA][FOLLOWUP][4.0] Let CODECOV_TOKEN transfer to build_and_test.yml [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #49541: [SPARK-50633][INFRA][FOLLOWUP][4.0] Let CODECOV_TOKEN transfer to build_and_test.yml URL: https://github.com/apache/spark/pull/49541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] [SPARK-50882][SQL][TESTS] Skip `TPCDSCollationQueryTestSuite.q22-v2.7` test in GitHub Action CI [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on code in PR #49558: URL: https://github.com/apache/spark/pull/49558#discussion_r1920891391 ## sql/core/src/test/scala/org/apache/spark/sql/TPCDSCollationQueryTestSuite.scala: ## @@ -251,7 +251,10 @@ class TPCDSCollationQueryTestSuite extends QueryTest w

Re: [PR] [SPARK-50882][SQL][TESTS] Skip `TPCDSCollationQueryTestSuite.q22-v2.7` test in GitHub Action CI [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #49558: URL: https://github.com/apache/spark/pull/49558#issuecomment-2599407503 Hi, @nikolamand-db , @cloud-fan . Although I know that this is a valuable test suite for Collation feature, can we exclude this long-running query from the CIs? -- This is an

Re: [PR] [SPARK-47336][SQL][CONNECT] Provide to PySpark a functionality to get estimated size of DataFrame in bytes [spark]

2025-01-17 Thread via GitHub
github-actions[bot] commented on PR #46368: URL: https://github.com/apache/spark/pull/46368#issuecomment-2599408190 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-49893] Respect user schema nullability for file data sources when DSV2 Table is used. [spark]

2025-01-17 Thread via GitHub
github-actions[bot] closed pull request #48321: [SPARK-49893] Respect user schema nullability for file data sources when DSV2 Table is used. URL: https://github.com/apache/spark/pull/48321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [SPARK-49162][SQL] Push down date_trunc function [spark]

2025-01-17 Thread via GitHub
github-actions[bot] closed pull request #47666: [SPARK-49162][SQL] Push down date_trunc function URL: https://github.com/apache/spark/pull/47666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-49768][SQL] Provide error conditions for `make_date`/`make_timestamp` errors `_LEGACY_ERROR_TEMP_2000` [spark]

2025-01-17 Thread via GitHub
github-actions[bot] closed pull request #48242: [SPARK-49768][SQL] Provide error conditions for `make_date`/`make_timestamp` errors `_LEGACY_ERROR_TEMP_2000` URL: https://github.com/apache/spark/pull/48242 -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on code in PR #49553: URL: https://github.com/apache/spark/pull/49553#discussion_r1920894533 ## mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala: ## @@ -903,7 +903,7 @@ class LinearRegressionSummary private[regression] ( val

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on code in PR #49553: URL: https://github.com/apache/spark/pull/49553#discussion_r1920895048 ## python/pyspark/ml/tests/test_regression.py: ## @@ -0,0 +1,181 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on PR #49547: URL: https://github.com/apache/spark/pull/49547#issuecomment-2599477389 Merged to master/4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
zhengruifeng closed pull request #49547: [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet URL: https://github.com/apache/spark/pull/49547 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[PR] [MINOR][DOCS] Fix miss semicolon on list file sql example [spark]

2025-01-17 Thread via GitHub
camilesing opened a new pull request, #49561: URL: https://github.com/apache/spark/pull/49561 ### What changes were proposed in this pull request? fix miss semicolon on list file SQL example ### Why are the changes needed? list file SQL example miss semicolon

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng closed pull request #49553: [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect URL: https://github.com/apache/spark/pull/49553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on PR #49553: URL: https://github.com/apache/spark/pull/49553#issuecomment-2599528794 The python linter passed in the last run merged to master/4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [DO NOT MERGE][SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49417: URL: https://github.com/apache/spark/pull/49417#discussion_r1920942256 ## sql/api/src/main/scala/org/apache/spark/sql/streaming/ExpiredTimerInfo.scala: ## @@ -26,7 +26,7 @@ import org.apache.spark.annotation.{Evolving, Experimental}

Re: [PR] [DO NOT MERGE][SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on PR #49417: URL: https://github.com/apache/spark/pull/49417#issuecomment-2599507319 > this depends on Spark Connect integration work. Before that, do not merge. We discussed this offline and we are not going to treat connect work as a blocker for 4.0. We have

Re: [PR] [DO NOT MERGE][SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49417: URL: https://github.com/apache/spark/pull/49417#discussion_r1920942256 ## sql/api/src/main/scala/org/apache/spark/sql/streaming/ExpiredTimerInfo.scala: ## @@ -26,7 +26,7 @@ import org.apache.spark.annotation.{Evolving, Experimental}

Re: [PR] [SPARK-50770][SS] Removing package scope for transformWithState operator APIs [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on PR #49417: URL: https://github.com/apache/spark/pull/49417#issuecomment-2599510084 @HeartSaVioR - PTAL when u get a chance, thx ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [Only Test][Don't review] Test codecov token fork transfer [spark]

2025-01-17 Thread via GitHub
panbingkun closed pull request #49542: [Only Test][Don't review] Test codecov token fork transfer URL: https://github.com/apache/spark/pull/49542 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-50855][SS][CONNECT] Spark Connect Support for TransformWithState [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49488: URL: https://github.com/apache/spark/pull/49488#discussion_r1920659644 ## connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateStreamingSuite.scala: ## @@ -0,0 +1,330 @@ +/* + * Licensed to the

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on code in PR #49553: URL: https://github.com/apache/spark/pull/49553#discussion_r1920659369 ## python/pyspark/ml/tests/test_regression.py: ## @@ -0,0 +1,181 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licens

Re: [PR] [SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0. [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #49552: URL: https://github.com/apache/spark/pull/49552#issuecomment-2599059684 Merged to master/4.0. Thank you, @LuciferYang . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] [SPARK-50883] Support altering multiple columns in the same command [spark]

2025-01-17 Thread via GitHub
ctring opened a new pull request, #49559: URL: https://github.com/apache/spark/pull/49559 ### What changes were proposed in this pull request? Current ALTER TABLE ... ALTER COLUMN syntax only allows altering one column at a time. For a large table with many columns, we nee

[PR] [SPARK-50882][SQL][TESTS] Skip `TPCDSCollationQueryTestSuite.q22-v2.7` test in GitHub Action CI [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun opened a new pull request, #49558: URL: https://github.com/apache/spark/pull/49558 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

Re: [PR] [SPARK-50880][SQL] Add a new visitBinaryComparison method to V2ExpressionSQLBuilder [spark]

2025-01-17 Thread via GitHub
beliefer commented on PR #49556: URL: https://github.com/apache/spark/pull/49556#issuecomment-2599589894 ping @cloud-fan cc @sunxiaoguang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920896222 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under

Re: [PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on PR #49554: URL: https://github.com/apache/spark/pull/49554#issuecomment-2599420439 let's fix the python test failure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] [SPARK-50856][SS][PYTHON][CONNECT] Spark Connect Support for TransformWithStateInPandas In Python [spark]

2025-01-17 Thread via GitHub
jingz-db opened a new pull request, #49560: URL: https://github.com/apache/spark/pull/49560 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How wa

Re: [PR] [SPARK-50871][BUILD] Upgrade `scala-parallel-collections` to 1.2.0 [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun closed pull request #49548: [SPARK-50871][BUILD] Upgrade `scala-parallel-collections` to 1.2.0 URL: https://github.com/apache/spark/pull/49548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-50855][SS][CONNECT] Spark Connect Support for TransformWithState [spark]

2025-01-17 Thread via GitHub
anishshri-db commented on code in PR #49488: URL: https://github.com/apache/spark/pull/49488#discussion_r1920658871 ## connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala: ## @@ -525,6 +557,66 @@ private class KeyValueGroupedDatasetImpl[

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920822887 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala: ## @@ -140,6 +143,93 @@ object ParserUtils extends SparkParserUtils { } }

Re: [PR] [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility [spark]

2025-01-17 Thread via GitHub
HeartSaVioR commented on PR #49275: URL: https://github.com/apache/spark/pull/49275#issuecomment-2599265195 Thanks, I'm merging on behalf of @brkyvz . Merging to master/4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility [spark]

2025-01-17 Thread via GitHub
HeartSaVioR closed pull request #49275: [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility URL: https://github.com/apache/spark/pull/49275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-47573][K8S] Support custom driver log url [spark]

2025-01-17 Thread via GitHub
dongjoon-hyun commented on PR #45728: URL: https://github.com/apache/spark/pull/45728#issuecomment-2599334259 Could you fix the compilation errors? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-37687][K8S] Removes the compile time dependency on the OkHttp http client [spark]

2025-01-17 Thread via GitHub
navinvishy commented on PR #48446: URL: https://github.com/apache/spark/pull/48446#issuecomment-2599464304 > Thank you for making this PR before. It's more natural to follow the upstream `Kubernetes-client` change. Sure, yes I agree solving upstream is better. -- This is an automat

Re: [PR] [MINOR][CORE] Fix compilation warnings in `UnsafeInMemorySorter.java` [spark]

2025-01-17 Thread via GitHub
LuciferYang closed pull request #49546: [MINOR][CORE] Fix compilation warnings in `UnsafeInMemorySorter.java` URL: https://github.com/apache/spark/pull/49546 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[PR] [SPARK-50870][CORE] Add the timezone when casting to timestamp in V2ScanRelationPushDown [spark]

2025-01-17 Thread via GitHub
changgyoopark-db opened a new pull request, #49549: URL: https://github.com/apache/spark/pull/49549 ### What changes were proposed in this pull request? Add the timezone information to a cast expression when the destination type requires it. ### Why are the changes

Re: [PR] [SPARK-50870][CORE] Add the timezone when casting to timestamp in V2ScanRelationPushDown [spark]

2025-01-17 Thread via GitHub
changgyoopark-db commented on PR #49549: URL: https://github.com/apache/spark/pull/49549#issuecomment-2597715270 @cloud-fan FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [PR] [MINOR][CORE] Fix compilation warnings in `UnsafeInMemorySorter.java` [spark]

2025-01-17 Thread via GitHub
LuciferYang commented on PR #49546: URL: https://github.com/apache/spark/pull/49546#issuecomment-2597701275 Merged into master for Spark 4.1.0. Thanks @dongjoon-hyun and @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
vladimirg-db commented on code in PR #49351: URL: https://github.com/apache/spark/pull/49351#discussion_r1919754080 ## sql/core/src/main/scala/org/apache/spark/sql/execution/ExplainUtils.scala: ## @@ -297,6 +297,7 @@ object ExplainUtils extends AdaptiveSparkPlanHelper { /**

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
yaooqinn commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1919757247 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,72 @@ class MySQLIntegrationSuite exte

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1919761396 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,72 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50851][ML][CONNECT][PYTHON] Express ML params with `proto.Expression.Literal` [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on PR #49529: URL: https://github.com/apache/spark/pull/49529#issuecomment-2597852853 thanks, merged to master/4.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50851][ML][CONNECT][PYTHON] Express ML params with `proto.Expression.Literal` [spark]

2025-01-17 Thread via GitHub
zhengruifeng closed pull request #49529: [SPARK-50851][ML][CONNECT][PYTHON] Express ML params with `proto.Expression.Literal` URL: https://github.com/apache/spark/pull/49529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng opened a new pull request, #49553: URL: https://github.com/apache/spark/pull/49553 ### What changes were proposed in this pull request? Support LinearRegression on connect ### Why are the changes needed? feature parity for connect ### Does this PR intr

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
zhengruifeng commented on PR #49553: URL: https://github.com/apache/spark/pull/49553#issuecomment-2598136642 @wbo4958 @HyukjinKwon @WeichenXu123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
stefankandic commented on code in PR #49554: URL: https://github.com/apache/spark/pull/49554#discussion_r1920130238 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -460,6 +469,8 @@ static CollationMeta loadCollationMeta(CollationIde

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1919772508 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,58 @@ class MySQLIntegrationSuite

[PR] Remove the explicit dependency on `Guava` from `plugins.sbt` [spark]

2025-01-17 Thread via GitHub
LuciferYang opened a new pull request, #49550: URL: https://github.com/apache/spark/pull/49550 ### What changes were proposed in this pull request? TBD ### Why are the changes needed? TBD ### Does this PR introduce _any_ user-facing change? No ### How was thi

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920147018 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +112,21 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50792][SQL][FOLLOWUP] Improve the push down information for binary [spark]

2025-01-17 Thread via GitHub
beliefer commented on PR #49555: URL: https://github.com/apache/spark/pull/49555#issuecomment-2598302725 ping @cloud-fan cc @sunxiaoguang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
mihailoale-db commented on PR #49526: URL: https://github.com/apache/spark/pull/49526#issuecomment-2598303534 CIs finished. @MaxGekk @cloud-fan could we merge it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920155719 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite exte

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920161697 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +112,23 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920161697 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +112,23 @@ private case class MySQLDialect() extends JdbcDialect with SQLConfHel

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920167547 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920180208 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala: ## @@ -63,6 +67,79 @@ case class SqlScriptingInterpreter(session: SparkSessio

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920185334 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920281642 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala: ## @@ -140,6 +143,93 @@ object ParserUtils extends SparkParserUtils { } }

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920172727 ## python/pyspark/ml/wrapper.py: ## @@ -353,7 +353,7 @@ def copy(self: "JP", extra: Optional["ParamMap"] = None) -> "JP": if extra is None: extra

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920169052 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920216506 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite exte

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
beliefer commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920216506 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite exte

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on code in PR #49526: URL: https://github.com/apache/spark/pull/49526#discussion_r1920259173 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4941,6 +4941,37 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with Ada

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on code in PR #49351: URL: https://github.com/apache/spark/pull/49351#discussion_r1920348348 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -4332,4 +4332,12 @@ private[sql] object QueryCompilationErrors extends

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on PR #49351: URL: https://github.com/apache/spark/pull/49351#issuecomment-2598604126 thanks, merging to master/4.0! I'll create a followup PR shortly to do some cleanup -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] [SPARK-50875][SQL] Add RTRIM collations to TVF [spark]

2025-01-17 Thread via GitHub
stevomitric commented on code in PR #49554: URL: https://github.com/apache/spark/pull/49554#discussion_r1920344710 ## common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java: ## @@ -460,6 +469,8 @@ static CollationMeta loadCollationMeta(CollationIden

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920343147 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends L

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-17 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1920355818 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,67 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
cloud-fan closed pull request #49351: [SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. URL: https://github.com/apache/spark/pull/49351 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
cloud-fan commented on code in PR #49526: URL: https://github.com/apache/spark/pull/49526#discussion_r1920361079 ## sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala: ## @@ -1174,4 +1175,22 @@ class DSV2CharVarcharTestSuite extends CharVarcharTestSuite

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920365670 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -2589,6 +2607,12 @@ }, "sqlState" : "HY000" }, + "INVALID_HANDLER_DECLARATION" : {

Re: [PR] [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on PR #49553: URL: https://github.com/apache/spark/pull/49553#issuecomment-2598517345 LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
davidm-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920311210 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends Logg

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
davidm-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920315665 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends Logg

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920290353 ## python/pyspark/ml/tests/connect/test_parity_evaluation.py: ## @@ -0,0 +1,49 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributo

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920288783 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920288448 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet [spark]

2025-01-17 Thread via GitHub
wbo4958 commented on code in PR #49547: URL: https://github.com/apache/spark/pull/49547#discussion_r1920289915 ## python/pyspark/ml/tests/test_evaluation.py: ## @@ -14,18 +14,368 @@ # See the License for the specific language governing permissions and # limitations under the L

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920329866 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" : {

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920330305 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" : {

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920334038 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -5414,6 +5444,11 @@ "Attach a comment to the namespace ." ] }, + "

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r192023 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -3203,6 +3227,12 @@ }, "sqlState" : "42616" }, + "INVALID_SQLSTATE" : { +"messag

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920331651 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -2589,6 +2607,12 @@ }, "sqlState" : "HY000" }, + "INVALID_HANDLER_DECLARATION" : { +

Re: [PR] [SPARK-50847] [SQL] Deny ApplyCharTypePadding from applying on specific In expressions [spark]

2025-01-17 Thread via GitHub
mihailoale-db commented on code in PR #49526: URL: https://github.com/apache/spark/pull/49526#discussion_r1920334041 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4941,6 +4941,37 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920341122 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -3203,6 +3227,12 @@ }, "sqlState" : "42616" }, + "INVALID_SQLSTATE" : { +"mess

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920341608 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" :

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920338631 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends L

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920340725 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -5414,6 +5444,11 @@ "Attach a comment to the namespace ." ] }, +

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920340784 ## sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4: ## @@ -79,6 +81,29 @@ setStatementWithOptionalVarKeyword LEFT_PAREN query RIG

Re: [PR] [WIP][SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components. [spark]

2025-01-17 Thread via GitHub
nemanjapetr-db commented on code in PR #49351: URL: https://github.com/apache/spark/pull/49351#discussion_r1915799531 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala: ## @@ -539,48 +582,58 @@ case class Union( children.

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920342235 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" :

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920342882 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,99 @@ class AstBuilder extends DataTypeAstBuilder script

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920341964 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1241,6 +1241,24 @@ ], "sqlState" : "42614" }, + "DUPLICATE_CONDITION_IN_SCOPE" :

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-17 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1920338631 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala: ## @@ -251,19 +251,26 @@ trait ColumnResolutionHelper extends L

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-17 Thread via GitHub
srielau commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1920370673 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,104 @@ class AstBuilder extends DataTypeAstBuilder script

  1   2   >