Re: [PR] [SPARK-50756][CORE] Use error class for exceptions in SparkConf.validateSettings [spark]

2025-01-15 Thread via GitHub
bozhang2820 commented on PR #49491: URL: https://github.com/apache/spark/pull/49491#issuecomment-2591915309 > Could you re-trigger those failed test pipeline by rebasing to `master` branch, @bozhang2820 ? Thanks @dongjoon-hyun! Rebased to master branch. -- This is an automated mess

Re: [PR] [SPARK-50822][BUILD] Setting version to 4.1.0-SNAPSHOT [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on PR #49495: URL: https://github.com/apache/spark/pull/49495#issuecomment-2591915353 I'll merge it after I cut the branch today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916238144 ## sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala: ## @@ -3097,4 +3097,19 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916244276 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala: ## @@ -61,6 +61,34 @@ private case class OracleDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916227472 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala: ## @@ -986,4 +986,39 @@ private[v2] trait V2JDBCTest extends Sh

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916417583 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala: ## @@ -97,18 +101,52 @@ class SqlScriptingExecution( None } - privat

[PR] [SPARK-46934][SQL][FOLLOWUP] Handle Special Characters for HMS [spark]

2025-01-15 Thread via GitHub
yaooqinn opened a new pull request, #49506: URL: https://github.com/apache/spark/pull/49506 ### What changes were proposed in this pull request? This PR explicitly marks struct type with special characters as a hive-incompatible type to `newSparkSQLSpecificMetastoreTable()` instead of

Re: [PR] [SPARK-50828][PYTHON][ML][CONNECT] Deprecate `pyspark.ml.connect` [spark]

2025-01-15 Thread via GitHub
zhengruifeng commented on PR #49504: URL: https://github.com/apache/spark/pull/49504#issuecomment-2592485442 merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-50828][PYTHON][ML][CONNECT] Deprecate `pyspark.ml.connect` [spark]

2025-01-15 Thread via GitHub
zhengruifeng closed pull request #49504: [SPARK-50828][PYTHON][ML][CONNECT] Deprecate `pyspark.ml.connect` URL: https://github.com/apache/spark/pull/49504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916427834 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala: ## @@ -81,12 +107,79 @@ class SqlScriptingExecutionFrame( scopes.

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
milastdbx commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916424131 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala: ## @@ -986,4 +986,39 @@ private[v2] trait V2JDBCTest extends Share

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916430329 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -965,3 +970,17 @@ class ForStatementExec( body.reset() } } +

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916430493 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala: ## @@ -81,12 +107,79 @@ class SqlScriptingExecutionFrame( scopes.

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916431329 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -190,7 +193,8 @@ class CompoundBodyExec( statements: Seq[Compoun

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916431076 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -965,3 +970,17 @@ class ForStatementExec( body.reset() } } +

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916432559 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala: ## @@ -50,20 +72,24 @@ class SqlScriptingExecutionContext { * @param e

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916430857 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -965,3 +970,17 @@ class ForStatementExec( body.reset() } } +

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916433838 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala: ## @@ -68,7 +71,9 @@ case class SingleStatement(parsedPlan:

Re: [PR] [SPARK-50633][FOLLOWUP] Let `CODECOV_TOKEN` transfer from `build_coverage.yml` to `build_and_test.yml` [spark]

2025-01-15 Thread via GitHub
panbingkun commented on PR #49507: URL: https://github.com/apache/spark/pull/49507#issuecomment-2592536774 cc @dongjoon-hyun @HyukjinKwon @zhengruifeng @LuciferYang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] [SPARK-50633][FOLLOWUP] Let `CODECOV_TOKEN` transfer from `build_coverage.yml` to `build_and_test.yml` [spark]

2025-01-15 Thread via GitHub
panbingkun opened a new pull request, #49507: URL: https://github.com/apache/spark/pull/49507 ### What changes were proposed in this pull request? The pr aims to let `CODECOV_TOKEN` transfer from `build_coverage.yml` to `build_and_test.yml`. ### Why are the changes needed? C

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916442929 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala: ## @@ -185,5 +247,7 @@ case class SqlScriptingInterpreter(session: SparkSessi

Re: [PR] [SPARK-50522][SQL] Support for indeterminate collation [spark]

2025-01-15 Thread via GitHub
cloud-fan closed pull request #49103: [SPARK-50522][SQL] Support for indeterminate collation URL: https://github.com/apache/spark/pull/49103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50822][BUILD] Setting version to 4.1.0-SNAPSHOT [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on PR #49495: URL: https://github.com/apache/spark/pull/49495#issuecomment-2592914417 4.0 branch has been cut, thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] [SPARK-50822][BUILD] Setting version to 4.1.0-SNAPSHOT [spark]

2025-01-15 Thread via GitHub
cloud-fan closed pull request #49495: [SPARK-50822][BUILD] Setting version to 4.1.0-SNAPSHOT URL: https://github.com/apache/spark/pull/49495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1916265920 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala: ## @@ -112,6 +112,21 @@ private case class MySQLDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916269138 ## common/utils/src/main/resources/error/error-conditions.json: ## @@ -1242,6 +1248,12 @@ }, "sqlState" : "4274K" }, + "DUPLICATE_CONDITION_IN_HANDLER_D

Re: [PR] [SPARK-50822][BUILD] Setting version to 4.1.0-SNAPSHOT [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #49495: URL: https://github.com/apache/spark/pull/49495#issuecomment-2591876774 All tests passed. Just waiting for the branch cut. ![Screenshot 2025-01-15 at 00 00 49](https://github.com/user-attachments/assets/6700a064-a701-4e63-85b5-599f4be55010)

Re: [PR] [SPARK-50822][BUILD] Setting version to 4.1.0-SNAPSHOT [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #49495: URL: https://github.com/apache/spark/pull/49495#issuecomment-2591927151 Yep, I replied on your last email. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-50806][SQL] Support InputRDDCodegen interruption on task cancellation [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on code in PR #49501: URL: https://github.com/apache/spark/pull/49501#discussion_r1916107165 ## sql/core/src/test/scala/org/apache/spark/sql/execution/HashAggregateCodegenInterruptionSuite.scala: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Softwar

Re: [PR] [SPARK-50714][SQL][SS] Enable schema evolution for TransformWithState when Avro encoding is used [spark]

2025-01-15 Thread via GitHub
HeartSaVioR commented on code in PR #49277: URL: https://github.com/apache/spark/pull/49277#discussion_r1916138007 ## sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala: ## @@ -374,67 +374,29 @@ object SchemaConverters extends Logging { } } - pri

Re: [PR] [SPARK-50810][BUILD][FOLLOWUP] Fix code change detection for profiler module [spark]

2025-01-15 Thread via GitHub
pan3793 commented on PR #49502: URL: https://github.com/apache/spark/pull/49502#issuecomment-2592026394 @LuciferYang thanks for reviewing and suggestion, I updated the manual test steps in the PR description. -- This is an automated message from the Apache Git Service. To respond to the m

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916456260 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala: ## @@ -63,6 +66,84 @@ case class SqlScriptingInterpreter(session: SparkSessio

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916465173 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala: ## @@ -63,6 +66,84 @@ case class SqlScriptingInterpreter(session: SparkSessio

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916427834 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionContext.scala: ## @@ -81,12 +107,79 @@ class SqlScriptingExecutionFrame( scopes.

Re: [PR] [SPARK-50768][CORE] Introduce TaskContext.createResourceUninterruptibly to avoid stream leak by task interruption [spark]

2025-01-15 Thread via GitHub
Ngone51 commented on code in PR #49413: URL: https://github.com/apache/spark/pull/49413#discussion_r1916499066 ## core/src/main/scala/org/apache/spark/TaskContextImpl.scala: ## @@ -296,4 +304,39 @@ private[spark] class TaskContextImpl( private[spark] override def fetchFailed:

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916515526 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -199,7 +203,8 @@ class CompoundBodyExec( } private var localI

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916514739 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala: ## @@ -97,18 +101,52 @@ class SqlScriptingExecution( None } - privat

Re: [PR] [SPARK-50768][CORE] Introduce TaskContext.createResourceUninterruptibly to avoid stream leak by task interruption [spark]

2025-01-15 Thread via GitHub
Ngone51 commented on code in PR #49413: URL: https://github.com/apache/spark/pull/49413#discussion_r1916531539 ## core/src/main/scala/org/apache/spark/TaskContextImpl.scala: ## @@ -296,4 +304,39 @@ private[spark] class TaskContextImpl( private[spark] override def fetchFailed:

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-15 Thread via GitHub
dusantism-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1916621505 ## sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala: ## @@ -49,6 +49,9 @@ class CatalogManager( // TODO: create a real SY

Re: [PR] [SPARK-50806][SQL] Support InputRDDCodegen interruption on task cancellation [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on PR #49501: URL: https://github.com/apache/spark/pull/49501#issuecomment-2592791859 @Ngone51 the new test failed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916626537 ## sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala: ## @@ -3097,4 +3097,19 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916627994 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala: ## @@ -986,4 +986,39 @@ private[v2] trait V2JDBCTest extends Sh

Re: [PR] [SPARK-50522][SQL] Support for indeterminate collation [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on PR #49103: URL: https://github.com/apache/spark/pull/49103#issuecomment-2592864830 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-50633][FOLLOWUP] Let `CODECOV_TOKEN` transfer from `build_coverage.yml` to `build_and_test.yml` [spark]

2025-01-15 Thread via GitHub
LuciferYang commented on code in PR #49507: URL: https://github.com/apache/spark/pull/49507#discussion_r1916640971 ## .github/workflows/build_and_test.yml: ## @@ -623,7 +628,7 @@ jobs: if: fromJSON(inputs.envs).PYSPARK_CODECOV == 'true' uses: codecov/codecov-action

Re: [PR] [SPARK-48530][SQL] Support for local variables in SQL Scripting [spark]

2025-01-15 Thread via GitHub
vladimirg-db commented on code in PR #49445: URL: https://github.com/apache/spark/pull/49445#discussion_r1916642009 ## sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala: ## @@ -49,6 +49,9 @@ class CatalogManager( // TODO: create a real SY

Re: [PR] [SPARK-50806][SQL] Support InputRDDCodegen interruption on task cancellation [spark]

2025-01-15 Thread via GitHub
Ngone51 commented on code in PR #49501: URL: https://github.com/apache/spark/pull/49501#discussion_r1916125326 ## sql/core/src/test/scala/org/apache/spark/sql/execution/HashAggregateCodegenInterruptionSuite.scala: ## @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] [SPARK-50815][PYTHON][SQL] Fix bug where passing null Variants in createDataFrame causes it to fail and add Variant support in createDataFrame in Spark Connect [spark]

2025-01-15 Thread via GitHub
cloud-fan closed pull request #49487: [SPARK-50815][PYTHON][SQL] Fix bug where passing null Variants in createDataFrame causes it to fail and add Variant support in createDataFrame in Spark Connect URL: https://github.com/apache/spark/pull/49487 -- This is an automated message from the Apach

[PR] [SPARK-50827][ML][CONNECT] Support plugin [spark]

2025-01-15 Thread via GitHub
wbo4958 opened a new pull request, #49503: URL: https://github.com/apache/spark/pull/49503 ### What changes were proposed in this pull request? This PR introduces connect ML with a plugin that could replace operators of spark.ml with third-party implementations. ### Why are the

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916350996 ## sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4: ## @@ -79,6 +81,29 @@ setStatementWithOptionalVarKeyword LEFT_PAREN query R

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916361249 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala: ## @@ -2302,6 +2302,51 @@ class SqlScriptingParserSuite extends Spa

[PR] [SPARK-50829][SQL] Add flag to disable session collation by default [spark]

2025-01-15 Thread via GitHub
stefankandic opened a new pull request, #49505: URL: https://github.com/apache/spark/pull/49505 ### What changes were proposed in this pull request? Putting session level collation under a flag which is disabled by default. ### Why are the changes needed? To not have feat

Re: [PR] [SPARK-50756][CORE] Use error class for exceptions in SparkConf.validateSettings [spark]

2025-01-15 Thread via GitHub
bozhang2820 commented on PR #49491: URL: https://github.com/apache/spark/pull/49491#issuecomment-2592336304 @JiexingLi, @HyukjinKwon, could you review again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916356924 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,97 @@ class AstBuilder extends DataTypeAstBuilder script

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916359070 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala: ## @@ -61,6 +61,34 @@ private case class OracleDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50522][SQL] Support for indeterminate collation [spark]

2025-01-15 Thread via GitHub
stefankandic commented on code in PR #49103: URL: https://github.com/apache/spark/pull/49103#discussion_r1916371217 ## sql/core/src/test/scala/org/apache/spark/sql/collation/IndeterminateCollationTestSuite.scala: ## @@ -0,0 +1,222 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] [SPARK-50815][PYTHON][SQL] Fix bug where passing null Variants in createDataFrame causes it to fail and add Variant support in createDataFrame in Spark Connect [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on PR #49487: URL: https://github.com/apache/spark/pull/49487#issuecomment-2591899921 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-50806][SQL] Support InputRDDCodegen interruption on task cancellation [spark]

2025-01-15 Thread via GitHub
Ngone51 commented on code in PR #49501: URL: https://github.com/apache/spark/pull/49501#discussion_r1916114001 ## sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala: ## @@ -485,8 +487,16 @@ trait InputRDDCodegen extends CodegenSupport { ""

Re: [PR] [SPARK-50813][SQL] Allow only unqualified label names inside SQL Scripts [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49485: URL: https://github.com/apache/spark/pull/49485#discussion_r1916212729 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala: ## @@ -163,6 +163,14 @@ class SqlScriptingLabelContext { bl.multip

Re: [PR] [SPARK-50813][SQL] Allow only unqualified label names inside SQL Scripts [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49485: URL: https://github.com/apache/spark/pull/49485#discussion_r1916213942 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala: ## @@ -2000,6 +2000,21 @@ class SqlScriptingParserSuite extends Spa

Re: [PR] [SPARK-50813][SQL] Allow only unqualified label names inside SQL Scripts [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49485: URL: https://github.com/apache/spark/pull/49485#discussion_r1916213942 ## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala: ## @@ -2000,6 +2000,21 @@ class SqlScriptingParserSuite extends Spa

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916227472 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala: ## @@ -986,4 +986,39 @@ private[v2] trait V2JDBCTest extends Sh

Re: [PR] [SPARK-50792][SQL] Format binary data as a binary literal in JDBC. [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49452: URL: https://github.com/apache/spark/pull/49452#discussion_r1916230916 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala: ## @@ -61,6 +61,34 @@ private case class OracleDialect() extends JdbcDialect with SQLCon

Re: [PR] [SPARK-50827][ML][CONNECT] Support plugin [spark]

2025-01-15 Thread via GitHub
grundprinzip commented on code in PR #49503: URL: https://github.com/apache/spark/pull/49503#discussion_r1916232428 ## sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala: ## @@ -269,8 +293,9 @@ private[ml] object MLUtils { * @return * the M

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916393029 ## sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4: ## @@ -79,6 +81,29 @@ setStatementWithOptionalVarKeyword LEFT_PAREN query R

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916390169 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala: ## @@ -199,7 +203,8 @@ class CompoundBodyExec( } private var localI

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916396388 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala: ## @@ -298,3 +303,53 @@ case class ForStatement( For

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
davidm-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916408495 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala: ## @@ -97,18 +101,52 @@ class SqlScriptingExecution( None } - privat

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
miland-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1916724775 ## sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala: ## @@ -185,5 +247,7 @@ case class SqlScriptingInterpreter(session: SparkSessi

Re: [PR] [SPARK-50833][INFRA] Update `buf` GitHub Action job to compare against branch-4.0 [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun closed pull request #49512: [SPARK-50833][INFRA] Update `buf` GitHub Action job to compare against branch-4.0 URL: https://github.com/apache/spark/pull/49512 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on code in PR #49492: URL: https://github.com/apache/spark/pull/49492#discussion_r1917416229 ## connector/profiler/README.md: ## @@ -39,7 +39,7 @@ For more information on async_profiler see the [Async Profiler Manual](https://k To enable code profiling,

Re: [PR] [SPARK-50819] Refactor Spark profiler module [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #49492: URL: https://github.com/apache/spark/pull/49492#issuecomment-2594089464 Merged to master/4.0 for Apache Spark 4.0.0. Thank you again, @pan3793 and @parthchandra . -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] [SPARK-49886][SS] Query level failure test for rocksDB format V2 [spark]

2025-01-15 Thread via GitHub
brkyvz commented on PR #48460: URL: https://github.com/apache/spark/pull/48460#issuecomment-2594139016 LGTM pending tests. @HeartSaVioR can you help merge this if tests pass? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] [SPARK-50804][SQL] to_protobuf() should not throw MatchError [spark]

2025-01-15 Thread via GitHub
HyukjinKwon opened a new pull request, #49520: URL: https://github.com/apache/spark/pull/49520 ### What changes were proposed in this pull request? Right now, if `to_protobuf()` is called with an argument that is not StructType, ` scala.MatchError` is thrown. We should categorize t

Re: [PR] [SPARK-49907][ML][CONNECT][FOLLOW-UP] Add ml connect related packages into PyPI packages [spark]

2025-01-15 Thread via GitHub
HyukjinKwon commented on PR #49519: URL: https://github.com/apache/spark/pull/49519#issuecomment-2594229208 Merged to master and branch-4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] [SPARK-49907][ML][CONNECT][FOLLOW-UP] Add ml connect related packages into PyPI packages [spark]

2025-01-15 Thread via GitHub
HyukjinKwon closed pull request #49519: [SPARK-49907][ML][CONNECT][FOLLOW-UP] Add ml connect related packages into PyPI packages URL: https://github.com/apache/spark/pull/49519 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-49907][ML][CONNECT][FOLLOW-UP] Add ml connect related packages into PyPI packages [spark]

2025-01-15 Thread via GitHub
zhengruifeng commented on PR #49519: URL: https://github.com/apache/spark/pull/49519#issuecomment-2594231578 Late LGTM, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] [SPARK-50808][CORE] Fix issue in writeAll with mixed types not getting written properly [spark]

2025-01-15 Thread via GitHub
HeartSaVioR commented on code in PR #49479: URL: https://github.com/apache/spark/pull/49479#discussion_r1917554426 ## common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java: ## @@ -176,11 +176,13 @@ public void writeAll(List values) throws Exception { fin

Re: [PR] [SPARK-50808][CORE] Fix issue in writeAll with mixed types not getting written properly [spark]

2025-01-15 Thread via GitHub
HeartSaVioR commented on code in PR #49479: URL: https://github.com/apache/spark/pull/49479#discussion_r1917554808 ## common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java: ## @@ -208,11 +208,13 @@ public void writeAll(List values) throws Exception { fin

Re: [PR] [SPARK-50831][SQL] Enable trimming collation by default [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #49510: URL: https://github.com/apache/spark/pull/49510#issuecomment-2594447923 Gentle ping, @stevomitric . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] [SPARK-50654][SS][TESTS][FOLLOWUP] Fix compile issue of `RocksDBStateStoreCheckpointFormatV2Suite` [spark]

2025-01-15 Thread via GitHub
HeartSaVioR commented on code in PR #49523: URL: https://github.com/apache/spark/pull/49523#discussion_r1917657147 ## sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreCheckpointFormatV2Suite.scala: ## @@ -622,8 +622,8 @@ class RocksDBStateS

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1917621813 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,56 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50795][SQL] Store timestamp as `long` type in `describe` LinkedHashMap [spark]

2025-01-15 Thread via GitHub
gengliangwang commented on code in PR #49513: URL: https://github.com/apache/spark/pull/49513#discussion_r1917698581 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala: ## @@ -80,20 +81,18 @@ trait MetadataMapSupport { .mkString("["

Re: [PR] [SPARK-50795][SQL] Store timestamp as `long` type in `describe` LinkedHashMap [spark]

2025-01-15 Thread via GitHub
gengliangwang commented on code in PR #49513: URL: https://github.com/apache/spark/pull/49513#discussion_r1917698581 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala: ## @@ -80,20 +81,18 @@ trait MetadataMapSupport { .mkString("["

Re: [PR] [SPARK-50795][SQL] Store timestamp as `long` type in `describe` LinkedHashMap [spark]

2025-01-15 Thread via GitHub
gengliangwang commented on code in PR #49513: URL: https://github.com/apache/spark/pull/49513#discussion_r1917708054 ## sql/core/src/main/scala/org/apache/spark/sql/execution/command/DescribeRelationJsonCommand.scala: ## @@ -110,7 +125,16 @@ case class DescribeRelationJsonComman

Re: [PR] [WIP][SPARK-50838][SQL]Add checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with UnionLoop cases. [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on code in PR #49518: URL: https://github.com/apache/spark/pull/49518#discussion_r1917524917 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala: ## @@ -1042,6 +1043,75 @@ trait CheckAnalysis extends PredicateHelper with

Re: [PR] [WIP][SPARK-50838][SQL]Add checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with UnionLoop cases. [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on code in PR #49518: URL: https://github.com/apache/spark/pull/49518#discussion_r1917524917 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala: ## @@ -1042,6 +1043,75 @@ trait CheckAnalysis extends PredicateHelper with

Re: [PR] [SPARK-50082][CORE] Remove some unnecessary Jersey-related warning logs [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #48611: URL: https://github.com/apache/spark/pull/48611#issuecomment-2594235201 Gentle ping, @wayneguow . Could you answer the above questions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-46934][SQL][FOLLOWUP] Handle Special Characters for HMS [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on code in PR #49506: URL: https://github.com/apache/spark/pull/49506#discussion_r1917524225 ## sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/QuotingUtils.scala: ## @@ -45,12 +45,12 @@ object QuotingUtils { private val validIdentPattern =

Re: [PR] [SPARK-50082][CORE] Remove some unnecessary Jersey-related warning logs [spark]

2025-01-15 Thread via GitHub
wayneguow commented on PR #48611: URL: https://github.com/apache/spark/pull/48611#issuecomment-2594243271 > Gentle ping, @wayneguow . Could you answer the above questions? @dongjoon-hyun Sorry, I forgot about this, I will confirm the details and give the final solution and reason in t

Re: [PR] [SPARK-50082][CORE] Remove some unnecessary Jersey-related warning logs [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #48611: URL: https://github.com/apache/spark/pull/48611#issuecomment-2594244455 Thank you so much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] [WIP][SPARK-50838][SQL]Add checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with UnionLoop cases. [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on code in PR #49518: URL: https://github.com/apache/spark/pull/49518#discussion_r1917526609 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala: ## @@ -1042,6 +1043,75 @@ trait CheckAnalysis extends PredicateHelper with

Re: [PR] [WIP][SPARK-50838][SQL]Add checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with UnionLoop cases. [spark]

2025-01-15 Thread via GitHub
cloud-fan commented on code in PR #49518: URL: https://github.com/apache/spark/pull/49518#discussion_r1917528400 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala: ## @@ -1042,6 +1043,75 @@ trait CheckAnalysis extends PredicateHelper with

Re: [PR] [SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting [spark]

2025-01-15 Thread via GitHub
dusantism-db commented on code in PR #49427: URL: https://github.com/apache/spark/pull/49427#discussion_r1917465961 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -159,15 +159,99 @@ class AstBuilder extends DataTypeAstBuilder scr

Re: [PR] [SPARK-50820][SQL] DSv2: Conditional nullification of metadata columns in DML [spark]

2025-01-15 Thread via GitHub
aokolnychyi commented on code in PR #49493: URL: https://github.com/apache/spark/pull/49493#discussion_r1917499324 ## sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/MetadataColumn.java: ## @@ -36,6 +36,45 @@ */ @Evolving public interface MetadataColumn { +

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1917621813 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,56 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1917621813 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,56 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50793][SQL] Fix MySQL cast function for DOUBLE, LONGTEXT, BIGINT and BLOB types [spark]

2025-01-15 Thread via GitHub
sunxiaoguang commented on code in PR #49453: URL: https://github.com/apache/spark/pull/49453#discussion_r1917621813 ## connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/MySQLIntegrationSuite.scala: ## @@ -241,6 +241,56 @@ class MySQLIntegrationSuite

Re: [PR] [SPARK-50654][SS][TESTS][FOLLOWUP] Fix compile issue of `RocksDBStateStoreCheckpointFormatV2Suite` [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun commented on PR #49523: URL: https://github.com/apache/spark/pull/49523#issuecomment-2594417946 I verified manually. **BEFORE** ``` $ build/sbt "sql/testOnly *RocksDBStateStoreCheckpointFormatV2Suite" ... [error] /Users/dongjoon/APACHE/spark-merge/sql/core/sr

Re: [PR] [SPARK-50654][SS][TESTS][FOLLOWUP] Fix compile issue of `RocksDBStateStoreCheckpointFormatV2Suite` [spark]

2025-01-15 Thread via GitHub
HeartSaVioR commented on PR #49523: URL: https://github.com/apache/spark/pull/49523#issuecomment-2594417691 Thanks! I was going to make a PR, but great to see you fixed quickly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [SPARK-50654][SS][TESTS][FOLLOWUP] Fix compile issue of `RocksDBStateStoreCheckpointFormatV2Suite` [spark]

2025-01-15 Thread via GitHub
dongjoon-hyun closed pull request #49523: [SPARK-50654][SS][TESTS][FOLLOWUP] Fix compile issue of `RocksDBStateStoreCheckpointFormatV2Suite` URL: https://github.com/apache/spark/pull/49523 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

  1   2   3   >