Re: [PR] [SPARK-51447][SQL] Add `stringToTime` and `stringToTimeAnsi` [spark]

2025-03-09 Thread via GitHub
MaxGekk commented on PR #50220: URL: https://github.com/apache/spark/pull/50220#issuecomment-2709574221 Merging to master. All GAs passed. Thank you, @yaooqinn @dongjoon-hyun for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [SPARK-51450][CORE] BarrierCoordinator thread not exiting in Spark standalone mode [spark]

2025-03-09 Thread via GitHub
jjayadeep06 commented on PR #50223: URL: https://github.com/apache/spark/pull/50223#issuecomment-2709483996 @beliefer - pls take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[PR] [SPARK-51450][CORE] BarrierCoordinator thread not exiting in Spark standalone mode [spark]

2025-03-09 Thread via GitHub
jjayadeep06 opened a new pull request, #50223: URL: https://github.com/apache/spark/pull/50223 What changes were proposed in this pull request? Cancel the timer class when onStop method is called Why are the changes needed? This change is needed to successfully exit the spark app

Re: [PR] [SPARK-51445][CORE][SQL][SS][CONNECT] Change the never changed `var` to `val` [spark]

2025-03-09 Thread via GitHub
beliefer commented on code in PR #50219: URL: https://github.com/apache/spark/pull/50219#discussion_r1986577519 ## core/src/test/scala/org/apache/spark/rpc/TestRpcEndpoint.scala: ## @@ -26,15 +26,15 @@ class TestRpcEndpoint extends ThreadSafeRpcEndpoint with TripleEquals {

Re: [PR] [SPARK-49479][CORE] Cancel the Timer non-daemon thread on stopping the BarrierCoordinator [spark]

2025-03-09 Thread via GitHub
jjayadeep06 commented on PR #50020: URL: https://github.com/apache/spark/pull/50020#issuecomment-2709458610 > Merged into branch-4.0/master @jjayadeep06 @srowen @jayadeep-jayaraman Thank you! Thank you @srowen / @beliefer for a very thorough review! -- This is an automated message

Re: [PR] [SPARK-51443] Fix singleVariantColumn in DSv2 and readStream. [spark]

2025-03-09 Thread via GitHub
chenhao-db commented on PR #50217: URL: https://github.com/apache/spark/pull/50217#issuecomment-2709453704 @cloud-fan could you help review? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[PR] [SPARK-51449][BUILD] Restore hive-llap-common to compile scope [spark]

2025-03-09 Thread via GitHub
pan3793 opened a new pull request, #50222: URL: https://github.com/apache/spark/pull/50222 ### What changes were proposed in this pull request? Restore `hive-llap-common` from `provided` to `compile` scope, this PR reverts https://github.com/apache/spark/pull/49725 and https:

Re: [PR] [SPARK-51418][SQL] Fix DataSource PARTITON TABLE w/ Hive type incompatible partition columns [spark]

2025-03-09 Thread via GitHub
yaooqinn commented on PR #50182: URL: https://github.com/apache/spark/pull/50182#issuecomment-2709325579 Thank you @dongjoon-hyun The CI has already passed https://github.com/yaooqinn/spark/actions/runs/13718779026, but w/o being properly updated here. Let's wait for the GA f

Re: [PR] [SPARK-51446][SQL] Improve the codecNameMap for the compression codec [spark]

2025-03-09 Thread via GitHub
beliefer commented on PR #50221: URL: https://github.com/apache/spark/pull/50221#issuecomment-2709427591 ping @dongjoon-hyun @yaooqinn @LuciferYang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] [SPARK-51365][SQL][TESTS] Add Envs to control the number of `SHUFFLE_EXCHANGE/RESULT_QUERY_STAGE` threads used in test cases related to `SharedSparkSession/TestHive` [spark]

2025-03-09 Thread via GitHub
LuciferYang commented on PR #50206: URL: https://github.com/apache/spark/pull/50206#issuecomment-2709336288 start https://github.com/apache/spark/actions/runs/13755976255 for double check -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] [SPARK-51365][SQL][TESTS] Add Envs to control the number of `SHUFFLE_EXCHANGE/RESULT_QUERY_STAGE` threads used in test cases related to `SharedSparkSession/TestHive` [spark]

2025-03-09 Thread via GitHub
LuciferYang commented on PR #50206: URL: https://github.com/apache/spark/pull/50206#issuecomment-2709316806 Thanks @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [SPARK-51365][SQL][TESTS] Add Envs to control the number of `SHUFFLE_EXCHANGE/RESULT_QUERY_STAGE` threads used in test cases related to `SharedSparkSession/TestHive` [spark]

2025-03-09 Thread via GitHub
dongjoon-hyun closed pull request #50206: [SPARK-51365][SQL][TESTS] Add Envs to control the number of `SHUFFLE_EXCHANGE/RESULT_QUERY_STAGE` threads used in test cases related to `SharedSparkSession/TestHive` URL: https://github.com/apache/spark/pull/50206 -- This is an automated message from

Re: [PR] [SPARK-51447][SQL] Add `stringToTime` and `stringToTimeAnsi` [spark]

2025-03-09 Thread via GitHub
dongjoon-hyun commented on PR #50220: URL: https://github.com/apache/spark/pull/50220#issuecomment-2709312717 It seems that `SparkSessionE2ESuite` hangs. Could you re-trigger the test pipeline, @MaxGekk ? ``` [info] *** Test still running after 40 minutes, 27 seconds: suite name: Spar

Re: [PR] [SPARK-51445][CORE][SQL][SS][CONNECT] Change the never changed `var` to `val` [spark]

2025-03-09 Thread via GitHub
beliefer commented on code in PR #50219: URL: https://github.com/apache/spark/pull/50219#discussion_r1986496699 ## core/src/test/scala/org/apache/spark/rpc/TestRpcEndpoint.scala: ## @@ -26,15 +26,15 @@ class TestRpcEndpoint extends ThreadSafeRpcEndpoint with TripleEquals {

Re: [PR] Revert "From json/xml should not change collations in the given schema" [spark]

2025-03-09 Thread via GitHub
github-actions[bot] commented on PR #49009: URL: https://github.com/apache/spark/pull/49009#issuecomment-2709159694 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-51445][CORE][SQL][SS][CONNECT] Change the never changed `var` to `val` [spark]

2025-03-09 Thread via GitHub
LuciferYang commented on PR #50219: URL: https://github.com/apache/spark/pull/50219#issuecomment-2709175659 > Are these all instances, @LuciferYang ? Supplemented the cases in the test code path, these are all instances now -- This is an automated message from the Apache Git Service

Re: [PR] [WIP][SPARK-50361][SQL] Fix bugs of using Lateral Column Alias in ORDER BY [spark]

2025-03-09 Thread via GitHub
github-actions[bot] commented on PR #48903: URL: https://github.com/apache/spark/pull/48903#issuecomment-2709159714 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-51365][SQL][TESTS] Add Envs to control the number of `SHUFFLE_EXCHANGE/RESULT_QUERY_STAGE` threads used in test cases related to `SharedSparkSession/TestHive` [spark]

2025-03-09 Thread via GitHub
LuciferYang commented on PR #50206: URL: https://github.com/apache/spark/pull/50206#issuecomment-2709159614 > +1, LGTM. Thank you, @LuciferYang . > > Could you revert the change on `build_and_test.yml` file? done -- This is an automated message from the Apache Git Service. To

Re: [PR] Remove session string calls [spark]

2025-03-09 Thread via GitHub
github-actions[bot] closed pull request #48974: Remove session string calls URL: https://github.com/apache/spark/pull/48974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] [SPARK-50456][PYTHON] Move hash functions to a separate file [spark]

2025-03-09 Thread via GitHub
github-actions[bot] commented on PR #49015: URL: https://github.com/apache/spark/pull/49015#issuecomment-2709159680 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-51444][CORE] Remove the unreachable `if` branch from `TaskSchedulerImpl#statusUpdate` [spark]

2025-03-09 Thread via GitHub
LuciferYang commented on PR #50218: URL: https://github.com/apache/spark/pull/50218#issuecomment-2709157539 Thanks @dongjoon-hyun and @amoghantarkar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-49479][CORE] Cancel the Timer non-daemon thread on stopping the BarrierCoordinator [spark]

2025-03-09 Thread via GitHub
beliefer commented on code in PR #50020: URL: https://github.com/apache/spark/pull/50020#discussion_r1984359791 ## core/src/main/scala/org/apache/spark/BarrierCoordinator.scala: ## @@ -80,8 +81,9 @@ private[spark] class BarrierCoordinator( states.forEachValue(1, clearStat

Re: [PR] [SPARK-51444][CORE] Remove the unreachable `if` branch from `TaskSchedulerImpl#statusUpdate` [spark]

2025-03-09 Thread via GitHub
dongjoon-hyun closed pull request #50218: [SPARK-51444][CORE] Remove the unreachable `if` branch from `TaskSchedulerImpl#statusUpdate` URL: https://github.com/apache/spark/pull/50218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [SPARK-51438][SQL] Make CatalystDataToProtobuf and ProtobufDataToCatalyst properly comparable and hashable [spark]

2025-03-09 Thread via GitHub
vladimirg-db commented on PR #50212: URL: https://github.com/apache/spark/pull/50212#issuecomment-2708980986 Ah, sure, @LuciferYang . This is important for the single-pass Analyzer project, to compare the logical plans between fixed-point and single-pass. Otherwise two objects would be inco

Re: [PR] [SPARK-51442][SQL] Add time formatters [spark]

2025-03-09 Thread via GitHub
MaxGekk closed pull request #50190: [SPARK-51442][SQL] Add time formatters URL: https://github.com/apache/spark/pull/50190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] [SPARK-51442][SQL] Add time formatters [spark]

2025-03-09 Thread via GitHub
MaxGekk commented on PR #50190: URL: https://github.com/apache/spark/pull/50190#issuecomment-2709033436 Merging to master. Thank you, @LuciferYang for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] [SPARK-51402][SQL][TESTS] Test TimeType in UDF [spark]

2025-03-09 Thread via GitHub
calilisantos commented on code in PR #50194: URL: https://github.com/apache/spark/pull/50194#discussion_r1986340130 ## sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala: ## @@ -862,7 +862,7 @@ class UDFSuite extends QueryTest with SharedSparkSession { .select(my

Re: [PR] [SPARK-51438][SQL] Make CatalystDataToProtobuf and ProtobufDataToCatalyst properly comparable and hashable [spark]

2025-03-09 Thread via GitHub
vladimirg-db commented on PR #50212: URL: https://github.com/apache/spark/pull/50212#issuecomment-2708981083 Added this to PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [SPARK-51438][SQL] Make CatalystDataToProtobuf and ProtobufDataToCatalyst properly comparable and hashable [spark]

2025-03-09 Thread via GitHub
LuciferYang commented on PR #50212: URL: https://github.com/apache/spark/pull/50212#issuecomment-2708970351 Could you further point out where in the current Spark code the `hashCode`/`equals` methods of `CatalystDataToProtobuf ` and `ProtobufDataToCatalyst ` are being used? -- This is an

Re: [PR] [SPARK-51447][SQL] Add `stringToTime` and `stringToTimeAnsi` [spark]

2025-03-09 Thread via GitHub
MaxGekk commented on PR #50220: URL: https://github.com/apache/spark/pull/50220#issuecomment-2708966502 @itholic @dongjoon-hyun @yaooqinn Could you review this PR, please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] [SPARK-51442][SQL] Add time formatters [spark]

2025-03-09 Thread via GitHub
LuciferYang commented on code in PR #50190: URL: https://github.com/apache/spark/pull/50190#discussion_r1986371822 ## sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/TimeFormatter.scala: ## @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] [SPARK-51442][SQL] Add time formatters [spark]

2025-03-09 Thread via GitHub
MaxGekk commented on PR #50190: URL: https://github.com/apache/spark/pull/50190#issuecomment-2708956782 @LuciferYang @gengliangwang Could you review this PR, please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout [spark]

2025-03-09 Thread via GitHub
allisonwang-db commented on PR #50198: URL: https://github.com/apache/spark/pull/50198#issuecomment-2705283002 cc @HyukjinKwon @itholic -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [SPARK-51402][SQL][TESTS] Test TimeType in UDF [spark]

2025-03-09 Thread via GitHub
calilisantos commented on code in PR #50194: URL: https://github.com/apache/spark/pull/50194#discussion_r1986340130 ## sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala: ## @@ -862,7 +862,7 @@ class UDFSuite extends QueryTest with SharedSparkSession { .select(my

Re: [PR] [SPARK-51402][SQL][TESTS] Test TimeType in UDF [spark]

2025-03-09 Thread via GitHub
calilisantos commented on code in PR #50194: URL: https://github.com/apache/spark/pull/50194#discussion_r1986339715 ## sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala: ## @@ -862,7 +862,7 @@ class UDFSuite extends QueryTest with SharedSparkSession { .select(my

Re: [PR] [SPARK-51402][SQL][TESTS] Test TimeType in UDF [spark]

2025-03-09 Thread via GitHub
calilisantos commented on code in PR #50194: URL: https://github.com/apache/spark/pull/50194#discussion_r1986339715 ## sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala: ## @@ -862,7 +862,7 @@ class UDFSuite extends QueryTest with SharedSparkSession { .select(my

Re: [PR] [SPARK-51402][SQL][TESTS] Test TimeType in UDF [spark]

2025-03-09 Thread via GitHub
calilisantos commented on code in PR #50194: URL: https://github.com/apache/spark/pull/50194#discussion_r1986339562 ## sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala: ## @@ -1197,6 +1197,35 @@ class UDFSuite extends QueryTest with SharedSparkSession { Row(Ro

Re: [PR] [SPARK-51402][SQL][TESTS] Test TimeType in UDF [spark]

2025-03-09 Thread via GitHub
calilisantos commented on code in PR #50194: URL: https://github.com/apache/spark/pull/50194#discussion_r1986339181 ## dev/create-release/release-build.sh: ## @@ -137,6 +137,12 @@ if [[ "$1" == "finalize" ]]; then --repository-url https://upload.pypi.org/legacy/ \ "pys

[PR] [WIP][SQL] Add `stringToTime` and `stringToTimeAnsi` [spark]

2025-03-09 Thread via GitHub
MaxGekk opened a new pull request, #50220: URL: https://github.com/apache/spark/pull/50220 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? No. ### How was

Re: [PR] [SPARK-51321][SQL] Add rpad and lpad support for PostgresDialect and MsSQLServerDialect expression pushdown [spark]

2025-03-09 Thread via GitHub
cloud-fan commented on PR #50060: URL: https://github.com/apache/spark/pull/50060#issuecomment-2703810297 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] [SPARK-51445][SQL][CONNECT] Change the never changed `var` to `val` [spark]

2025-03-09 Thread via GitHub
LuciferYang opened a new pull request, #50219: URL: https://github.com/apache/spark/pull/50219 ### What changes were proposed in this pull request? This PR replaces unchanged `var` with `val`. ### Why are the changes needed? Use `val` instead of `var` when possible.

Re: [PR] [SPARK-50763][SQL] Add Analyzer rule for resolving SQL table functions [spark]

2025-03-09 Thread via GitHub
allisonwang-db commented on code in PR #49471: URL: https://github.com/apache/spark/pull/49471#discussion_r1984402797 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -2655,6 +2656,93 @@ class Analyzer(override val catalogManager: Cata