Re: [PR] [SPARK-51781] Update `README.md` and integration test with Apache Spark 4.0.0 RC4 [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun closed pull request #51: [SPARK-51781] Update `README.md` and integration test with Apache Spark 4.0.0 RC4 URL: https://github.com/apache/spark-connect-swift/pull/51 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

Re: [PR] [SPARK-51781] Update `README.md` and integration test with Apache Spark 4.0.0 RC4 [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun commented on PR #51: URL: https://github.com/apache/spark-connect-swift/pull/51#issuecomment-2799043500 Thank you, @viirya . Merged to main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [SPARK-51784] Support `xml` in `DataFrame(Reader/Writer)` [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun commented on PR #53: URL: https://github.com/apache/spark-connect-swift/pull/53#issuecomment-2799043779 Thank you, @viirya . Merged to main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] [SPARK-51784] Support `xml` in `DataFrame(Reader/Writer)` [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun closed pull request #53: [SPARK-51784] Support `xml` in `DataFrame(Reader/Writer)` URL: https://github.com/apache/spark-connect-swift/pull/53 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-51783] Support `ubuntu-24.04-arm` in `build-test` test pipeline [spark-kubernetes-operator]

2025-04-12 Thread via GitHub
dongjoon-hyun commented on PR #172: URL: https://github.com/apache/spark-kubernetes-operator/pull/172#issuecomment-2799044458 Thank you, @viirya . Merged to main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] [SPARK-51783] Support `ubuntu-24.04-arm` in `build-test` test pipeline [spark-kubernetes-operator]

2025-04-12 Thread via GitHub
dongjoon-hyun closed pull request #172: [SPARK-51783] Support `ubuntu-24.04-arm` in `build-test` test pipeline URL: https://github.com/apache/spark-kubernetes-operator/pull/172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] [SPARK-51781] Update `README.md` and integration test with Apache Spark 4.0.0 RC4 [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun opened a new pull request, #51: URL: https://github.com/apache/spark-connect-swift/pull/51 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

[PR] [SPARK-51782] Add `ubuntu-24.04-arm` to `build` and `integration-test-linux` test pipelines [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun opened a new pull request, #52: URL: https://github.com/apache/spark-connect-swift/pull/52 … ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

Re: [PR] [SPARK-51782] Add `ubuntu-24.04-arm` to `build` and `integration-test-linux` test pipelines [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun commented on PR #52: URL: https://github.com/apache/spark-connect-swift/pull/52#issuecomment-2798801326 I reported this issue to the upstream. - https://github.com/swift-actions/setup-swift/issues/705 -- This is an automated message from the Apache Git Service. To respond

[PR] [SPARK-51783] Support `ubuntu-24.04-arm` in `build-test` test pipeline [spark-kubernetes-operator]

2025-04-12 Thread via GitHub
dongjoon-hyun opened a new pull request, #172: URL: https://github.com/apache/spark-kubernetes-operator/pull/172 ### What changes were proposed in this pull request? This PR aims to support `ubuntu-24.04-arm` in `build-test` test pipeline ### Why are the changes needed? `

Re: [PR] [SPARK-51777][SQL][CORE] Register sql.columnar.* classes to KryoSerializer [spark]

2025-04-12 Thread via GitHub
yaooqinn commented on PR #50566: URL: https://github.com/apache/spark/pull/50566#issuecomment-2798812263 Thank you @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[PR] [SPARK-51784] Support `xml` in `DataFrame(Reader/Writer)` [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun opened a new pull request, #53: URL: https://github.com/apache/spark-connect-swift/pull/53 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change?

Re: [PR] [SPARK-51782] Add `ubuntu-24.04-arm` to `build` and `integration-test-linux` test pipelines [spark-connect-swift]

2025-04-12 Thread via GitHub
dongjoon-hyun commented on PR #52: URL: https://github.com/apache/spark-connect-swift/pull/52#issuecomment-2798800367 Currently, this PR is blocked by `setup-swift` GitHub Action. https://github.com/user-attachments/assets/eaf39065-3db7-43fb-bd51-c71e309582d7"; /> -- This is an au

Re: [PR] [SPARK-51752][SQL] Enable rCTE referencing from within a CTE [spark]

2025-04-12 Thread via GitHub
Pajaraja commented on code in PR #50546: URL: https://github.com/apache/spark/pull/50546#discussion_r2040788341 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala: ## @@ -255,6 +273,21 @@ object CTESubstitution extends Rule[LogicalPlan] {

Re: [PR] [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys [spark]

2025-04-12 Thread via GitHub
mridulm commented on code in PR #50033: URL: https://github.com/apache/spark/pull/50033#discussion_r2040749867 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -1554,6 +1554,7 @@ private[spark] class DAGScheduler( case sms: ShuffleMapStage if st

Re: [PR] [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys [spark]

2025-04-12 Thread via GitHub
mridulm commented on code in PR #50033: URL: https://github.com/apache/spark/pull/50033#discussion_r2040762323 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2171,21 +2189,41 @@ private[spark] class DAGScheduler( abortStage(

Re: [PR] [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys [spark]

2025-04-12 Thread via GitHub
mridulm commented on code in PR #50033: URL: https://github.com/apache/spark/pull/50033#discussion_r2040762323 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2171,21 +2189,41 @@ private[spark] class DAGScheduler( abortStage(

Re: [PR] [SPARK-51756][CORE] Computes RowBasedChecksum in ShuffleWriters [spark]

2025-04-12 Thread via GitHub
mridulm commented on code in PR #50230: URL: https://github.com/apache/spark/pull/50230#discussion_r2040114948 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -5724,6 +5724,21 @@ object SQLConf { .booleanConf .createWithDefault(tru

Re: [PR] [SPARK-51752][SQL] Enable rCTE referencing from within a CTE [spark]

2025-04-12 Thread via GitHub
cloud-fan commented on code in PR #50546: URL: https://github.com/apache/spark/pull/50546#discussion_r2040664694 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala: ## @@ -358,7 +385,7 @@ object CTESubstitution extends Rule[LogicalPlan] {

Re: [PR] [SPARK-51752][SQL] Enable rCTE referencing from within a CTE [spark]

2025-04-12 Thread via GitHub
cloud-fan commented on code in PR #50546: URL: https://github.com/apache/spark/pull/50546#discussion_r2040664563 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala: ## @@ -255,6 +273,21 @@ object CTESubstitution extends Rule[LogicalPlan]

Re: [PR] [SPARK-51752][SQL] Enable rCTE referencing from within a CTE [spark]

2025-04-12 Thread via GitHub
cloud-fan commented on code in PR #50546: URL: https://github.com/apache/spark/pull/50546#discussion_r2040663825 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala: ## @@ -220,18 +224,31 @@ object CTESubstitution extends Rule[LogicalPlan]

Re: [PR] [SPARK-51752][SQL] Enable rCTE referencing from within a CTE [spark]

2025-04-12 Thread via GitHub
cloud-fan commented on code in PR #50546: URL: https://github.com/apache/spark/pull/50546#discussion_r2040663762 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala: ## @@ -220,18 +224,31 @@ object CTESubstitution extends Rule[LogicalPlan]

Re: [PR] [SPARK-51770] Set `Content-Type` headers for Prometheus v3 [spark-kubernetes-operator]

2025-04-12 Thread via GitHub
dongjoon-hyun closed pull request #171: [SPARK-51770] Set `Content-Type` headers for Prometheus v3 URL: https://github.com/apache/spark-kubernetes-operator/pull/171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-51770] Set `Content-Type` headers for Prometheus v3 [spark-kubernetes-operator]

2025-04-12 Thread via GitHub
dongjoon-hyun commented on PR #171: URL: https://github.com/apache/spark-kubernetes-operator/pull/171#issuecomment-2798767126 Merged to main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-51777][SQL][CORE] Register sql.columnar.* classes to KryoSerializer [spark]

2025-04-12 Thread via GitHub
dongjoon-hyun closed pull request #50566: [SPARK-51777][SQL][CORE] Register sql.columnar.* classes to KryoSerializer URL: https://github.com/apache/spark/pull/50566 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-51638][CORE] Fix fetching the remote disk stored RDD blocks via the external shuffle service [spark]

2025-04-12 Thread via GitHub
mridulm commented on code in PR #50439: URL: https://github.com/apache/spark/pull/50439#discussion_r2040989059 ## core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala: ## @@ -863,36 +863,36 @@ class BlockManagerMasterEndpoint( blockId: BlockId,

Re: [PR] [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys [spark]

2025-04-12 Thread via GitHub
mridulm commented on PR #50033: URL: https://github.com/apache/spark/pull/50033#issuecomment-2799759556 @attilapiros I am not entirely sure about specific context of your comment :-) If you [mean this](https://github.com/apache/spark/pull/50033#discussion_r2040777376) or [this](http

Re: [PR] [SPARK-51272][CORE]. Fix for the race condition in Scheduler causing failure in retrying all partitions in case of indeterministic shuffle keys [spark]

2025-04-12 Thread via GitHub
attilapiros commented on PR #50033: URL: https://github.com/apache/spark/pull/50033#issuecomment-2799692246 @mridulm IMHO regarding an indeterministic result stage we should abort the stage more aggressively as we cannot re-execute any of its tasks twice as on the executor side repeating th

Re: [PR] [SPARK-51691][CORE][TESTS] SerializationDebugger should swallow exception when try to find the reason of serialization problem [spark]

2025-04-12 Thread via GitHub
mridulm commented on code in PR #50489: URL: https://github.com/apache/spark/pull/50489#discussion_r2041032560 ## core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala: ## @@ -110,8 +112,13 @@ private[spark] object SerializationDebugger extends Logging {

Re: [PR] [SPARK-51660][CORE] Gracefully handle when MDC is not supported [spark]

2025-04-12 Thread via GitHub
mridulm closed pull request #50452: [SPARK-51660][CORE] Gracefully handle when MDC is not supported URL: https://github.com/apache/spark/pull/50452 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] [SPARK-51660][CORE] Gracefully handle when MDC is not supported [spark]

2025-04-12 Thread via GitHub
mridulm commented on PR #50452: URL: https://github.com/apache/spark/pull/50452#issuecomment-2799830954 Merged to master. Thanks for fixing this @robreeves ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a