Re: [PR] [SPARK-49114] Subcategorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
riyaverm-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704618700 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala: ## @@ -2608,6 +2608,30 @@ private[sql] object QueryExecutionErrors extends Q

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704632155 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala: ## @@ -284,6 +284,11 @@ private[sql] class HDFSBacked

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704632339 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala: ## @@ -376,6 +376,11 @@ private[sql] class RocksDBStateS

Re: [PR] [SPARK-48821][SQL] Support Update in DataFrameWriterV2 [spark]

2024-08-05 Thread via GitHub
szehon-ho commented on PR #47233: URL: https://github.com/apache/spark/pull/47233#issuecomment-2269927548 Actually , updated the patch. Figured out I can use multiple case class (like MergeIntoWriter) to have compile time safety and SQL-like syntax, without the messy error checks. Thanks

Re: [PR] [SPARK-44167][SQL] Add Catalog APIs for loading stored procedures [spark]

2024-08-05 Thread via GitHub
aokolnychyi commented on PR #47190: URL: https://github.com/apache/spark/pull/47190#issuecomment-2269928133 I've updated this PR to address what was previously discussed. A reference implementation can be found in PR #47183. It is ready for another round, @cloud-fan @allisonport-db @H

[PR] [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun opened a new pull request, #47617: URL: https://github.com/apache/spark/pull/47617 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was t

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704639795 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala: ## @@ -527,7 +546,7 @@ private[sql] class RocksDBStateSt

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
ericm-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704640676 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala: ## @@ -527,7 +546,7 @@ private[sql] class RocksDBStateStoreP

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on PR #47524: URL: https://github.com/apache/spark/pull/47524#issuecomment-2269934300 @ericm-db - could you update the PR description and describe all the changes this PR is doing ? -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
ericm-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704650195 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala: ## @@ -166,21 +170,36 @@ class StateSchemaCompatibilit

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
ericm-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704650420 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala: ## @@ -166,21 +170,36 @@ class StateSchemaCompatibilit

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
ericm-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704653746 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala: ## @@ -119,6 +119,8 @@ trait ReadStateStore { */ trait StateStore extend

Re: [PR] [SPARK-49045] Add docker image build for operator [spark-kubernetes-operator]

2024-08-05 Thread via GitHub
jiangzho commented on code in PR #28: URL: https://github.com/apache/spark-kubernetes-operator/pull/28#discussion_r1704659495 ## build-tools/docker/Dockerfile: ## @@ -0,0 +1,29 @@ +FROM gradle:8.9.0-jdk17-alpine AS builder +ARG APP_VERSION +WORKDIR /app +COPY . . +RUN ./gradlew

[PR] [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun opened a new pull request, #47618: URL: https://github.com/apache/spark/pull/47618 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704663415 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala: ## @@ -328,12 +335,19 @@ private[sql] class RocksDBState

[PR] [MINOR][INFRA] Add `bin/docker-image-tool.sh` to KUBERNETES label [spark]

2024-08-05 Thread via GitHub
williamhyun opened a new pull request, #47619: URL: https://github.com/apache/spark/pull/47619 ### What changes were proposed in this pull request? This PR aims to add `bin/docker-image-tool.sh` to KUBERNETES label rule. ### Why are the changes needed? To label PRs better.

Re: [PR] [SPARK-48592][INFRA] Add scala style check for logging message inline variables [spark]

2024-08-05 Thread via GitHub
gengliangwang commented on PR #46947: URL: https://github.com/apache/spark/pull/46947#issuecomment-2270003002 I am closing this one since we already merged https://github.com/apache/spark/pull/47239 -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] [SPARK-48592][INFRA] Add scala style check for logging message inline variables [spark]

2024-08-05 Thread via GitHub
gengliangwang closed pull request #46947: [SPARK-48592][INFRA] Add scala style check for logging message inline variables URL: https://github.com/apache/spark/pull/46947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [SPARK-49097][INFRA] Add Python3 environment detection for the `build_error_docs` method in `build_api_decs.rb` [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun closed pull request #47590: [SPARK-49097][INFRA] Add Python3 environment detection for the `build_error_docs` method in `build_api_decs.rb` URL: https://github.com/apache/spark/pull/47590 -- This is an automated message from the Apache Git Service. To respond to the message, ple

[PR] [SPARK-48778][SQL][TESTS] Improve collation support testing - add unit tests for string expressions [spark]

2024-08-05 Thread via GitHub
uros-db opened a new pull request, #47620: URL: https://github.com/apache/spark/pull/47620 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

Re: [PR] [SPARK-49014][BUILD] Bump Apache Avro to 1.12.0 [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun closed pull request #47498: [SPARK-49014][BUILD] Bump Apache Avro to 1.12.0 URL: https://github.com/apache/spark/pull/47498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-49014][BUILD] Bump Apache Avro to 1.12.0 [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47498: URL: https://github.com/apache/spark/pull/47498#issuecomment-2270019768 Merged to master for Apache Spark 4.0.0-preview2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] [SPARK-49014][BUILD] Bump Apache Avro to 1.12.0 [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47498: URL: https://github.com/apache/spark/pull/47498#issuecomment-2270021527 Thank you again, @Fokko . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47618: URL: https://github.com/apache/spark/pull/47618#issuecomment-2270039726 Could you review this when you have some time, @viirya ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704694329 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala: ## @@ -2608,6 +2608,33 @@ private[sql] object QueryExecutionErrors extends

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704694606 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ## @@ -200,8 +200,7 @@ class RocksDBFileManager( case 2

Re: [PR] [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47618: URL: https://github.com/apache/spark/pull/47618#issuecomment-2270058111 Thank you, @viirya ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
chaoqin-li1123 commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704703998 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ## @@ -200,8 +200,7 @@ class RocksDBFileManager( case

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704706025 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ## @@ -200,8 +200,7 @@ class RocksDBFileManager( case 2

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704707518 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ## @@ -200,8 +200,7 @@ class RocksDBFileManager( case 2

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
riyaverm-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704707814 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ## @@ -200,8 +200,7 @@ class RocksDBFileManager( case 2 =

Re: [PR] [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements [spark]

2024-08-05 Thread via GitHub
costas-db commented on code in PR #47428: URL: https://github.com/apache/spark/pull/47428#discussion_r1704708106 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -977,6 +977,17 @@ object SQLConf { .booleanConf .createWithDefault(tru

Re: [PR] [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on PR #47528: URL: https://github.com/apache/spark/pull/47528#issuecomment-2270068851 Don't think test failure is related here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47617: URL: https://github.com/apache/spark/pull/47617#issuecomment-2270100191 All tests passed. ![Screenshot 2024-08-05 at 16 39 29](https://github.com/user-attachments/assets/2e203562-1e87-4638-882f-15c9decd7654) -- This is an automated message

[PR] [SPARK-48937][SQL] Add collation support for StringToMap string expressions [spark]

2024-08-05 Thread via GitHub
uros-db opened a new pull request, #47621: URL: https://github.com/apache/spark/pull/47621 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

[PR] [WIP][SQL] Update StringType interface for collation support [spark]

2024-08-05 Thread via GitHub
uros-db opened a new pull request, #47622: URL: https://github.com/apache/spark/pull/47622 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

[PR] [SPARK-47359][SQL][FOLLOWUP] Remove unused genCode for StringTranslate [spark]

2024-08-05 Thread via GitHub
uros-db opened a new pull request, #47623: URL: https://github.com/apache/spark/pull/47623 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

Re: [PR] [Don't review, only for test][SPARK-48022][BUILD] Upgrade `jersey` to `3.1.6` [spark]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #46252: URL: https://github.com/apache/spark/pull/46252#issuecomment-2270138903 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-47488][k8s]fix driver pod stuck when driver on k8s [spark]

2024-08-05 Thread via GitHub
github-actions[bot] commented on PR #45667: URL: https://github.com/apache/spark/pull/45667#issuecomment-2270138933 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704750854 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala: ## @@ -242,7 +242,26 @@ class IncrementalExecution( me

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704752567 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImplBase.scala: ## @@ -39,7 +39,7 @@ abstract class StatefulProcessorHan

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704752839 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala: ## @@ -703,6 +704,30 @@ object StreamExecution { "py4j.protocol.Py4

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704753115 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala: ## @@ -703,6 +704,30 @@ object StreamExecution { "py4j.protocol.Py4

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704753400 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala: ## @@ -577,9 +582,12 @@ case class TransformWithStateExec( */

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704753908 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala: ## @@ -653,12 +665,32 @@ object TransformWithStateExec { i

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704755686 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala: ## @@ -166,21 +170,37 @@ class StateSchemaCompatib

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704755946 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala: ## @@ -166,21 +170,37 @@ class StateSchemaCompatib

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704758235 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala: ## @@ -703,6 +704,30 @@ object StreamExecution { "py4j.protocol.Py4

Re: [PR] [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements [spark]

2024-08-05 Thread via GitHub
costas-db commented on code in PR #47428: URL: https://github.com/apache/spark/pull/47428#discussion_r1704762210 ## sql/core/src/test/scala/org/apache/spark/sql/ParsingImprovementsSuite.scala: ## @@ -0,0 +1,183 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
ericm-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704769194 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala: ## @@ -242,7 +242,26 @@ class IncrementalExecution( metada

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
ericm-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704769471 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StatefulProcessorHandleImplBase.scala: ## @@ -39,7 +39,7 @@ abstract class StatefulProcessorHandleI

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
riyaverm-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704769665 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ## @@ -200,8 +200,7 @@ class RocksDBFileManager( case 2 =

Re: [PR] [SPARK-49112][CONNECT][TEST] Make `createLocalRelationProto` support `TimestampType` [spark]

2024-08-05 Thread via GitHub
HyukjinKwon commented on PR #47608: URL: https://github.com/apache/spark/pull/47608#issuecomment-2270168627 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-49112][CONNECT][TEST] Make `createLocalRelationProto` support `TimestampType` [spark]

2024-08-05 Thread via GitHub
HyukjinKwon closed pull request #47608: [SPARK-49112][CONNECT][TEST] Make `createLocalRelationProto` support `TimestampType` URL: https://github.com/apache/spark/pull/47608 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47616: URL: https://github.com/apache/spark/pull/47616#discussion_r1704770668 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ## @@ -200,8 +200,7 @@ class RocksDBFileManager( case 2

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
riyaverm-db commented on PR #47616: URL: https://github.com/apache/spark/pull/47616#issuecomment-2270169768 @HeartSaVioR requesting final review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704774783 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala: ## @@ -242,7 +242,26 @@ class IncrementalExecution( me

Re: [PR] [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families [spark]

2024-08-05 Thread via GitHub
ericm-db commented on code in PR #47524: URL: https://github.com/apache/spark/pull/47524#discussion_r1704783685 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala: ## @@ -242,7 +242,26 @@ class IncrementalExecution( metada

Re: [PR] [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer` [spark]

2024-08-05 Thread via GitHub
wayneguow commented on code in PR #47582: URL: https://github.com/apache/spark/pull/47582#discussion_r1704798016 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -194,6 +200,9 @@ private[sql] class AvroDeserializer( case (FLOAT, Flo

Re: [PR] [SPARK-47359][SQL][FOLLOWUP] Remove unused genCode for StringTranslate [spark]

2024-08-05 Thread via GitHub
HyukjinKwon commented on PR #47623: URL: https://github.com/apache/spark/pull/47623#issuecomment-2270203263 Can you fill the PR description? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] [SPARK-48937][SQL] Add collation support for StringToMap string expressions [spark]

2024-08-05 Thread via GitHub
HyukjinKwon commented on PR #47621: URL: https://github.com/apache/spark/pull/47621#issuecomment-2270203639 Mind filling the PR description? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date [spark]

2024-08-05 Thread via GitHub
HyukjinKwon closed pull request #47618: [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date URL: https://github.com/apache/spark/pull/47618 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date [spark]

2024-08-05 Thread via GitHub
HyukjinKwon commented on PR #47618: URL: https://github.com/apache/spark/pull/47618#issuecomment-2270203965 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [MINOR][SQL] Remove orphans in ProtoToParsedPlanTestSuite and PlanGenerationTestSuite [spark]

2024-08-05 Thread via GitHub
HyukjinKwon commented on PR #47603: URL: https://github.com/apache/spark/pull/47603#issuecomment-2270204609 alright, let me revert this back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite [spark]

2024-08-05 Thread via GitHub
HyukjinKwon commented on PR #47605: URL: https://github.com/apache/spark/pull/47605#issuecomment-2270204990 Oops, my bad. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite [spark]

2024-08-05 Thread via GitHub
HyukjinKwon closed pull request #47605: [SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite URL: https://github.com/apache/spark/pull/47605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[PR] [WIP][SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite [spark]

2024-08-05 Thread via GitHub
HyukjinKwon opened a new pull request, #47624: URL: https://github.com/apache/spark/pull/47624 ### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/47579 that updates the Spark Connect location in the test `PlanGenerationTe

Re: [PR] [SPARK-49083][CONNECT] Allow from_xml and from_json to natively work with json schemas [spark]

2024-08-05 Thread via GitHub
HyukjinKwon closed pull request #47573: [SPARK-49083][CONNECT] Allow from_xml and from_json to natively work with json schemas URL: https://github.com/apache/spark/pull/47573 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] [SPARK-49083][CONNECT] Allow from_xml and from_json to natively work with json schemas [spark]

2024-08-05 Thread via GitHub
HyukjinKwon commented on PR #47573: URL: https://github.com/apache/spark/pull/47573#issuecomment-2270213907 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
cloud-fan commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704808193 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala: ## @@ -323,6 +323,14 @@ class SessionCatalog( externalCatalog.database

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
cloud-fan commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704810269 ## sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala: ## @@ -107,16 +107,19 @@ class CatalogManager( } def setCurrentNam

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
cloud-fan commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704810892 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4876,6 +4878,13 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with Ada

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
cloud-fan commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704811240 ## sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala: ## @@ -4876,6 +4878,13 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with Ada

Re: [PR] [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id [spark]

2024-08-05 Thread via GitHub
HeartSaVioR commented on code in PR #47528: URL: https://github.com/apache/spark/pull/47528#discussion_r1704813193 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala: ## @@ -439,8 +447,9 @@ case class TransformWithStateExec( n

Re: [PR] [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id [spark]

2024-08-05 Thread via GitHub
HeartSaVioR commented on code in PR #47528: URL: https://github.com/apache/spark/pull/47528#discussion_r1704813542 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/OperatorStateMetadata.scala: ## @@ -291,30 +312,54 @@ class OperatorStateMetadataV2Writer(

Re: [PR] [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id [spark]

2024-08-05 Thread via GitHub
HeartSaVioR commented on code in PR #47528: URL: https://github.com/apache/spark/pull/47528#discussion_r1704814814 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/metadata/StateMetadataSource.scala: ## @@ -207,12 +217,9 @@ class StateMetadataPartit

Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]

2024-08-05 Thread via GitHub
pan3793 commented on PR #47500: URL: https://github.com/apache/spark/pull/47500#issuecomment-2270235943 kindly ping @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
amaliujia commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704820575 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala: ## @@ -323,6 +323,14 @@ class SessionCatalog( externalCatalog.database

Re: [PR] [MINOR][INFRA] Add `bin/docker-image-tool.sh` to KUBERNETES label [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun closed pull request #47619: [MINOR][INFRA] Add `bin/docker-image-tool.sh` to KUBERNETES label URL: https://github.com/apache/spark/pull/47619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [MINOR][INFRA] Add `bin/docker-image-tool.sh` to KUBERNETES label [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47619: URL: https://github.com/apache/spark/pull/47619#issuecomment-2270248782 Merged to mastser. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47618: URL: https://github.com/apache/spark/pull/47618#issuecomment-2270249406 Thank you, @HyukjinKwon . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47617: URL: https://github.com/apache/spark/pull/47617#issuecomment-2270250303 Could you review this PR, @yaooqinn ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47617: URL: https://github.com/apache/spark/pull/47617#issuecomment-2270254572 Thank you, @yaooqinn ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun closed pull request #47617: [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles URL: https://github.com/apache/spark/pull/47617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles [spark]

2024-08-05 Thread via GitHub
dongjoon-hyun commented on PR #47617: URL: https://github.com/apache/spark/pull/47617#issuecomment-2270255251 Merged to master for Apache Spark 4.0.0-preview2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

Re: [PR] [SPARK-47430][SQL] Rework group by map type to fix bind reference exception [spark]

2024-08-05 Thread via GitHub
ulysses-you commented on code in PR #47545: URL: https://github.com/apache/spark/pull/47545#discussion_r1704830398 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala: ## @@ -246,8 +246,6 @@ abstract class Optimizer(catalogManager: CatalogManag

Re: [PR] [SPARK-47430][SQL] Rework group by map type to fix bind reference exception [spark]

2024-08-05 Thread via GitHub
ulysses-you commented on code in PR #47545: URL: https://github.com/apache/spark/pull/47545#discussion_r1704830398 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala: ## @@ -246,8 +246,6 @@ abstract class Optimizer(catalogManager: CatalogManag

Re: [PR] [SPARK-48939][AVRO] Support reading Avro with recursive schema reference [spark]

2024-08-05 Thread via GitHub
HeartSaVioR commented on PR #47425: URL: https://github.com/apache/spark/pull/47425#issuecomment-2270296789 @gengliangwang Would you mind helping reviewing the change as you've been one of the main reviewers for Avro? I can give a try, but I don't feel like I'm qualified to review and sign-

Re: [PR] [SPARK-48821][SQL] Support Update in DataFrameWriterV2 [spark]

2024-08-05 Thread via GitHub
szehon-ho commented on PR #47233: URL: https://github.com/apache/spark/pull/47233#issuecomment-2270302760 rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on code in PR #47528: URL: https://github.com/apache/spark/pull/47528#discussion_r1704854526 ## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala: ## @@ -439,8 +447,9 @@ case class TransformWithStateExec(

Re: [PR] [SPARK-49021][SS] Add support for reading transformWithState value state variables with state data source reader [spark]

2024-08-05 Thread via GitHub
HeartSaVioR commented on PR #47574: URL: https://github.com/apache/spark/pull/47574#issuecomment-2270311641 General comment: JIRA ticket seems to be mentioning list / map state. What's the right thing? Please make a correction in either thing. -- This is an automated message from the Apac

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
yaooqinn commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704860976 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala: ## @@ -343,6 +351,15 @@ class SessionCatalog( synchronized { currentDb

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
amaliujia commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704861748 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala: ## @@ -343,6 +351,15 @@ class SessionCatalog( synchronized { currentDb

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
amaliujia commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704861748 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala: ## @@ -343,6 +351,15 @@ class SessionCatalog( synchronized { currentDb

Re: [PR] [SPARK-49021][SS] Add support for reading transformWithState value state variables with state data source reader [spark]

2024-08-05 Thread via GitHub
anishshri-db commented on PR #47574: URL: https://github.com/apache/spark/pull/47574#issuecomment-2270317845 > General comment: JIRA ticket seems to be mentioning list / map state and PR title is mentioning value state. What's correct? Please make a correction. Apologies - JIRA ticket

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
yaooqinn commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704862902 ## sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala: ## @@ -106,19 +106,25 @@ class CatalogManager( } } - def setCurr

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
yaooqinn commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704864528 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala: ## @@ -343,6 +351,15 @@ class SessionCatalog( synchronized { currentDb

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
HeartSaVioR commented on PR #47616: URL: https://github.com/apache/spark/pull/47616#issuecomment-2270321688 Thanks! Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog [spark]

2024-08-05 Thread via GitHub
amaliujia commented on code in PR #47592: URL: https://github.com/apache/spark/pull/47592#discussion_r1704866858 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala: ## @@ -343,6 +351,15 @@ class SessionCatalog( synchronized { currentDb

Re: [PR] [SPARK-49114] Sub-categorize cannot load state store errors [spark]

2024-08-05 Thread via GitHub
HeartSaVioR closed pull request #47616: [SPARK-49114] Sub-categorize cannot load state store errors URL: https://github.com/apache/spark/pull/47616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

<    1   2   3   >