HyukjinKwon closed pull request #47435: [MINOR][DOCS] Fix some typos in
`LZFBenchmark`
URL: https://github.com/apache/spark/pull/47435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
HyukjinKwon commented on PR #47435:
URL: https://github.com/apache/spark/pull/47435#issuecomment-2242240353
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on code in PR #47437:
URL: https://github.com/apache/spark/pull/47437#discussion_r1686068685
##
python/pyspark/errors/exceptions/base.py:
##
@@ -137,11 +137,11 @@ def _log_exception(self) -> None:
if query_context.contextType().name == "DataFra
HyukjinKwon commented on code in PR #47437:
URL: https://github.com/apache/spark/pull/47437#discussion_r1686067684
##
python/pyspark/errors/exceptions/base.py:
##
@@ -137,11 +137,11 @@ def _log_exception(self) -> None:
if query_context.contextType().name == "DataFra
panbingkun commented on code in PR #47438:
URL: https://github.com/apache/spark/pull/47438#discussion_r1686077120
##
.github/workflows/benchmark.yml:
##
@@ -50,7 +58,7 @@ jobs:
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
env:
- SPARK_BENCHMARK_N
panbingkun commented on PR #47438:
URL: https://github.com/apache/spark/pull/47438#issuecomment-2242265722
Using the modified `workflows/benchmark` above, a
benchmark(`org.apache.spark.io.ZStandardBenchmark`) was triggered as follows:
https://github.com/panbingkun/spark/actions/runs/1003643
mihailom-db commented on PR #47364:
URL: https://github.com/apache/spark/pull/47364#issuecomment-2242283304
Hi @panbingkun, thanks for taking initiative to push this work forward. The
design of the table was discussed previously and the structure that was agreed
upon should take a slightly
LuciferYang commented on code in PR #46515:
URL: https://github.com/apache/spark/pull/46515#discussion_r1686131240
##
common/network-common/src/main/java/org/apache/spark/network/crypto/CtrTransportCipher.java:
##
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundat
mihailom-db commented on code in PR #47364:
URL: https://github.com/apache/spark/pull/47364#discussion_r1686116407
##
docs/sql-ref-ansi-compliance.md:
##
@@ -442,6 +442,7 @@ Below is a list of all the keywords in Spark SQL.
|CODEGEN|non-reserved|non-reserved|non-reserved|
|COL
panbingkun commented on PR #47364:
URL: https://github.com/apache/spark/pull/47364#issuecomment-2242344407
> Apart from SQL API, we need to support other APIs as well, which should be
used by calling `Session.catalog.collation`. Because of this, your approach
might need to be reworked a bit
mihailom-db commented on PR #47364:
URL: https://github.com/apache/spark/pull/47364#issuecomment-2242350155
I believe for now we agreed to have only `SHOW COLLATION(S)` as a command,
and then add support for both LIKE and ILIKE operators for searching. But it is
enough to have LIKE as a sta
panbingkun commented on PR #47364:
URL: https://github.com/apache/spark/pull/47364#issuecomment-2242353399
> I believe for now we agreed to have only `SHOW COLLATION(S)` as a command,
and then add support for both LIKE and ILIKE operators for searching. But it is
enough to have LIKE as a st
LuciferYang commented on code in PR #47414:
URL: https://github.com/apache/spark/pull/47414#discussion_r1686150662
##
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##
@@ -1275,11 +1275,32 @@ class JDBCV2Suite extends QueryTest with
SharedSparkSession with
panbingkun commented on code in PR #47364:
URL: https://github.com/apache/spark/pull/47364#discussion_r1686161417
##
docs/sql-ref-ansi-compliance.md:
##
@@ -442,6 +442,7 @@ Below is a list of all the keywords in Spark SQL.
|CODEGEN|non-reserved|non-reserved|non-reserved|
|COLL
mihailom-db commented on code in PR #47364:
URL: https://github.com/apache/spark/pull/47364#discussion_r1686171509
##
docs/sql-ref-ansi-compliance.md:
##
@@ -442,6 +442,7 @@ Below is a list of all the keywords in Spark SQL.
|CODEGEN|non-reserved|non-reserved|non-reserved|
|COL
panbingkun commented on code in PR #47364:
URL: https://github.com/apache/spark/pull/47364#discussion_r1686179885
##
docs/sql-ref-ansi-compliance.md:
##
@@ -442,6 +442,7 @@ Below is a list of all the keywords in Spark SQL.
|CODEGEN|non-reserved|non-reserved|non-reserved|
|COLL
panbingkun commented on code in PR #47364:
URL: https://github.com/apache/spark/pull/47364#discussion_r1686181436
##
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##
@@ -918,4 +967,8 @@ public static String getClosestSuggestionsOnInvalidNa
panbingkun commented on code in PR #47364:
URL: https://github.com/apache/spark/pull/47364#discussion_r1686181924
##
common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:
##
@@ -91,7 +91,7 @@ public Optional getVersion() {
/**
* Entry encap
cxzl25 commented on code in PR #46805:
URL: https://github.com/apache/spark/pull/46805#discussion_r1686216130
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/BlockStoreClient.java:
##
@@ -46,6 +53,10 @@
public abstract class BlockStoreClient implements
anikakelhanka opened a new pull request, #47439:
URL: https://github.com/apache/spark/pull/47439
…partitionOveriteMode=dynamic
### What changes were proposed in this pull request?
**Issue:**
In the Spark versions post v3.0.2, the SUCCESS Marker file is missing on the
roo
mikoszilard commented on PR #47419:
URL: https://github.com/apache/spark/pull/47419#issuecomment-2242518732
Thank you very much @dongjoon-hyun. I'm really happy that I could start
contributing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please
HyukjinKwon commented on PR #47438:
URL: https://github.com/apache/spark/pull/47438#issuecomment-2242520759
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #47438: [SPARK-48962][INFRA] Make the input
parameters of `workflows/benchmark` selectable
URL: https://github.com/apache/spark/pull/47438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
yaooqinn commented on PR #47181:
URL: https://github.com/apache/spark/pull/47181#issuecomment-2242591010
The use of version-numbered configurations in Spark can be unfriendly for
users compared to legacy configurations. Various version policies in Spark,
such as Thrift Server Versions, API
cxzl25 commented on code in PR #46805:
URL: https://github.com/apache/spark/pull/46805#discussion_r1686322586
##
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/BlockStoreClient.java:
##
@@ -161,6 +172,22 @@ public void getHostLocalDirs(
String[] exe
wayneguow commented on code in PR #47414:
URL: https://github.com/apache/spark/pull/47414#discussion_r1686324246
##
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##
@@ -1275,11 +1275,32 @@ class JDBCV2Suite extends QueryTest with
SharedSparkSession with E
yaooqinn opened a new pull request, #47440:
URL: https://github.com/apache/spark/pull/47440
### What changes were proposed in this pull request?
Support JIRA_ACCESS_TOKEN in translate-contributors.py
### Why are the changes needed?
Remove plaintext password in
yaooqinn commented on PR #47440:
URL: https://github.com/apache/spark/pull/47440#issuecomment-2242624638
cc @dongjoon-hyun @HyukjinKwon @cloud-fan thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
yaooqinn opened a new pull request, #47441:
URL: https://github.com/apache/spark/pull/47441
### What changes were proposed in this pull request?
Recognized these contribtuor translations
```diff
+Yikf - Kaifei Yi
+jackylee-ch - Junqing Li
+liujiayi771 - Jiayi Liu
+mahesh
bozhang2820 commented on code in PR #46302:
URL: https://github.com/apache/spark/pull/46302#discussion_r1686367506
##
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##
@@ -176,7 +176,13 @@ object SQLExecution extends Logging {
shuffleIds.
davidm-db opened a new pull request, #47442:
URL: https://github.com/apache/spark/pull/47442
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How w
stevomitric commented on code in PR #47372:
URL: https://github.com/apache/spark/pull/47372#discussion_r1686423780
##
sql/core/src/test/scala/org/apache/spark/sql/CollationSQLExpressionsSuite.scala:
##
@@ -2295,6 +2295,827 @@ class CollationSQLExpressionsSuite
assert(typeEx
LuciferYang commented on PR #47418:
URL: https://github.com/apache/spark/pull/47418#issuecomment-2242847274
I haven't reviewed the code changes in the pr yet, but:
1. The PR title should reflect the work done in the current PR as much as
possible, the current title looks more like a Jira
yaooqinn commented on PR #47181:
URL: https://github.com/apache/spark/pull/47181#issuecomment-2242860051
Also, any number of features in a version can lead to backporting issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
LuciferYang commented on code in PR #47418:
URL: https://github.com/apache/spark/pull/47418#discussion_r1686486058
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/V1Writes.scala:
##
@@ -43,7 +43,7 @@ trait V1WriteCommand extends DataWritingCommand {
/**
urosstan-db opened a new pull request, #47443:
URL: https://github.com/apache/spark/pull/47443
### What changes were proposed in this pull request?
Add new interface (`ExternallyPlannedV1Scan`) for `V1Scan` which will
prevent `DataSourceV2Strategy` to do planning of optimized scan node.
urosstan-db commented on code in PR #47443:
URL: https://github.com/apache/spark/pull/47443#discussion_r1686489425
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##
@@ -108,6 +108,9 @@ class DataSourceV2Strategy(session: Spar
WeichenXu123 closed pull request #47411: [SPARK-48941][PYTHON][ML] Replace RDD
read / write API invocation with Dataframe read / write API
URL: https://github.com/apache/spark/pull/47411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
WeichenXu123 commented on PR #47411:
URL: https://github.com/apache/spark/pull/47411#issuecomment-2242944242
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
miland-db commented on code in PR #47442:
URL: https://github.com/apache/spark/pull/47442#discussion_r1686546091
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##
@@ -175,10 +175,22 @@ class AstBuilder extends DataTypeAstBuilder with
SQLCo
davidm-db commented on code in PR #47403:
URL: https://github.com/apache/spark/pull/47403#discussion_r1686579965
##
sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -650,14 +657,27 @@ class SparkSession private(
private[sql] def sql(sqlText: String, args
cloud-fan commented on PR #47433:
URL: https://github.com/apache/spark/pull/47433#issuecomment-2243070022
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
cloud-fan closed pull request #47433: [SPARK-48959][SQL] Make
`NoSuchNamespaceException` extend `NoSuchDatabaseException` to restore the
exception handling
URL: https://github.com/apache/spark/pull/47433
--
This is an automated message from the Apache Git Service.
To respond to the message,
uros-db opened a new pull request, #47444:
URL: https://github.com/apache/spark/pull/47444
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was
CavemanIV commented on PR #40921:
URL: https://github.com/apache/spark/pull/40921#issuecomment-2243095339
many thanks for @cxzl25 adding UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spe
davidm-db commented on code in PR #47403:
URL: https://github.com/apache/spark/pull/47403#discussion_r1686757282
##
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##
@@ -71,14 +85,14 @@ trait NonLeafStatementExec extends CompoundStatement
davidm-db commented on code in PR #47403:
URL: https://github.com/apache/spark/pull/47403#discussion_r1686758572
##
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##
@@ -91,21 +105,41 @@ class SingleStatementExec(
}
override def r
davidm-db commented on code in PR #47403:
URL: https://github.com/apache/spark/pull/47403#discussion_r1686780223
##
sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala:
##
@@ -91,21 +105,41 @@ class SingleStatementExec(
}
override def r
zedtang commented on code in PR #47301:
URL: https://github.com/apache/spark/pull/47301#discussion_r1686736484
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala:
##
@@ -201,6 +201,22 @@ final class DataFrameWriter[T] private[sql] (ds:
Dat
dongjoon-hyun closed pull request #47440: [SPARK-48963][INFRA] Support
`JIRA_ACCESS_TOKEN` in translate-contributors.py
URL: https://github.com/apache/spark/pull/47440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
ericm-db opened a new pull request, #47445:
URL: https://github.com/apache/spark/pull/47445
### What changes were proposed in this pull request?
Introducing the OperatorStateMetadataV2 format that integrates with the
TransformWithStateExec operator. This is used to keep inform
dongjoon-hyun commented on PR #47440:
URL: https://github.com/apache/spark/pull/47440#issuecomment-2243304347
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen
ericm-db closed pull request #47273: [SPARK-48849][SS]Create
OperatorStateMetadataV2 for the TransformWithStateExec operator
URL: https://github.com/apache/spark/pull/47273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
dongjoon-hyun closed pull request #47432: [SPARK-48958][BUILD] Upgrade
`zstd-jni` to 1.5.6-4
URL: https://github.com/apache/spark/pull/47432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the speci
milastdbx commented on PR #47181:
URL: https://github.com/apache/spark/pull/47181#issuecomment-2243402879
@yaooqinn
> However, the current system is not working in that way.
What do you mean?
> The rules for data type mapping cannot be determined by dialect versions
alone. Vari
srielau commented on code in PR #47405:
URL: https://github.com/apache/spark/pull/47405#discussion_r1686852508
##
sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/parsers.scala:
##
@@ -249,20 +249,24 @@ class ParseException private(
override def getMessage: String
tonyye commented on PR #22673:
URL: https://github.com/apache/spark/pull/22673#issuecomment-2243404837
I know this is an old issue but does anyone know if this has changed in more
recent versions of Spark? Is reading sorted data read in by spark in the same
order? @darabos or @dgrnbrg do ei
jingz-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1686896779
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala:
##
@@ -66,7 +67,7 @@ class StateSchemaCompatibilityChe
ericm-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1686910533
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala:
##
@@ -66,7 +67,7 @@ class StateSchemaCompatibilityChe
andylam-db commented on PR #47375:
URL: https://github.com/apache/spark/pull/47375#issuecomment-2243487503
bumping @cloud-fan @HyukjinKwon for merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
ericm-db opened a new pull request, #47446:
URL: https://github.com/apache/spark/pull/47446
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How wa
gengliangwang closed pull request #47405: [SPARK-48929] Fix view internal error
and clean up parser exception context
URL: https://github.com/apache/spark/pull/47405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
gengliangwang commented on PR #47405:
URL: https://github.com/apache/spark/pull/47405#issuecomment-2243557854
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specifi
anishshri-db commented on PR #47431:
URL: https://github.com/apache/spark/pull/47431#issuecomment-2243616217
Tests are all green. Link here -
https://github.com/anishshri-db/spark/actions/runs/10035345281/job/27763418833
--
This is an automated message from the Apache Git Service.
To resp
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687021568
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##
@@ -208,14 +208,25 @@ class IncrementalExecution(
}
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687022712
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/metadata/StateMetadataSource.scala:
##
@@ -188,29 +191,56 @@ class StateMetadataPart
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687023328
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/metadata/StateMetadataSource.scala:
##
@@ -188,29 +191,56 @@ class StateMetadataPart
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687023864
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala:
##
@@ -325,6 +340,19 @@ class HDFSMetadataLog[T <: AnyRef :
ClassTag](s
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687024442
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OperatorStateMetadataLog.scala:
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Fou
dtenedor commented on PR #47447:
URL: https://github.com/apache/spark/pull/47447#issuecomment-2243631525
cc @allisonwang-db @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687025381
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala:
##
@@ -382,12 +397,45 @@ case class TransformWithStateExec(
dtenedor opened a new pull request, #47447:
URL: https://github.com/apache/spark/pull/47447
### What changes were proposed in this pull request?
This bug covers improving an error message in the event of invalid UDTF
calls. For example:
```
select * from udtf(
observed
jingz-db closed pull request #47306: [WIP] State data integration
URL: https://github.com/apache/spark/pull/47306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe
jingz-db opened a new pull request, #47448:
URL: https://github.com/apache/spark/pull/47448
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How wa
jingz-db closed pull request #47448: State source value
URL: https://github.com/apache/spark/pull/47448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
jingz-db opened a new pull request, #47449:
URL: https://github.com/apache/spark/pull/47449
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How wa
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687042904
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/metadata/StateMetadataSource.scala:
##
@@ -188,29 +191,56 @@ class StateMetadataPart
dongjoon-hyun commented on PR #47441:
URL: https://github.com/apache/spark/pull/47441#issuecomment-2243661735
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen
dongjoon-hyun closed pull request #47441: [MINOR][INFRA] Add more know
translations for contributors
URL: https://github.com/apache/spark/pull/47441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to t
jingz-db closed pull request #47449: New value state
URL: https://github.com/apache/spark/pull/47449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: rev
jingz-db opened a new pull request, #47450:
URL: https://github.com/apache/spark/pull/47450
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How wa
anishshri-db commented on code in PR #47445:
URL: https://github.com/apache/spark/pull/47445#discussion_r1687054341
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala:
##
@@ -219,7 +222,8 @@ object StateSchemaCompatibi
zedtang opened a new pull request, #47451:
URL: https://github.com/apache/spark/pull/47451
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was
chaoqin-li1123 commented on code in PR #47393:
URL: https://github.com/apache/spark/pull/47393#discussion_r1687060460
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##
@@ -426,7 +441,29 @@ class RocksDBFileManager(
* - SST
zedtang commented on PR #47451:
URL: https://github.com/apache/spark/pull/47451#issuecomment-2243690425
This PR depends on https://github.com/apache/spark/pull/47301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
zedtang commented on PR #47452:
URL: https://github.com/apache/spark/pull/47452#issuecomment-2243708336
This PR depends on https://github.com/apache/spark/pull/47451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
zedtang opened a new pull request, #47452:
URL: https://github.com/apache/spark/pull/47452
### What changes were proposed in this pull request?
Introduce clusterBy DataFrameWriter API for Python.
### Why are the changes needed?
Introduce more ways for users to int
zedtang commented on code in PR #47376:
URL: https://github.com/apache/spark/pull/47376#discussion_r1687098487
##
sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala:
##
@@ -303,7 +331,7 @@ final class DataStreamWriter[T] private[sql](ds:
Dataset[T]) {
dongjoon-hyun opened a new pull request, #20:
URL: https://github.com/apache/spark-kubernetes-operator/pull/20
…
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing chang
dongjoon-hyun commented on PR #20:
URL:
https://github.com/apache/spark-kubernetes-operator/pull/20#issuecomment-2243900381
Thank you, @huaxingao !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
dongjoon-hyun closed pull request #20: [SPARK-48968] Avoid unnecessary task
configuration in `spark-operator-api`
URL: https://github.com/apache/spark-kubernetes-operator/pull/20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
anishshri-db commented on code in PR #47133:
URL: https://github.com/apache/spark/pull/47133#discussion_r1687194253
##
python/pyspark/sql/pandas/group_ops.py:
##
@@ -358,6 +362,120 @@ def applyInPandasWithState(
)
return DataFrame(jdf, self.session)
+
+de
bogao007 commented on code in PR #47133:
URL: https://github.com/apache/spark/pull/47133#discussion_r1687203660
##
python/pyspark/sql/streaming/state_api_client.py:
##
@@ -0,0 +1,162 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor licen
chaoqin-li1123 commented on PR #47393:
URL: https://github.com/apache/spark/pull/47393#issuecomment-2243936187
Can you also update the pr description about the new conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
dongjoon-hyun commented on PR #47075:
URL: https://github.com/apache/spark/pull/47075#issuecomment-2243948941
Merged to master for Apache Spark 4.0.0-preview2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL ab
dongjoon-hyun closed pull request #47075: [SPARK-48703][SQL][TESTS] Upgrade
`mssql-jdbc` to 12.6.3.jre11
URL: https://github.com/apache/spark/pull/47075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
dongjoon-hyun opened a new pull request, #21:
URL: https://github.com/apache/spark-kubernetes-operator/pull/21
…
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing chang
jiangzho commented on code in PR #46716:
URL: https://github.com/apache/spark/pull/46716#discussion_r1687244271
##
core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala:
##
@@ -24,15 +24,21 @@ import jakarta.servlet.http.HttpServletRequest
import org.eclipse
dongjoon-hyun commented on PR #21:
URL:
https://github.com/apache/spark-kubernetes-operator/pull/21#issuecomment-2243985331
Oh, you are online. I hope you are getting better and better.
Thank you so much, @viirya !
--
This is an automated message from the Apache Git Service.
To respo
dongjoon-hyun closed pull request #21: [SPARK-48969] Fix `spark-operator`
module to define test framework explicitly
URL: https://github.com/apache/spark-kubernetes-operator/pull/21
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
1 - 100 of 156 matches
Mail list logo
