gengliangwang commented on code in PR #50761:
URL: https://github.com/apache/spark/pull/50761#discussion_r2070951619
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraint.scala:
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Fo
attilapiros commented on PR #50230:
URL: https://github.com/apache/spark/pull/50230#issuecomment-2846072525
I miss the logging: I would prefer to have some log lines which helps to
figure out what has happened regarding the row based checksums during a run. At
least at debug level. WDYT?
anishshri-db commented on code in PR #50773:
URL: https://github.com/apache/spark/pull/50773#discussion_r2070974574
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ChecksumCheckpointFileManager.scala:
##
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Softwa
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070893252
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070893252
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
aokolnychyi commented on code in PR #50761:
URL: https://github.com/apache/spark/pull/50761#discussion_r2070887598
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraint.scala:
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foun
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070895177
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
ueshin commented on PR #50777:
URL: https://github.com/apache/spark/pull/50777#issuecomment-2845935604
The test failure should be fixed by #50778.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
sririshindra commented on code in PR #50769:
URL: https://github.com/apache/spark/pull/50769#discussion_r2070179418
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AggregateResolver.scala:
##
@@ -329,4 +332,17 @@ class AggregateResolver(operatorRes
ahshahid commented on PR #50757:
URL: https://github.com/apache/spark/pull/50757#issuecomment-2845350793
I also doubt that user would be able to specify the inDeterminancy flag in
the Map operations as I think it is going to make it complicated for user to
understand its impact, and if inad
peter-toth commented on PR #50757:
URL: https://github.com/apache/spark/pull/50757#issuecomment-2845349959
Thanks @cloud-fan , somehow I missed that PR. Runtime shuffle checksum seems
like a good idea, but it must come with some costs as well.
--
This is an automated message from the Apac
viirya commented on PR #180:
URL:
https://github.com/apache/spark-kubernetes-operator/pull/180#issuecomment-2845367431
Pending CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific com
ahshahid commented on code in PR #50757:
URL: https://github.com/apache/spark/pull/50757#discussion_r2070588096
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:
##
@@ -103,13 +103,21 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
ahshahid commented on code in PR #50757:
URL: https://github.com/apache/spark/pull/50757#discussion_r2070581649
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:
##
@@ -103,13 +103,21 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
liviazhu-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070593834
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##
@@ -446,17 +459,48 @@ private[sql] class RocksDBStateS
dongjoon-hyun opened a new pull request, #102:
URL: https://github.com/apache/spark-connect-swift/pull/102
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
micheal-o commented on code in PR #50045:
URL: https://github.com/apache/spark/pull/50045#discussion_r2069927112
##
common/utils/src/main/resources/error/error-conditions.json:
##
@@ -324,6 +324,17 @@
"The change log writer version cannot be ."
]
},
+
ericm-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070607523
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##
@@ -446,17 +459,48 @@ private[sql] class RocksDBStateStor
ericm-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070607523
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##
@@ -446,17 +459,48 @@ private[sql] class RocksDBStateStor
liviazhu-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070612638
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##
@@ -446,17 +459,48 @@ private[sql] class RocksDBStateS
ericm-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070615297
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##
@@ -446,17 +459,48 @@ private[sql] class RocksDBStateStor
dongjoon-hyun commented on PR #180:
URL:
https://github.com/apache/spark-kubernetes-operator/pull/180#issuecomment-2845432060
Thank you, @viirya !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
ericm-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070624489
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##
@@ -446,17 +459,48 @@ private[sql] class RocksDBStateStor
ericm-db commented on PR #50742:
URL: https://github.com/apache/spark/pull/50742#issuecomment-2845440177
> Looks good! Could you add a test in StateStoreRDDSuite to check the
ThreadLocal logic correctly passes the readstore to the writestore too?
Yup, working on that rn!
--
This is an
sririshindra commented on PR #50769:
URL: https://github.com/apache/spark/pull/50769#issuecomment-2845007884
@vladimirg-db , Could you please point me any existing tests that might have
covered this scenario?
I am thinking there should be a test that checks for the following query
sririshindra commented on PR #50769:
URL: https://github.com/apache/spark/pull/50769#issuecomment-2845039363
> @sririshindra here's a test that fails for single-pass Analyzer at the
moment:
https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-tests/inputs/order-by.sql
dongjoon-hyun commented on PR #50775:
URL: https://github.com/apache/spark/pull/50775#issuecomment-2845044708
All K8s related unit and integration tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
dongjoon-hyun commented on PR #50775:
URL: https://github.com/apache/spark/pull/50775#issuecomment-2845045137
Could you review this PR when you have some time, @LuciferYang ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
vladimirg-db commented on PR #50769:
URL: https://github.com/apache/spark/pull/50769#issuecomment-2845048241
@sririshindra single-pass Analyzer is a project to replace the current
fixed-point Analyzer and is currently under development. It's not yet enabled
by default (not running in CI). I
LuciferYang commented on PR #50775:
URL: https://github.com/apache/spark/pull/50775#issuecomment-2845064073
Merged into master. Thanks @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
LuciferYang closed pull request #50775: [SPARK-51973][K8S][BUILD] Upgrade
`kubernetes-client` to 7.2.0 for K8s 1.33
URL: https://github.com/apache/spark/pull/50775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
LuciferYang commented on PR #50775:
URL: https://github.com/apache/spark/pull/50775#issuecomment-2845061751
It seems that the failure of PySpark is not related to the current PR (Pull
Request). We can merge this one.
--
This is an automated message from the Apache Git Service.
To respond
peter-toth commented on code in PR #50757:
URL: https://github.com/apache/spark/pull/50757#discussion_r2069998646
##
core/src/main/scala/org/apache/spark/rdd/RDD.scala:
##
Review Comment:
1., 2. This PR doesn't change deterministic calculation of plan nodes so it
shouldn't
vladimirg-db commented on PR #50769:
URL: https://github.com/apache/spark/pull/50769#issuecomment-2844630029
@cloud-fan tests passed, please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
cloud-fan commented on PR #50757:
URL: https://github.com/apache/spark/pull/50757#issuecomment-2844972082
My worry is that the `Expression#deterministic` is a bit abused in Spark,
e.g. `SparkPartitionID`, `InputFileName` are marked as nondeterministic, but
they produce the same result when
ueshin opened a new pull request, #50777:
URL: https://github.com/apache/spark/pull/50777
### What changes were proposed in this pull request?
Updates pandas API on Spark option doc.
### Why are the changes needed?
The descriptions for some options are outdated.
##
dongjoon-hyun closed pull request #180: [SPARK-51978] Upgrade
`kubernetes-client` to 7.2.0 for K8s 1.33
URL: https://github.com/apache/spark-kubernetes-operator/pull/180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
szehon-ho commented on code in PR #50701:
URL: https://github.com/apache/spark/pull/50701#discussion_r2069680175
##
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala:
##
@@ -3560,11 +3560,10 @@ class DataSourceV2SQLSuiteV1Filter
val excep
wjszlachta-man opened a new pull request, #50774:
URL: https://github.com/apache/spark/pull/50774
### What changes were proposed in this pull request?
On glibc based Linux systems `select()` can monitor only file descriptor
numbers that are less than `FD_SETSIZE` (1024).
This i
dongjoon-hyun opened a new pull request, #101:
URL: https://github.com/apache/spark-connect-swift/pull/101
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
dongjoon-hyun commented on PR #85:
URL:
https://github.com/apache/spark-connect-swift/pull/85#issuecomment-2844701280
Thank you, @viirya ! Merged to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
dongjoon-hyun closed pull request #85: [SPARK-51871] Improve `SQLTests` to
check column names
URL: https://github.com/apache/spark-connect-swift/pull/85
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
stanlocht commented on PR #50644:
URL: https://github.com/apache/spark/pull/50644#issuecomment-2844704988
Hi @HyukjinKwon, @zhengruifeng, @asl3 — just following up to see if you
might have a chance to review the PR when time allows. Appreciate your time and
input!
--
This is an automated
vladimirg-db commented on code in PR #50769:
URL: https://github.com/apache/spark/pull/50769#discussion_r2070189488
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AggregateResolver.scala:
##
@@ -109,7 +111,8 @@ class AggregateResolver(operatorReso
dongjoon-hyun commented on PR #101:
URL:
https://github.com/apache/spark-connect-swift/pull/101#issuecomment-2845281778
Thank you! Merged to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
dongjoon-hyun closed pull request #101: [SPARK-51976] Add `array`, `map`,
`timestamp`, `posexplode` test queries
URL: https://github.com/apache/spark-connect-swift/pull/101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
dongjoon-hyun opened a new pull request, #180:
URL: https://github.com/apache/spark-kubernetes-operator/pull/180
### What changes were proposed in this pull request?
This PR aims to upgrade `kubernetes-client` to 7.2.0 like Apache Spark.
- https://github.com/apache/spark/pull/50775
ahshahid commented on code in PR #50757:
URL: https://github.com/apache/spark/pull/50757#discussion_r2070564298
##
core/src/main/scala/org/apache/spark/rdd/RDD.scala:
##
Review Comment:
> @ahshahid Regarding point 3 I was open for your change and asked you to
extend your i
peter-toth commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070569934
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under o
allisonwang-db opened a new pull request, #50776:
URL: https://github.com/apache/spark/pull/50776
### What changes were proposed in this pull request?
This PR adds more SQL query tests for SQL User-defined table function with
various valid and invalid return columns.
##
ahshahid commented on PR #50757:
URL: https://github.com/apache/spark/pull/50757#issuecomment-2845342978
IMHO the issue of inDeterministic value of an expression should be looked
only from the basis of whether ShuffleStage can loose/add row because of
inDeterministic nature of the expressio
allisonwang-db commented on PR #50776:
URL: https://github.com/apache/spark/pull/50776#issuecomment-2845344033
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
dongjoon-hyun commented on PR #50775:
URL: https://github.com/apache/spark/pull/50775#issuecomment-2845096329
Thank you, @LuciferYang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specifi
dongjoon-hyun opened a new pull request, #50775:
URL: https://github.com/apache/spark/pull/50775
…s 1.33
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
LuciferYang commented on PR #50764:
URL: https://github.com/apache/spark/pull/50764#issuecomment-2844795461
Thanks @dongjoon-hyun @HyukjinKwon and @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
dongjoon-hyun commented on PR #102:
URL:
https://github.com/apache/spark-connect-swift/pull/102#issuecomment-2845240542
Thank you, @viirya . Merged to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
dongjoon-hyun closed pull request #102: [SPARK-51977] Improve `SparkSQLRepl` to
support multiple lines
URL: https://github.com/apache/spark-connect-swift/pull/102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL ab
liviazhu-db commented on code in PR #50045:
URL: https://github.com/apache/spark/pull/50045#discussion_r2070512349
##
common/utils/src/main/resources/error/error-conditions.json:
##
@@ -324,6 +324,17 @@
"The change log writer version cannot be ."
]
},
liviazhu-db commented on code in PR #50045:
URL: https://github.com/apache/spark/pull/50045#discussion_r2070515391
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:
##
@@ -426,7 +423,7 @@ private[sql] class HDFSBackedSt
liviazhu-db commented on code in PR #50045:
URL: https://github.com/apache/spark/pull/50045#discussion_r2070519903
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreErrors.scala:
##
@@ -435,15 +452,31 @@ class StateStoreFailedToGetChangelogWrite
liviazhu-db commented on code in PR #50045:
URL: https://github.com/apache/spark/pull/50045#discussion_r2070521155
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreErrors.scala:
##
@@ -435,15 +452,31 @@ class StateStoreFailedToGetChangelogWrite
dongjoon-hyun commented on PR #101:
URL:
https://github.com/apache/spark-connect-swift/pull/101#issuecomment-2845256608
Could you review this test case PR when you have some time, @huaxingao ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please l
liviazhu-db commented on code in PR #50045:
URL: https://github.com/apache/spark/pull/50045#discussion_r2070515391
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:
##
@@ -426,7 +423,7 @@ private[sql] class HDFSBackedSt
vrozov commented on PR #49928:
URL: https://github.com/apache/spark/pull/49928#issuecomment-2844998622
@cloud-fan The positive test cases are already covered in the
`JavaDataFrameReaderWriterSuite.java`. Please see
https://github.com/apache/spark/blob/7019d5e63b7218049bacf3392716bf6faf8f82a
vladimirg-db commented on PR #50769:
URL: https://github.com/apache/spark/pull/50769#issuecomment-2845016716
@sririshindra here's a test that fails for single-pass Analyzer at the
moment:
https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-tests/inputs/order-by.sql#L
anishshri-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070673830
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala:
##
@@ -27,6 +27,28 @@ import org.apache.spark.sql.internal.SessionSt
ericm-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070674011
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala:
##
@@ -27,6 +27,28 @@ import org.apache.spark.sql.internal.SessionState
anishshri-db commented on code in PR #50742:
URL: https://github.com/apache/spark/pull/50742#discussion_r2070681239
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala:
##
@@ -27,6 +27,43 @@ import org.apache.spark.sql.internal.SessionSt
LuciferYang opened a new pull request, #50778:
URL: https://github.com/apache/spark/pull/50778
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How
ueshin opened a new pull request, #50779:
URL: https://github.com/apache/spark/pull/50779
### What changes were proposed in this pull request?
Prepares the test environment for pandas API on Spark with ANSI mode enabled.
- Remove forcibly disabling ANSI mode in tests
- Add a
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070923376
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070928028
##
core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java:
##
@@ -330,7 +344,8 @@ private long[] mergeSpillsUsingStandardWriter(SpillInfo[]
spil
anishshri-db commented on code in PR #50773:
URL: https://github.com/apache/spark/pull/50773#discussion_r2070916110
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##
@@ -1247,6 +1255,8 @@ class RocksDB(
silentDeleteRecursively(loc
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070921381
##
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java:
##
@@ -199,6 +214,14 @@ public long[] getPartitionLengths() {
return par
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070928418
##
core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java:
##
@@ -163,6 +167,13 @@ public long getPeakMemoryUsedBytes() {
return peakMemoryU
gjxdxh opened a new pull request, #50780:
URL: https://github.com/apache/spark/pull/50780
### What changes were proposed in this pull request?
Adding a new jobTags parameter for QueryStartedEvent so that it can be
connected to the actual spark connect command that triggered this s
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070893252
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070895177
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070945270
##
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRowChecksum.scala:
##
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundat
asl3 commented on code in PR #50644:
URL: https://github.com/apache/spark/pull/50644#discussion_r2059059636
##
python/pyspark/testing/utils.py:
##
@@ -580,6 +598,7 @@ def compare_datatypes_ignore_nullable(dt1: Any, dt2: Any):
if TYPE_CHECKING:
import pandas
+
Review Com
attilapiros commented on code in PR #50757:
URL: https://github.com/apache/spark/pull/50757#discussion_r2070814828
##
core/src/main/scala/org/apache/spark/rdd/RDD.scala:
##
Review Comment:
Please check this lines in RDD.scala:
https://github.com/apache/spark/blob/085bfc
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070910587
##
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java:
##
@@ -104,6 +105,14 @@ final class BypassMergeSortShuffleWriter
private l
gengliangwang commented on code in PR #50761:
URL: https://github.com/apache/spark/pull/50761#discussion_r2070911373
##
common/utils/src/main/resources/error/error-conditions.json:
##
@@ -544,6 +544,14 @@
],
"sqlState" : "56000"
},
+ "CHECK_CONSTRAINT_VIOLATION" :
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070910837
##
core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java:
##
@@ -132,6 +141,8 @@ final class BypassMergeSortShuffleWriter
this.ser
gengliangwang commented on code in PR #50761:
URL: https://github.com/apache/spark/pull/50761#discussion_r2070913049
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraint.scala:
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Fo
gengliangwang commented on code in PR #50761:
URL: https://github.com/apache/spark/pull/50761#discussion_r2070938781
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraint.scala:
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Fo
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070938550
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
gengliangwang commented on code in PR #50761:
URL: https://github.com/apache/spark/pull/50761#discussion_r2070941624
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala:
##
@@ -259,3 +263,94 @@ case class ForeignKeyConstraint(
copy(use
attilapiros commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2070942000
##
core/src/main/java/org/apache/spark/shuffle/checksum/RowBasedChecksum.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
gengliangwang commented on code in PR #50761:
URL: https://github.com/apache/spark/pull/50761#discussion_r2070943644
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraint.scala:
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Fo
ahshahid commented on PR #50757:
URL: https://github.com/apache/spark/pull/50757#issuecomment-2845812865
I am out right now.. will check back.. but what I m trying to say is that a
map stage should be marked inDeterminate iff the partitioner is using
indeterministic Val. And afaik only
wengh commented on code in PR #50684:
URL: https://github.com/apache/spark/pull/50684#discussion_r2071064159
##
python/pyspark/sql/datasource.py:
##
@@ -539,6 +539,11 @@ def pushFilters(self, filters: List["Filter"]) ->
Iterable["Filter"]:
This method is allowed to mod
wengh commented on code in PR #50684:
URL: https://github.com/apache/spark/pull/50684#discussion_r2071064597
##
python/docs/source/user_guide/sql/python_data_source.rst:
##
@@ -356,17 +356,28 @@ For library that are used inside a method, it must be
imported inside the method
vrozov commented on code in PR #50594:
URL: https://github.com/apache/spark/pull/50594#discussion_r2071066514
##
core/src/main/scala/org/apache/spark/util/UninterruptibleThread.scala:
##
@@ -92,11 +110,27 @@ private[spark] class UninterruptibleThread(
* interrupted until it
wengh commented on code in PR #50684:
URL: https://github.com/apache/spark/pull/50684#discussion_r2071064597
##
python/docs/source/user_guide/sql/python_data_source.rst:
##
@@ -356,17 +356,28 @@ For library that are used inside a method, it must be
imported inside the method
szehon-ho commented on PR #50701:
URL: https://github.com/apache/spark/pull/50701#issuecomment-2846276506
@cloud-fan @aokolnychyi can you guys take a look when you have time? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Gi
szehon-ho commented on code in PR #50701:
URL: https://github.com/apache/spark/pull/50701#discussion_r2071075799
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##
@@ -3994,10 +3994,45 @@ class Analyzer(override val catalogManager:
CatalogM
micheal-o commented on code in PR #50595:
URL: https://github.com/apache/spark/pull/50595#discussion_r2071086887
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala:
##
@@ -1173,6 +1225,107 @@ object StateStore extends Logging {
}
}
dongjoon-hyun closed pull request #50778: [SPARK-51980][PYTHON][TESTS] Enable
`--use-pep517` in `dev/run-pip-tests`
URL: https://github.com/apache/spark/pull/50778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL a
micheal-o commented on code in PR #50595:
URL: https://github.com/apache/spark/pull/50595#discussion_r2071091954
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala:
##
@@ -,60 +1159,64 @@ object StateStore extends Logging {
}
}
1 - 100 of 112 matches
Mail list logo
