HeartSaVioR commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2662329607
cc. @dongjoon-hyun @HyukjinKwon Please take a look. Thanks!
cc. @cloud-fan for visibility of the fix for blocker issue
--
This is an automated message from the Apache Git Service.
HeartSaVioR commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2662330595
* #49984 (4.0)
* #49985 (3.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the sp
HeartSaVioR commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2662322067
I'm going to submit PRs for 4.0/3.5 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
HeartSaVioR commented on code in PR #49983:
URL: https://github.com/apache/spark/pull/49983#discussion_r1957769009
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala:
##
@@ -1471,6 +1471,75 @@ class StreamingQuerySuite extends StreamTest with
Be
fusheng9399 commented on code in PR #49875:
URL: https://github.com/apache/spark/pull/49875#discussion_r1957769480
##
python/pyspark/sql/functions/builtin.py:
##
@@ -20115,11 +20115,26 @@ def get_json_object(col: "ColumnOrName", path: str)
-> Column:
Examples
--
HeartSaVioR opened a new pull request, #49984:
URL: https://github.com/apache/spark/pull/49984
### What changes were proposed in this pull request?
This PR proposes to implement the graceful deprecation of incorrect config
introduced in SPARK-49699.
SPARK-49699 was included in
HeartSaVioR opened a new pull request, #49985:
URL: https://github.com/apache/spark/pull/49985
### What changes were proposed in this pull request?
This PR proposes to implement the graceful deprecation of incorrect config
introduced in SPARK-49699.
SPARK-49699 was included in
cloud-fan commented on code in PR #49983:
URL: https://github.com/apache/spark/pull/49983#discussion_r1958020639
##
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##
@@ -4115,6 +4115,7 @@ object SQLConf {
.doc("Allow PruneFilters to remove stream
ostronaut commented on code in PR #49959:
URL: https://github.com/apache/spark/pull/49959#discussion_r1957847665
##
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala:
##
@@ -48,9 +49,6 @@ case class ShowTablesExec(
}
private def i
itholic commented on PR #49973:
URL: https://github.com/apache/spark/pull/49973#issuecomment-2662495514
Late LGTM. Thanks for addressing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
wayneguow commented on code in PR #49980:
URL: https://github.com/apache/spark/pull/49980#discussion_r1958082575
##
.github/workflows/build_and_test.yml:
##
@@ -1028,8 +1028,8 @@ jobs:
needs: precondition
if: fromJson(needs.precondition.outputs.required).tpcds-1g == 't
LuciferYang commented on code in PR #49980:
URL: https://github.com/apache/spark/pull/49980#discussion_r1958090621
##
.github/workflows/benchmark.yml:
##
@@ -67,8 +67,7 @@ jobs:
# Any TPC-DS related updates on this job need to be applied to tpcds-1g job
of build_and_test.yml
pan3793 commented on PR #49986:
URL: https://github.com/apache/spark/pull/49986#issuecomment-2662640045
cc @zhengruifeng @panbingkun, could you please take a look? and do you have
a better idea of how to implement the configuration?
--
This is an automated message from the Apache Git Serv
zhengruifeng commented on code in PR #49982:
URL: https://github.com/apache/spark/pull/49982#discussion_r1957816031
##
python/pyspark/errors/utils.py:
##
@@ -268,6 +266,8 @@ def wrapper(*args: Any, **kwargs: Any) -> Any:
finally:
set_current
cloud-fan commented on PR #49970:
URL: https://github.com/apache/spark/pull/49970#issuecomment-2662739977
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
pan3793 opened a new pull request, #49986:
URL: https://github.com/apache/spark/pull/49986
### What changes were proposed in this pull request?
This PR proposes introducing a new configuration `spark.ml.allowNativeBlas`,
when set to `false`, Spark always uses Java BLAS even wh
pan3793 commented on PR #49986:
URL: https://github.com/apache/spark/pull/49986#issuecomment-2662622310
the current approach works with `spark-submit`
```
spark-submit --conf spark.ml.allowNativeBlas=false ...
```
but does not work with creating embedded `SparkContext` in user
beliefer commented on PR #49981:
URL: https://github.com/apache/spark/pull/49981#issuecomment-2662927982
ping @cloud-fan cc @vitaliili-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
beliefer commented on PR #49981:
URL: https://github.com/apache/spark/pull/49981#issuecomment-2662929102
ping @MaxGekk cc @vitaliili-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specif
beliefer opened a new pull request, #49987:
URL: https://github.com/apache/spark/pull/49987
### What changes were proposed in this pull request?
This PR proposes to format the SqlBaseParser.g4
### Why are the changes needed?
Currently, some declaration in SqlBaseParser.g4 is
HeartSaVioR commented on PR #49978:
URL: https://github.com/apache/spark/pull/49978#issuecomment-2663042734
Thanks! Merging to master/4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the spec
beliefer commented on PR #49987:
URL: https://github.com/apache/spark/pull/49987#issuecomment-2662933859
ping @MaxGekk @dongjoon-hyun cc @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
HeartSaVioR closed pull request #49978: [SPARK-51237][SS] Add API details for
new transformWithState helper APIs as needed
URL: https://github.com/apache/spark/pull/49978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
cloud-fan commented on code in PR #49962:
URL: https://github.com/apache/spark/pull/49962#discussion_r1958259900
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala:
##
@@ -340,12 +341,43 @@ object ResolveDefaultColumns extends Query
cloud-fan commented on code in PR #49962:
URL: https://github.com/apache/spark/pull/49962#discussion_r1958258070
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala:
##
@@ -340,12 +341,43 @@ object ResolveDefaultColumns extends Query
asfgit closed pull request #49921: [SPARK-51192][CONNECT] Expose
`processWithoutResponseObserverForTesting` in `SparkConnectPlanner`
URL: https://github.com/apache/spark/pull/49921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
LuciferYang commented on PR #49987:
URL: https://github.com/apache/spark/pull/49987#issuecomment-2663256574
I don't object to formatting this file, but are there any Maven/SBT plugins
that can automate the formatting process? For example, `spotless-maven-plugin`?
Or any others?
--
This i
HeartSaVioR commented on PR #49978:
URL: https://github.com/apache/spark/pull/49978#issuecomment-2663042333
https://github.com/anishshri-db/spark/actions/runs/13364971025/job/37322247149
Failure is unrelated.
--
This is an automated message from the Apache Git Service.
To respond to th
cloud-fan closed pull request #49970: [SPARK-51228][SQL] Introduce subquery
normalization to NormalizePlan
URL: https://github.com/apache/spark/pull/49970
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
steveloughran commented on PR #49779:
URL: https://github.com/apache/spark/pull/49779#issuecomment-2663444170
I can't think of any changes, unless we want to set that audit stuff even if
caller context is not being set.
--
This is an automated message from the Apache Git Service.
To resp
steveloughran commented on PR #49893:
URL: https://github.com/apache/spark/pull/49893#issuecomment-2663472530
ok, so with this #49779 doesn't need the changes in the production code, it
can just
* rely on the context flag being set
* skip the tests to unset it
happy
--
This
asfgit closed pull request #49604: [SPARK-50849][Connect] Add example project
to demonstrate Spark Connect Server Libraries
URL: https://github.com/apache/spark/pull/49604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
hvanhovell commented on PR #49604:
URL: https://github.com/apache/spark/pull/49604#issuecomment-2663184327
Merging master/4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
asfgit closed pull request #49964: [SPARK-51085][SQL] Restore SQLContext
Companion
URL: https://github.com/apache/spark/pull/49964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific commen
sunxiaoguang commented on PR #49453:
URL: https://github.com/apache/spark/pull/49453#issuecomment-2663207841
> Thank you @sunxiaoguang, the changes almost look good to me, except for
two minor comments.
Hello @beliefer, @yaooqinn might be busy on something and it's been quite a
while
pan3793 commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1958982565
##
mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:
##
@@ -39,8 +39,11 @@ private[spark] object BLAS extends Serializable {
// For level-3 routines,
wayneguow opened a new pull request, #49988:
URL: https://github.com/apache/spark/pull/49988
### What changes were proposed in this pull request?
This PR aims to upgrade Github Action image to `ubuntu-latest` and solved
the `TPCDSQueryBenchmark` and `k8s-integration-tests` com
panbingkun commented on code in PR #49875:
URL: https://github.com/apache/spark/pull/49875#discussion_r1958974063
##
python/pyspark/sql/functions/builtin.py:
##
@@ -20115,11 +20115,47 @@ def get_json_object(col: "ColumnOrName", path: str)
-> Column:
Examples
---
srowen commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1958980059
##
core/src/main/scala/org/apache/spark/SparkContext.scala:
##
@@ -3436,6 +3437,20 @@ object SparkContext extends Logging {
supplement(DRIVER_JAVA_OPTIONS)
sup
pan3793 commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1958986727
##
core/src/main/scala/org/apache/spark/SparkContext.scala:
##
@@ -3436,6 +3437,20 @@ object SparkContext extends Logging {
supplement(DRIVER_JAVA_OPTIONS)
su
zhengruifeng closed pull request #49975: [SPARK-51234][PYTHON][DOCS] Document
an import change in `from pyspark.sql.functions import *`
URL: https://github.com/apache/spark/pull/49975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
pan3793 commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1958978004
##
core/src/main/scala/org/apache/spark/SparkContext.scala:
##
@@ -3436,6 +3437,20 @@ object SparkContext extends Logging {
supplement(DRIVER_JAVA_OPTIONS)
su
panbingkun commented on code in PR #49875:
URL: https://github.com/apache/spark/pull/49875#discussion_r1958984613
##
python/pyspark/sql/functions/builtin.py:
##
@@ -20115,11 +20115,47 @@ def get_json_object(col: "ColumnOrName", path: str)
-> Column:
Examples
---
zhengruifeng commented on PR #49975:
URL: https://github.com/apache/spark/pull/49975#issuecomment-2664458299
thanks, merged to master/4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the speci
beliefer commented on code in PR #49981:
URL: https://github.com/apache/spark/pull/49981#discussion_r1958990822
##
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##
@@ -1160,7 +1160,6 @@ primaryExpression
| primaryExpression DOUBLE_COLON data
wayneguow commented on PR #49988:
URL: https://github.com/apache/spark/pull/49988#issuecomment-2664492465
cc @LuciferYang , because there were more changes on branch 3.5 than the
master branch, I used a new jira ID.
--
This is an automated message from the Apache Git Service.
To respond t
cloud-fan commented on code in PR #49678:
URL: https://github.com/apache/spark/pull/49678#discussion_r1959005841
##
sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala:
##
@@ -2721,6 +2721,25 @@ class DataFrameSuite extends QueryTest
parameters = Map("name"
zhengruifeng opened a new pull request, #49989:
URL: https://github.com/apache/spark/pull/49989
nit
### What changes were proposed in this pull request?
### Why are the changes needed?
Add missing backticks in `Upgrading from PySpark 3.5 to 4.0`
see
ht
HeartSaVioR commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2664583582
The migration logic can't be removed anytime soon. I'd say we could only
remove it in 4.2 or so.
I guess the main point here is when we can stop allowing users to specify
the i
panbingkun commented on code in PR #49875:
URL: https://github.com/apache/spark/pull/49875#discussion_r1958981584
##
python/pyspark/sql/functions/builtin.py:
##
@@ -20115,11 +20115,47 @@ def get_json_object(col: "ColumnOrName", path: str)
-> Column:
Examples
---
cloud-fan commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2664519413
If we have this grace handling in 3.5.5, we should have it in 4.0.0 as well,
otherwise it's a breaking changing in 4.0.
--
This is an automated message from the Apache Git Service.
To
LuciferYang commented on code in PR #49988:
URL: https://github.com/apache/spark/pull/49988#discussion_r1959225771
##
.github/workflows/build_and_test.yml:
##
@@ -51,7 +51,7 @@ on:
jobs:
precondition:
name: Check changes
-runs-on: ubuntu-22.04
+runs-on: ubuntu-l
anishshri-db commented on code in PR #49304:
URL: https://github.com/apache/spark/pull/49304#discussion_r1959234122
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##
@@ -656,31 +803,75 @@ class RocksDB(
*
* @note This update is n
anishshri-db commented on code in PR #49304:
URL: https://github.com/apache/spark/pull/49304#discussion_r1959234822
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##
@@ -847,35 +828,47 @@ class RocksDBStateStoreChangeDa
MaxGekk commented on code in PR #49981:
URL: https://github.com/apache/spark/pull/49981#discussion_r1958739470
##
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##
@@ -1160,7 +1160,6 @@ primaryExpression
| primaryExpression DOUBLE_COLON dataT
cloud-fan commented on PR #49678:
URL: https://github.com/apache/spark/pull/49678#issuecomment-2664513906
Does it fix the same issue as https://github.com/apache/spark/pull/48325 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
pan3793 commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1959011315
##
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##
@@ -1049,6 +1049,10 @@ private[spark] class Client(
javaOpts += s"-Djava.n
pan3793 commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1958982565
##
mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:
##
@@ -39,8 +39,11 @@ private[spark] object BLAS extends Serializable {
// For level-3 routines,
pan3793 commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1959011507
##
core/src/main/scala/org/apache/spark/SparkContext.scala:
##
@@ -3436,6 +3437,20 @@ object SparkContext extends Logging {
supplement(DRIVER_JAVA_OPTIONS)
su
fusheng9399 commented on code in PR #49875:
URL: https://github.com/apache/spark/pull/49875#discussion_r1959006832
##
python/pyspark/sql/functions/builtin.py:
##
@@ -20115,11 +20115,47 @@ def get_json_object(col: "ColumnOrName", path: str)
-> Column:
Examples
--
dongjoon-hyun commented on PR #49973:
URL: https://github.com/apache/spark/pull/49973#issuecomment-2664096336
Thank you, @itholic .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
HeartSaVioR commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2664160038
W.r.t. release manager for Spark 3.5.5, either is fine for me. I'm happy to
take the step if you prefer to let me take it.
--
This is an automated message from the Apache Git Servic
zhengruifeng commented on PR #49986:
URL: https://github.com/apache/spark/pull/49986#issuecomment-2664384195
I think this PR needs reviews from @srowen @WeichenXu123 and @luhenry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
zhengruifeng commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1958946765
##
core/src/main/scala/org/apache/spark/SparkContext.scala:
##
@@ -3436,6 +3437,20 @@ object SparkContext extends Logging {
supplement(DRIVER_JAVA_OPTIONS)
zhengruifeng commented on code in PR #49875:
URL: https://github.com/apache/spark/pull/49875#discussion_r1958951388
##
python/pyspark/sql/functions/builtin.py:
##
@@ -20115,11 +20115,47 @@ def get_json_object(col: "ColumnOrName", path: str)
-> Column:
Examples
-
dongjoon-hyun commented on code in PR #49987:
URL: https://github.com/apache/spark/pull/49987#discussion_r1958773406
##
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##
@@ -219,8 +220,8 @@ statement
ADD (COLUMN | COLUMNS)
column
the-sakthi commented on PR #49975:
URL: https://github.com/apache/spark/pull/49975#issuecomment-2664211699
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscri
the-sakthi commented on PR #49953:
URL: https://github.com/apache/spark/pull/49953#issuecomment-2664215062
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscri
the-sakthi commented on PR #49926:
URL: https://github.com/apache/spark/pull/49926#issuecomment-2664217556
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscri
dongjoon-hyun commented on PR #49984:
URL: https://github.com/apache/spark/pull/49984#issuecomment-2664144152
I sent an email for further discussion
- https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd
--
This is an automated message from the Apache Git Service.
To respond
dongjoon-hyun commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2664143940
I sent an email for further discussion
- https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd
--
This is an automated message from the Apache Git Service.
To respond
dongjoon-hyun commented on PR #49985:
URL: https://github.com/apache/spark/pull/49985#issuecomment-2664145186
Thank you for making this PR, @HeartSaVioR .
I sent an email for further discussion.
- https://lists.apache.org/thread/qwxb21g5xjl7xfp4rozqmg1g0ndfw2jd
--
This is an auto
HeartSaVioR commented on PR #49983:
URL: https://github.com/apache/spark/pull/49983#issuecomment-2664154886
@dongjoon-hyun
Let me clarify a bit.
1. I have claimed that the config is not something user (even admin) would
understand what it is and try to flip. That said, removin
srowen commented on code in PR #49986:
URL: https://github.com/apache/spark/pull/49986#discussion_r1958955170
##
core/src/main/scala/org/apache/spark/SparkContext.scala:
##
@@ -3436,6 +3437,20 @@ object SparkContext extends Logging {
supplement(DRIVER_JAVA_OPTIONS)
sup
74 matches
Mail list logo
