grundprinzip commented on PR #79:
URL: https://github.com/apache/spark-connect-go/pull/79#issuecomment-2566980302
merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific co
grundprinzip closed pull request #79: Add column.getItem to Columns
URL: https://github.com/apache/spark-connect-go/pull/79
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
grundprinzip opened a new pull request, #101:
URL: https://github.com/apache/spark-connect-go/pull/101
### What changes were proposed in this pull request?
Support for `df.ApproxQuantile()` and the aggregated helper `df.Stat().*`
### Why are the changes needed?
Compatibility
vicennial commented on PR #49342:
URL: https://github.com/apache/spark/pull/49342#issuecomment-2566983359
PTAL @hvanhovell / @HyukjinKwon / @xupefei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
vicennial commented on code in PR #49341:
URL: https://github.com/apache/spark/pull/49341#discussion_r1900387249
##
sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala:
##
@@ -447,6 +453,12 @@ class ArtifactManager(session: SparkSession) extends
Logging
grundprinzip opened a new pull request, #100:
URL: https://github.com/apache/spark-connect-go/pull/100
### What changes were proposed in this pull request?
Add support for `df.Agg() and `df.AggWithMap()`.
### Why are the changes needed?
Compatibility
### Does this PR intro
grundprinzip commented on PR #102:
URL: https://github.com/apache/spark-connect-go/pull/102#issuecomment-2567152641
Merging to master to fix broken CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
grundprinzip closed pull request #102: Fix broken build
URL: https://github.com/apache/spark-connect-go/pull/102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe,
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900510562
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
HyukjinKwon commented on PR #49260:
URL: https://github.com/apache/spark/pull/49260#issuecomment-2567245039
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
zhengruifeng commented on code in PR #49338:
URL: https://github.com/apache/spark/pull/49338#discussion_r1900491893
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15341,13 +15341,32 @@ def regexp_count(str: "ColumnOrName", regexp:
"ColumnOrName") -> Column:
Examples
cloud-fan commented on code in PR #49126:
URL: https://github.com/apache/spark/pull/49126#discussion_r1900531045
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CreateSQLFunctionCommand.scala:
##
@@ -72,8 +85,312 @@ case class CreateSQLFunctionCommand(
i
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900532615
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
zhengruifeng opened a new pull request, #49343:
URL: https://github.com/apache/spark/pull/49343
### What changes were proposed in this pull request?
Add a daily build for Pandas API on Spark with old dependencies
### Why are the changes needed?
The PS part requires a newe
cloud-fan commented on code in PR #49310:
URL: https://github.com/apache/spark/pull/49310#discussion_r1900532721
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##
@@ -68,9 +68,15 @@ object RewriteWithExpression extends Rule[Lo
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900532662
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
zhengruifeng opened a new pull request, #49344:
URL: https://github.com/apache/spark/pull/49344
### What changes were proposed in this pull request?
Upgrade the minimum version of Pandas in PS to 2.2.0
But the version used in `require_minimum_pandas_version` is still 2.2.0,
cloud-fan commented on PR #49310:
URL: https://github.com/apache/spark/pull/49310#issuecomment-2567279038
The Spark Connect test failure is unrelated, thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
cloud-fan closed pull request #49310: [SPARK-50683][SQL] Inline the common
expression in With if used once
URL: https://github.com/apache/spark/pull/49310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to g
cloud-fan commented on PR #49234:
URL: https://github.com/apache/spark/pull/49234#issuecomment-2567279749
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific c
cloud-fan closed pull request #49234: [SPARK-50614][SQL] Add Variant shredding
support for Parquet
URL: https://github.com/apache/spark/pull/49234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
cloud-fan commented on code in PR #49103:
URL: https://github.com/apache/spark/pull/49103#discussion_r1900547208
##
sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala:
##
@@ -127,6 +128,17 @@ case object StringType
}
}
+/**
+ * String type that was the resu
cloud-fan commented on code in PR #49139:
URL: https://github.com/apache/spark/pull/49139#discussion_r1900549047
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:
##
@@ -51,6 +51,48 @@ import org.apache.spark.sql.types._
import org.apache.spa
cloud-fan commented on code in PR #49103:
URL: https://github.com/apache/spark/pull/49103#discussion_r1900545499
##
common/utils/src/main/resources/error/error-conditions.json:
##
@@ -1920,7 +1920,25 @@
},
"INDETERMINATE_COLLATION" : {
"message" : [
- "Function c
cloud-fan commented on code in PR #49103:
URL: https://github.com/apache/spark/pull/49103#discussion_r1900548063
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala:
##
@@ -38,10 +39,10 @@ object CollationTypeCoercion {
}
def
cloud-fan commented on code in PR #49139:
URL: https://github.com/apache/spark/pull/49139#discussion_r1900549538
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##
@@ -691,6 +691,20 @@ object DescribeRelation {
def getOutputAttrs:
cloud-fan commented on code in PR #49139:
URL: https://github.com/apache/spark/pull/49139#discussion_r1900549648
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:
##
@@ -746,6 +750,261 @@ case class DescribeTableCommand(
}
}
+/**
+ * Command t
cloud-fan commented on code in PR #49139:
URL: https://github.com/apache/spark/pull/49139#discussion_r1900552760
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:
##
@@ -746,6 +750,261 @@ case class DescribeTableCommand(
}
}
+/**
+ * Command t
cloud-fan commented on code in PR #49139:
URL: https://github.com/apache/spark/pull/49139#discussion_r1900553086
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:
##
@@ -746,6 +750,261 @@ case class DescribeTableCommand(
}
}
+/**
+ * Command t
cloud-fan commented on code in PR #49139:
URL: https://github.com/apache/spark/pull/49139#discussion_r1900553337
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:
##
@@ -746,6 +750,261 @@ case class DescribeTableCommand(
}
}
+/**
+ * Command t
cloud-fan commented on PR #49202:
URL: https://github.com/apache/spark/pull/49202#issuecomment-2567379857
After more thought, I think we should think of filter pushdown in a
different way. Once you push a predicate through `Project` and expand the
attribute reference into an expensive expre
ueshin commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900610142
##
python/pyspark/sql/table_arg.py:
##
@@ -0,0 +1,52 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See
HyukjinKwon commented on code in PR #49341:
URL: https://github.com/apache/spark/pull/49341#discussion_r1900507294
##
sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala:
##
@@ -447,6 +453,12 @@ class ArtifactManager(session: SparkSession) extends
Loggin
drexler-sky commented on code in PR #49338:
URL: https://github.com/apache/spark/pull/49338#discussion_r1900507521
##
python/pyspark/sql/functions/builtin.py:
##
@@ -15341,13 +15341,32 @@ def regexp_count(str: "ColumnOrName", regexp:
"ColumnOrName") -> Column:
Examples
HyukjinKwon commented on code in PR #49341:
URL: https://github.com/apache/spark/pull/49341#discussion_r1900507576
##
sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala:
##
@@ -447,6 +453,12 @@ class ArtifactManager(session: SparkSession) extends
Loggin
cloud-fan commented on code in PR #49126:
URL: https://github.com/apache/spark/pull/49126#discussion_r1900523254
##
sql/core/src/main/scala/org/apache/spark/sql/catalyst/catalog/SQLFunction.scala:
##
@@ -92,4 +137,42 @@ object SQLFunction {
}
}
}
+
+ def isSQLFun
cloud-fan commented on PR #49202:
URL: https://github.com/apache/spark/pull/49202#issuecomment-2567333471
This LGTM overall, my last suggestion is to narrow the scope of the `With`
expression generated during filter pushdown. We should rewrite the `With`
expression immediately at the end of
zml1206 commented on PR #49202:
URL: https://github.com/apache/spark/pull/49202#issuecomment-2567340459
> My idea is, when rewriting `With` inside `Filter`, and the generated
`Project` is the same as the `Project` above `Filter`, we remove the above
`Project`.
There is a problem in t
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900580802
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900509261
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900509861
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900509475
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900564862
##
sql/core/src/main/scala/org/apache/spark/sql/TableArg.scala:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ *
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900565391
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
HyukjinKwon closed pull request #49260: [SPARK-50642][CONNECT][SS]Fix the state
schema for FlatMapGroupsWithState in spark connect when there is no initial
state
URL: https://github.com/apache/spark/pull/49260
--
This is an automated message from the Apache Git Service.
To respond to the mes
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900508839
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,71 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
cloud-fan commented on code in PR #49126:
URL: https://github.com/apache/spark/pull/49126#discussion_r1900521793
##
sql/api/src/main/scala/org/apache/spark/sql/types/StructField.scala:
##
@@ -147,6 +147,17 @@ case class StructField(
if (metadata.contains("comment")) Option(
zml1206 commented on code in PR #49310:
URL: https://github.com/apache/spark/pull/49310#discussion_r1900523989
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##
@@ -68,9 +68,15 @@ object RewriteWithExpression extends Rule[Logi
cloud-fan commented on code in PR #49126:
URL: https://github.com/apache/spark/pull/49126#discussion_r1900525129
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CreateSQLFunctionCommand.scala:
##
@@ -72,8 +85,312 @@ case class CreateSQLFunctionCommand(
i
cloud-fan commented on code in PR #49126:
URL: https://github.com/apache/spark/pull/49126#discussion_r1900525249
##
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CreateSQLFunctionCommand.scala:
##
@@ -72,8 +85,312 @@ case class CreateSQLFunctionCommand(
i
zhengruifeng closed pull request #49338: [SPARK-50702][PYTHON] Refine the
docstring of regexp_count, regexp_extract and regexp_extract_all
URL: https://github.com/apache/spark/pull/49338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to G
zhengruifeng commented on PR #49338:
URL: https://github.com/apache/spark/pull/49338#issuecomment-2567270204
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
cloud-fan commented on code in PR #49126:
URL: https://github.com/apache/spark/pull/49126#discussion_r1900529242
##
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala:
##
@@ -828,24 +893,90 @@ class DDLParserSuite extends AnalysisTest with
Share
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900589155
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900589541
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900590578
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900589155
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900590578
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
zml1206 commented on PR #49202:
URL: https://github.com/apache/spark/pull/49202#issuecomment-2567358874
> My idea is, when rewriting `With` inside `Filter`, and the generated
`Project` is the same as the `Project` above `Filter`, we remove the above
`Project`.
There is a problem. The
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900590578
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
xinrong-meng commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900590578
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return f
ueshin commented on code in PR #49055:
URL: https://github.com/apache/spark/pull/49055#discussion_r1900594054
##
python/pyspark/sql/tests/test_udtf.py:
##
@@ -1064,6 +1074,73 @@ def eval(self, row: Row):
func = udtf(TestUDTF, returnType="a: int")
return func
github-actions[bot] commented on PR #48202:
URL: https://github.com/apache/spark/pull/48202#issuecomment-2567197405
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #48133:
URL: https://github.com/apache/spark/pull/48133#issuecomment-2567197414
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] closed pull request #48018: [SPARK-29474] [CORE] [WIP] CLI
support for Spark-on-Docker-on-Yarn
URL: https://github.com/apache/spark/pull/48018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL ab
vicennial opened a new pull request, #49342:
URL: https://github.com/apache/spark/pull/49342
### What changes were proposed in this pull request?
Adds a new boolean `allow_reconnect` field to `ReleaseSessionRequest`.
When set to `true` in the request, the server will not pla
grundprinzip opened a new pull request, #102:
URL: https://github.com/apache/spark-connect-go/pull/102
### What changes were proposed in this pull request?
Fix a broken build from stale CI.
### Why are the changes needed?
Fix
### Does this PR introduce _any_ user-facing ch
67 matches
Mail list logo
