[
https://issues.apache.org/jira/browse/SPARK-51674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-51674:
---
Description:
We now have all Classic-only APIs annotated with @ClassicOnly. This tag will be
visibl
[
https://issues.apache.org/jira/browse/SPARK-51674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-51674:
---
Description:
We now have all Classic-only APIs annotated with @ClassicOnly. This tag will be
visibl
Pengfei Xu created SPARK-51674:
--
Summary: Remove unnecessary Spark Connect doc link from Spark
website
Key: SPARK-51674
URL: https://issues.apache.org/jira/browse/SPARK-51674
Project: Spark
Iss
Pengfei Xu created SPARK-51288:
--
Summary: Add link to Scala API doc of Spark Connect
Key: SPARK-51288
URL: https://issues.apache.org/jira/browse/SPARK-51288
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-50846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17924703#comment-17924703
]
Pengfei Xu commented on SPARK-50846:
[~ostronaut] The PR #49111 is merged :D
> Cons
[
https://issues.apache.org/jira/browse/SPARK-50846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17916195#comment-17916195
]
Pengfei Xu edited comment on SPARK-50846 at 1/22/25 9:37 PM:
-
[
https://issues.apache.org/jira/browse/SPARK-50846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17916195#comment-17916195
]
Pengfei Xu commented on SPARK-50846:
Hey [~ostronaut], this task depends on a PR tha
[
https://issues.apache.org/jira/browse/SPARK-50846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50846:
---
Issue Type: Improvement (was: Bug)
> Consolidate Aggregator handling in ColumnNodeToProtoConverter
Pengfei Xu created SPARK-50846:
--
Summary: Consolidate Aggregator handling in
ColumnNodeToProtoConverter
Key: SPARK-50846
URL: https://issues.apache.org/jira/browse/SPARK-50846
Project: Spark
Is
[
https://issues.apache.org/jira/browse/SPARK-49229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17908465#comment-17908465
]
Pengfei Xu commented on SPARK-49229:
Working on this one.
> Deduplicate Scala UDF h
[
https://issues.apache.org/jira/browse/SPARK-49088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17908069#comment-17908069
]
Pengfei Xu commented on SPARK-49088:
I wonder if we should remove it from Unresolved
[
https://issues.apache.org/jira/browse/SPARK-49088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17908049#comment-17908049
]
Pengfei Xu commented on SPARK-49088:
Starting on this one.
> Remove ignoreNulls fro
Pengfei Xu created SPARK-50658:
--
Summary: Move all Connect invocations of system.internal function
to the proper namespace (PySpark)
Key: SPARK-50658
URL: https://issues.apache.org/jira/browse/SPARK-50658
[
https://issues.apache.org/jira/browse/SPARK-49087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17906476#comment-17906476
]
Pengfei Xu commented on SPARK-49087:
Start working on this one.
> Move all Connect
Pengfei Xu created SPARK-50372:
--
Summary: Bugfix: not all DF execute paths notify Observation.get
Key: SPARK-50372
URL: https://issues.apache.org/jira/browse/SPARK-50372
Project: Spark
Issue Typ
[
https://issues.apache.org/jira/browse/SPARK-50366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50366:
---
Description: Follow-up from the discussion in
https://github.com/apache/spark/pull/47815#discussion_
Pengfei Xu created SPARK-50366:
--
Summary: Isolate tags on thread level for SparkSession in Classic
Key: SPARK-50366
URL: https://issues.apache.org/jira/browse/SPARK-50366
Project: Spark
Issue Ty
[
https://issues.apache.org/jira/browse/SPARK-50244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50244:
---
Description:
Problems:
* ADD JAR is now isolated because it's an SQL command.
* We now disable isola
Pengfei Xu created SPARK-50244:
--
Summary: Artifact isolation for Hive
Key: SPARK-50244
URL: https://issues.apache.org/jira/browse/SPARK-50244
Project: Spark
Issue Type: Improvement
Com
Pengfei Xu created SPARK-50243:
--
Summary: Cache classloaders generated by ArtifactManager
Key: SPARK-50243
URL: https://issues.apache.org/jira/browse/SPARK-50243
Project: Spark
Issue Type: Impro
[
https://issues.apache.org/jira/browse/SPARK-50207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894873#comment-17894873
]
Pengfei Xu commented on SPARK-50207:
Duplicate of https://issues.apache.org/jira/bro
[
https://issues.apache.org/jira/browse/SPARK-50207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu resolved SPARK-50207.
Resolution: Duplicate
> Disable `SparkSessionJobTaggingAndCancellationSuite.Cancellation APIs in
[
https://issues.apache.org/jira/browse/SPARK-50205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50205:
---
Priority: Critical (was: Blocker)
> Re-enable `SparkSessionJobTaggingAndCancellationSuite.Cancellat
[
https://issues.apache.org/jira/browse/SPARK-50205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50205:
---
Description: This test case needs a refactor to use only 2 threads instead
of 3, because having 3 th
[
https://issues.apache.org/jira/browse/SPARK-50205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50205:
---
Description:
https://github.com/apache/spark/actions/runs/10915451051/job/30295259985
This test case
[
https://issues.apache.org/jira/browse/SPARK-50207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50207:
---
Description:
* [https://github.com/apache/spark/actions/runs/10915451051/job/30295259985]
{code:jav
Pengfei Xu created SPARK-50207:
--
Summary: Disable
`SparkSessionJobTaggingAndCancellationSuite.Cancellation APIs in SparkSession
are isolated`
Key: SPARK-50207
URL: https://issues.apache.org/jira/browse/SPARK-50207
Pengfei Xu created SPARK-50205:
--
Summary: CLONE - Re-enable
`SparkSessionJobTaggingAndCancellationSuite.Cancellation APIs in SparkSession
are isolated`
Key: SPARK-50205
URL: https://issues.apache.org/jira/browse/SPA
[
https://issues.apache.org/jira/browse/SPARK-50205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-50205:
---
Summary: Re-enable `SparkSessionJobTaggingAndCancellationSuite.Cancellation
APIs in SparkSession are
Pengfei Xu created SPARK-49854:
--
Summary: Clone artifact manager during sparkSession.cloneSession
Key: SPARK-49854
URL: https://issues.apache.org/jira/browse/SPARK-49854
Project: Spark
Issue Typ
Pengfei Xu created SPARK-49758:
--
Summary: Allow adding Scala UDFs in UDFRegistration
Key: SPARK-49758
URL: https://issues.apache.org/jira/browse/SPARK-49758
Project: Spark
Issue Type: Task
[
https://issues.apache.org/jira/browse/SPARK-49249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu reopened SPARK-49249:
> Implement new Connect APIs in SQL Core
> --
>
>
[
https://issues.apache.org/jira/browse/SPARK-49249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu reopened SPARK-49249:
> Implement new Connect APIs in SQL Core
> --
>
>
[
https://issues.apache.org/jira/browse/SPARK-49249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu reopened SPARK-49249:
> Implement new Connect APIs in SQL Core
> --
>
>
Pengfei Xu created SPARK-49337:
--
Summary: Add tag-related APIs in Connect back to PySpark
Key: SPARK-49337
URL: https://issues.apache.org/jira/browse/SPARK-49337
Project: Spark
Issue Type: Bug
Pengfei Xu created SPARK-49320:
--
Summary: Add tag-related APs to Spark Core
Key: SPARK-49320
URL: https://issues.apache.org/jira/browse/SPARK-49320
Project: Spark
Issue Type: Bug
Compo
[
https://issues.apache.org/jira/browse/SPARK-49096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49096:
---
Summary: (Tentitavely) Investigate Mima bug when checking Dataset APIs
(was: Investgate Mima bug wh
Pengfei Xu created SPARK-49249:
--
Summary: Implement new Connect APIs in SQL Core
Key: SPARK-49249
URL: https://issues.apache.org/jira/browse/SPARK-49249
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-49060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49060:
---
Parent: SPARK-49248
Issue Type: Sub-task (was: New Feature)
> Clean up Mima rules for sql<>
[
https://issues.apache.org/jira/browse/SPARK-49096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49096:
---
Parent: SPARK-49248
Issue Type: Sub-task (was: New Feature)
> Investgate Mima bug when chec
[
https://issues.apache.org/jira/browse/SPARK-49096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49096:
---
Parent: (was: SPARK-49060)
Issue Type: New Feature (was: Sub-task)
> Investgate Mima bu
[
https://issues.apache.org/jira/browse/SPARK-48683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-48683:
---
Parent: SPARK-49248
Issue Type: Sub-task (was: New Feature)
> Schema evolution with `df.mer
[
https://issues.apache.org/jira/browse/SPARK-48714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-48714:
---
Parent: SPARK-49248
Issue Type: Sub-task (was: New Feature)
> Implement df.mergeInto in PyS
[
https://issues.apache.org/jira/browse/SPARK-48510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-48510:
---
Parent: SPARK-49248
Issue Type: Sub-task (was: New Feature)
> Support UDAF.toColumn API in
[
https://issues.apache.org/jira/browse/SPARK-48008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-48008:
---
Parent: SPARK-49248
Issue Type: Sub-task (was: New Feature)
> Support UDAF in Spark Connect
[
https://issues.apache.org/jira/browse/SPARK-47545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-47545:
---
Parent: SPARK-49248
Issue Type: Sub-task (was: New Feature)
> [Connect] DF observe support
[
https://issues.apache.org/jira/browse/SPARK-49248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49248:
---
Issue Type: Epic (was: Umbrella)
> Scala Client Parity with existing Dataset/DataFrame API
> --
[
https://issues.apache.org/jira/browse/SPARK-49248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49248:
---
Issue Type: Umbrella (was: Epic)
> Scala Client Parity with existing Dataset/DataFrame API
> --
[
https://issues.apache.org/jira/browse/SPARK-49248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49248:
---
Description: Implement missing APIs in the Scala Client to reach full
parity with the existing Datas
Pengfei Xu created SPARK-49248:
--
Summary: Implement missing APIs in the Scala Client to reach full
parity with the existing Dataset interface.
Key: SPARK-49248
URL: https://issues.apache.org/jira/browse/SPARK-49248
[
https://issues.apache.org/jira/browse/SPARK-49122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49122:
---
Summary: Add addArtifact API to the Spark SQL Core (was: Add addArtifact
API to the Spark Connect S
Pengfei Xu created SPARK-49096:
--
Summary: Investgate Mima bug when checking Dataset APIs
Key: SPARK-49096
URL: https://issues.apache.org/jira/browse/SPARK-49096
Project: Spark
Issue Type: Sub-ta
[
https://issues.apache.org/jira/browse/SPARK-49096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-49096:
---
Description:
Mima check might complain the following Dataset rules does not filter any
problem.
Thi
Pengfei Xu created SPARK-49060:
--
Summary: Clean up Mima rules for sql<>connect compatibility checks
Key: SPARK-49060
URL: https://issues.apache.org/jira/browse/SPARK-49060
Project: Spark
Issue T
[
https://issues.apache.org/jira/browse/SPARK-45155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-45155:
---
Fix Version/s: 4.0.0
> Add API Docs
>
>
> Key: SPARK-45155
>
Pengfei Xu created SPARK-48714:
--
Summary: Implement df.mergeInto in PySpark
Key: SPARK-48714
URL: https://issues.apache.org/jira/browse/SPARK-48714
Project: Spark
Issue Type: New Feature
[
https://issues.apache.org/jira/browse/SPARK-48683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-48683:
---
Component/s: SQL
(was: Spark Core)
> Schema evolution with `df.mergeInto` losin
Pengfei Xu created SPARK-48683:
--
Summary: Schema evolution with `df.mergeInto` losing `when` clauses
Key: SPARK-48683
URL: https://issues.apache.org/jira/browse/SPARK-48683
Project: Spark
Issue
Pengfei Xu created SPARK-48510:
--
Summary: Support UDAF.toColumn API in Spark Connect
Key: SPARK-48510
URL: https://issues.apache.org/jira/browse/SPARK-48510
Project: Spark
Issue Type: New Featur
[
https://issues.apache.org/jira/browse/SPARK-48008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengfei Xu updated SPARK-48008:
---
Description:
Currently Spark Connect supports only UDFs. We need to add support for UDAFs,
specific
Pengfei Xu created SPARK-47545:
--
Summary: [Connect] DF observe support for the scala client
Key: SPARK-47545
URL: https://issues.apache.org/jira/browse/SPARK-47545
Project: Spark
Issue Type: New
61 matches
Mail list logo
