[jira] [Updated] (SPARK-50828) Deprecate pyspark.ml.connect

[ https://issues.apache.org/jira/browse/SPARK-50828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-50828: -- Fix Version/s: 4.0.0 (was: 4.1.0) > Deprecate pyspark.ml.connect >

[jira] [Resolved] (SPARK-50828) Deprecate pyspark.ml.connect

[ https://issues.apache.org/jira/browse/SPARK-50828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-50828. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49504 [https://

[jira] [Commented] (SPARK-49878) Spark Structured Streaming will process incorrect Kafka offsets

[ https://issues.apache.org/jira/browse/SPARK-49878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913318#comment-17913318 ] Dmytro Tsyliuryk commented on SPARK-49878: -- Hi [~tjshippey]. I've done a bit of

[jira] [Assigned] (SPARK-50822) Setting version to 4.1.0-SNAPSHOT

[ https://issues.apache.org/jira/browse/SPARK-50822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-50822: --- Assignee: Dongjoon Hyun > Setting version to 4.1.0-SNAPSHOT > -

[jira] [Resolved] (SPARK-50822) Setting version to 4.1.0-SNAPSHOT

[ https://issues.apache.org/jira/browse/SPARK-50822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50822. - Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49495 [https://gith

[jira] [Commented] (SPARK-50816) 3.5.4 Spark Apache release is not verifiable

[ https://issues.apache.org/jira/browse/SPARK-50816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913238#comment-17913238 ] Yang Jie commented on SPARK-50816: -- Thank you, [~dongjoon]  and [~vrozov] . I'm sorry t

[jira] [Updated] (SPARK-50814) Remove unused SQL error documentation pages

[ https://issues.apache.org/jira/browse/SPARK-50814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50814: -- Fix Version/s: 4.0.0 (was: 4.1.0) > Remove unused SQL error documentati

[jira] [Created] (SPARK-50830) Add a flag to guard returning of the single-pass result in the dual run analysis

Mihailo Aleksic created SPARK-50830: --- Summary: Add a flag to guard returning of the single-pass result in the dual run analysis Key: SPARK-50830 URL: https://issues.apache.org/jira/browse/SPARK-50830

[jira] [Resolved] (SPARK-50522) Support for indeterminate collation

[ https://issues.apache.org/jira/browse/SPARK-50522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50522. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49103 [https://gith

[jira] [Created] (SPARK-50827) Support plugin for Spark Connect ML

Bobby Wang created SPARK-50827: -- Summary: Support plugin for Spark Connect ML Key: SPARK-50827 URL: https://issues.apache.org/jira/browse/SPARK-50827 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-50815) Fix bug where passing null Variants in createDataFrame causes it to fail

[ https://issues.apache.org/jira/browse/SPARK-50815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-50815: --- Assignee: Harsh Motwani > Fix bug where passing null Variants in createDataFrame causes it

[jira] [Resolved] (SPARK-50815) Fix bug where passing null Variants in createDataFrame causes it to fail

[ https://issues.apache.org/jira/browse/SPARK-50815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50815. - Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49487 [https://gith

[jira] [Updated] (SPARK-50815) Fix bug where passing null Variants in createDataFrame causes it to fail

[ https://issues.apache.org/jira/browse/SPARK-50815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-50815: Fix Version/s: 4.0.0 (was: 4.1.0) > Fix bug where passing null Variants in

[jira] [Updated] (SPARK-50827) Support plugin for Spark Connect ML

[ https://issues.apache.org/jira/browse/SPARK-50827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50827: --- Labels: pull-request-available (was: ) > Support plugin for Spark Connect ML >

[jira] [Updated] (SPARK-50839) Spark Connect createDataFrame behavior does not match with classic Spark

[ https://issues.apache.org/jira/browse/SPARK-50839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh Motwani updated SPARK-50839: -- Description: Classic Spark and Spark Connect behave differently when `List(Integer)` is passe

[jira] [Assigned] (SPARK-50819) Refactor Spark profiler module

[ https://issues.apache.org/jira/browse/SPARK-50819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-50819: - Assignee: Cheng Pan > Refactor Spark profiler module > -- >

[jira] [Comment Edited] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913501#comment-17913501 ] Asif edited comment on SPARK-50840 at 1/16/25 12:50 AM: The belo

[jira] [Commented] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913501#comment-17913501 ] Asif commented on SPARK-50840: -- The following test fails. {quote} test("SPARK-50840: Hive t

[jira] [Comment Edited] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913501#comment-17913501 ] Asif edited comment on SPARK-50840 at 1/16/25 12:48 AM: test("SP

[jira] [Comment Edited] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913501#comment-17913501 ] Asif edited comment on SPARK-50840 at 1/16/25 12:49 AM: test("SP

[jira] [Created] (SPARK-50841) Move pyspark.ml.remote into pyspark.ml.connect

Bobby Wang created SPARK-50841: -- Summary: Move pyspark.ml.remote into pyspark.ml.connect Key: SPARK-50841 URL: https://issues.apache.org/jira/browse/SPARK-50841 Project: Spark Issue Type: Sub-ta

[jira] [Updated] (SPARK-50812) Support pyspark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-50812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-50812: -- Component/s: ML > Support pyspark.ml on Connect > - > >

[jira] [Updated] (SPARK-50838) Implement checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with Unionloop cases.

[ https://issues.apache.org/jira/browse/SPARK-50838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50838: --- Labels: pull-request-available (was: ) > Implement checkRecursion to check if all the rules

[jira] [Updated] (SPARK-50812) Support pyspark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-50812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-50812: -- Affects Version/s: 4.0 4.1 (was: 4.0.0) > Su

[jira] [Resolved] (SPARK-50756) Use error class in SparkConf.validateSettings

[ https://issues.apache.org/jira/browse/SPARK-50756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-50756. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49491 [https://gi

[jira] [Created] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

Asif created SPARK-50840: Summary: TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ Key: SPARK-50840 URL: https://issues.apache.org/jira/browse/SPARK-50

[jira] [Comment Edited] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913501#comment-17913501 ] Asif edited comment on SPARK-50840 at 1/16/25 12:52 AM: The belo

[jira] [Comment Edited] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913501#comment-17913501 ] Asif edited comment on SPARK-50840 at 1/16/25 12:52 AM: The belo

[jira] [Comment Edited] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913501#comment-17913501 ] Asif edited comment on SPARK-50840 at 1/16/25 12:53 AM: The belo

[jira] [Updated] (SPARK-50840) TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ

[ https://issues.apache.org/jira/browse/SPARK-50840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50840: --- Labels: pull-request-available spark-sql (was: spark-sql) > TimestampType wrongly gets mapp

[jira] [Resolved] (SPARK-50839) Spark Connect createDataFrame behavior does not match with classic Spark

[ https://issues.apache.org/jira/browse/SPARK-50839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-50839. -- Resolution: Invalid > Spark Connect createDataFrame behavior does not match with classic Spark

[jira] [Commented] (SPARK-50839) Spark Connect createDataFrame behavior does not match with classic Spark

[ https://issues.apache.org/jira/browse/SPARK-50839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913531#comment-17913531 ] Hyukjin Kwon commented on SPARK-50839: -- Actually I think we should just leave this

[jira] [Resolved] (SPARK-50819) Refactor Spark profiler module

[ https://issues.apache.org/jira/browse/SPARK-50819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50819. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49492 [https://

[jira] [Assigned] (SPARK-50817) Fix rocksdb error reporting

[ https://issues.apache.org/jira/browse/SPARK-50817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-50817: - Assignee: Wei Liu > Fix rocksdb error reporting > > >

[jira] [Resolved] (SPARK-50817) Fix rocksdb error reporting

[ https://issues.apache.org/jira/browse/SPARK-50817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50817. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49489 [https://

[jira] [Resolved] (SPARK-50804) to_protobuf() should not throw MatchError

[ https://issues.apache.org/jira/browse/SPARK-50804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-50804. -- Resolution: Fixed Issue resolved by pull request 49520 [https://github.com/apache/spark/pull/4

[jira] [Assigned] (SPARK-50804) to_protobuf() should not throw MatchError

[ https://issues.apache.org/jira/browse/SPARK-50804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-50804: Assignee: Hyukjin Kwon > to_protobuf() should not throw MatchError >

[jira] [Assigned] (SPARK-49886) Query Level Correctness verification

[ https://issues.apache.org/jira/browse/SPARK-49886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-49886: Assignee: Wei Liu > Query Level Correctness verification > --

[jira] [Resolved] (SPARK-49886) Query Level Correctness verification

[ https://issues.apache.org/jira/browse/SPARK-49886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-49886. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48460 [https://gi

[jira] [Updated] (SPARK-49886) Query Level Correctness verification

[ https://issues.apache.org/jira/browse/SPARK-49886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-49886: - Component/s: Structured Streaming (was: SS) > Query Level Correctness verif

[jira] [Assigned] (SPARK-50654) CommitMetadata should set stateUniqueIds to None in V1

[ https://issues.apache.org/jira/browse/SPARK-50654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-50654: Assignee: Wei Liu > CommitMetadata should set stateUniqueIds to None in V1 >

[jira] [Resolved] (SPARK-50654) CommitMetadata should set stateUniqueIds to None in V1

[ https://issues.apache.org/jira/browse/SPARK-50654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-50654. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49278 [https://gi

[jira] [Updated] (SPARK-50838) Implement checkRecursion to check if all the rules about recursive queries are fulfilled. Adjust optimizer with Unionloop cases.

[ https://issues.apache.org/jira/browse/SPARK-50838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Cupac updated SPARK-50838: Summary: Implement checkRecursion to check if all the rules about recursive queries are fulfilled.

[jira] [Resolved] (SPARK-50811) Support enabling JVM profiler on driver

[ https://issues.apache.org/jira/browse/SPARK-50811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50811. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49483 [https://

[jira] [Updated] (SPARK-49872) Spark History UI -- StreamConstraintsException: String length (20054016) exceeds the maximum length (20000000)

[ https://issues.apache.org/jira/browse/SPARK-49872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-49872: - Affects Version/s: 3.5.4 > Spark History UI -- StreamConstraintsException: String length (20054016) > e

[jira] [Assigned] (SPARK-50811) Support enabling JVM profiler on driver

[ https://issues.apache.org/jira/browse/SPARK-50811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-50811: - Assignee: Cheng Pan > Support enabling JVM profiler on driver > ---

[jira] [Updated] (SPARK-50811) Support enabling JVM profiler on driver

[ https://issues.apache.org/jira/browse/SPARK-50811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50811: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Support enabling JVM pr

[jira] [Created] (SPARK-50843) Support access submodel in TreeEnsembleModel

Ruifeng Zheng created SPARK-50843: - Summary: Support access submodel in TreeEnsembleModel Key: SPARK-50843 URL: https://issues.apache.org/jira/browse/SPARK-50843 Project: Spark Issue Type: Su

[jira] [Updated] (SPARK-50843) Support access submodel in TreeEnsembleModel

[ https://issues.apache.org/jira/browse/SPARK-50843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-50843: -- Description:   GBTmodel.trees fails with   {code:java} java.lang.UnsupportedOperationExcepti

[jira] [Updated] (SPARK-50843) Support access submodel in TreeEnsembleModel

[ https://issues.apache.org/jira/browse/SPARK-50843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-50843: -- Description: {code:java} java.lang.UnsupportedOperationExceptionat org.apache.spark.sql.c

[jira] [Resolved] (SPARK-48199) [Arbitrary State Support] Allow users to set EventTime timestamp properly in TransformWithState for watermark evaluation

[ https://issues.apache.org/jira/browse/SPARK-48199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anish Shrigondekar resolved SPARK-48199. Resolution: Fixed > [Arbitrary State Support] Allow users to set EventTime timesta

[jira] [Created] (SPARK-50842) Replace Murmur3_x86_32 with Murmur_x64_32

Yibo Cai created SPARK-50842: Summary: Replace Murmur3_x86_32 with Murmur_x64_32 Key: SPARK-50842 URL: https://issues.apache.org/jira/browse/SPARK-50842 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-47047) Add support for transformWithState operator with state data source reader

[ https://issues.apache.org/jira/browse/SPARK-47047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anish Shrigondekar resolved SPARK-47047. Resolution: Fixed > Add support for transformWithState operator with state data so

[jira] [Resolved] (SPARK-49212) [PySpark] Implement schema evolution support

[ https://issues.apache.org/jira/browse/SPARK-49212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anish Shrigondekar resolved SPARK-49212. Resolution: Fixed > [PySpark] Implement schema evolution support > ---

[jira] [Resolved] (SPARK-50792) Format binary data as a binary literal in JDBC.

[ https://issues.apache.org/jira/browse/SPARK-50792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50792. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49452 [https://gith

[jira] [Commented] (SPARK-50816) 3.5.4 Spark Apache release is not verifiable

[ https://issues.apache.org/jira/browse/SPARK-50816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913439#comment-17913439 ] Dongjoon Hyun commented on SPARK-50816: --- I removed it from `dist/release`. ASF Dow

[jira] [Updated] (SPARK-50816) 3.5.4 Spark Apache release is not verifiable

[ https://issues.apache.org/jira/browse/SPARK-50816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50816: -- Attachment: Screenshot 2025-01-15 at 12.00.58.png > 3.5.4 Spark Apache release is not verifiab

[jira] [Created] (SPARK-50831) Enable trimming collation by default

Stevo Mitric created SPARK-50831: Summary: Enable trimming collation by default Key: SPARK-50831 URL: https://issues.apache.org/jira/browse/SPARK-50831 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-50831) Enable trimming collation by default

[ https://issues.apache.org/jira/browse/SPARK-50831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50831: --- Labels: pull-request-available (was: ) > Enable trimming collation by default > ---

[jira] [Created] (SPARK-50836) Add non-ANSI SQL Daily CI to branch-4.0

Dongjoon Hyun created SPARK-50836: - Summary: Add non-ANSI SQL Daily CI to branch-4.0 Key: SPARK-50836 URL: https://issues.apache.org/jira/browse/SPARK-50836 Project: Spark Issue Type: Sub-tas

[jira] [Updated] (SPARK-50836) Add non-ANSI SQL Daily CI for branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50836: --- Labels: pull-request-available (was: ) > Add non-ANSI SQL Daily CI for branch-4.0 > ---

[jira] [Updated] (SPARK-50836) Add non-ANSI SQL Daily CI for branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50836: -- Summary: Add non-ANSI SQL Daily CI for branch-4.0 (was: Add non-ANSI SQL Daily CI to branch-4

[jira] [Updated] (SPARK-50830) Return single-pass result as the dual run analysis result

[ https://issues.apache.org/jira/browse/SPARK-50830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50830: --- Labels: pull-request-available (was: ) > Return single-pass result as the dual run analysis

[jira] [Updated] (SPARK-50830) Add a flag to guard returning of the single-pass result in the dual run analysis

[ https://issues.apache.org/jira/browse/SPARK-50830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihailo Aleksic updated SPARK-50830: Description: Right now fixed-point resolved plan is return as a result of analysis of the

[jira] [Created] (SPARK-50832) Add GitHub Action jobs for branch-4.0

Dongjoon Hyun created SPARK-50832: - Summary: Add GitHub Action jobs for branch-4.0 Key: SPARK-50832 URL: https://issues.apache.org/jira/browse/SPARK-50832 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-50832) Add GitHub Action jobs for branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50832: --- Labels: pull-request-available (was: ) > Add GitHub Action jobs for branch-4.0 > --

[jira] [Updated] (SPARK-50833) Update `buf` GitHub Action job to compare against branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50833: -- Summary: Update `buf` GitHub Action job to compare against branch-4.0 (was: Upgrade `buf` Git

[jira] [Updated] (SPARK-50833) Upgrade `buf` GitHub Action job to compare against branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50833: --- Labels: pull-request-available (was: ) > Upgrade `buf` GitHub Action job to compare against

[jira] [Created] (SPARK-50834) Add Daily Github Action CI to `branch-4.0`

Dongjoon Hyun created SPARK-50834: - Summary: Add Daily Github Action CI to `branch-4.0` Key: SPARK-50834 URL: https://issues.apache.org/jira/browse/SPARK-50834 Project: Spark Issue Type: Sub-

[jira] [Created] (SPARK-50835) Add `Python-only` Daily CI to branch-4.0

Dongjoon Hyun created SPARK-50835: - Summary: Add `Python-only` Daily CI to branch-4.0 Key: SPARK-50835 URL: https://issues.apache.org/jira/browse/SPARK-50835 Project: Spark Issue Type: Sub-ta

[jira] [Updated] (SPARK-50835) Add `Python-only` Daily CI to branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50835: --- Labels: pull-request-available (was: ) > Add `Python-only` Daily CI to branch-4.0 > ---

[jira] [Updated] (SPARK-50830) Return single-pass result as the dual run analysis result

[ https://issues.apache.org/jira/browse/SPARK-50830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihailo Aleksic updated SPARK-50830: Summary: Return single-pass result as the dual run analysis result (was: Add a flag to gu

[jira] [Created] (SPARK-50828) Deprecate pyspark.ml.connect

Ruifeng Zheng created SPARK-50828: - Summary: Deprecate pyspark.ml.connect Key: SPARK-50828 URL: https://issues.apache.org/jira/browse/SPARK-50828 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-50828) Deprecate pyspark.ml.connect

[ https://issues.apache.org/jira/browse/SPARK-50828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50828: --- Labels: pull-request-available (was: ) > Deprecate pyspark.ml.connect > ---

[jira] [Commented] (SPARK-50816) 3.5.4 Spark Apache release is not verifiable

[ https://issues.apache.org/jira/browse/SPARK-50816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913245#comment-17913245 ] Kent Yao commented on SPARK-50816: -- Since 3.5.4 is ready, please remove 3.5.3 from the

[jira] [Created] (SPARK-50829) Add flag to disable session collation by default

Stefan Kandic created SPARK-50829: - Summary: Add flag to disable session collation by default Key: SPARK-50829 URL: https://issues.apache.org/jira/browse/SPARK-50829 Project: Spark Issue Type

[jira] [Updated] (SPARK-50829) Add flag to disable session collation by default

[ https://issues.apache.org/jira/browse/SPARK-50829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50829: --- Labels: pull-request-available (was: ) > Add flag to disable session collation by default >

[jira] [Created] (SPARK-50833) Upgrade `buf` GitHub Action job to compare against branch-4.0

Dongjoon Hyun created SPARK-50833: - Summary: Upgrade `buf` GitHub Action job to compare against branch-4.0 Key: SPARK-50833 URL: https://issues.apache.org/jira/browse/SPARK-50833 Project: Spark

[jira] [Commented] (SPARK-50816) 3.5.4 Spark Apache release is not verifiable

[ https://issues.apache.org/jira/browse/SPARK-50816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913407#comment-17913407 ] Vlad Rozov commented on SPARK-50816: [~yao]  this JIRA is closed. I think it will be

[jira] [Resolved] (SPARK-50832) Add GitHub Action jobs for branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50832. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 49511 [https://

[jira] [Updated] (SPARK-50834) Add Daily Github Action CI to `branch-4.0`

[ https://issues.apache.org/jira/browse/SPARK-50834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50834: -- Fix Version/s: 4.0.0 (was: 4.1.0) > Add Daily Github Action CI to `bran

[jira] [Resolved] (SPARK-50834) Add Daily Github Action CI to `branch-4.0`

[ https://issues.apache.org/jira/browse/SPARK-50834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50834. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49514 [https://

[jira] [Updated] (SPARK-50835) Add `Python-only` Daily CI to branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50835: -- Fix Version/s: 4.0.0 (was: 4.1.0) > Add `Python-only` Daily CI to branc

[jira] [Resolved] (SPARK-50835) Add `Python-only` Daily CI to branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50835. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49515 [https://

[jira] [Updated] (SPARK-50739) Changes in ResolveWithCTE.scala to have the analyzer grok recursive anchors

[ https://issues.apache.org/jira/browse/SPARK-50739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Cupac updated SPARK-50739: Description: Introduction of UnionLoop and UnionLoopRef logical plan classes. Changes in ResolveWi

[jira] [Resolved] (SPARK-50740) Addition of InsertLoops to replace recursive Union with UnionLoop and CTERelationRef referencing itself with UnionLoopRef

[ https://issues.apache.org/jira/browse/SPARK-50740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Cupac resolved SPARK-50740. - Resolution: Duplicate > Addition of InsertLoops to replace recursive Union with UnionLoop and >

[jira] [Closed] (SPARK-50740) Addition of InsertLoops to replace recursive Union with UnionLoop and CTERelationRef referencing itself with UnionLoopRef

[ https://issues.apache.org/jira/browse/SPARK-50740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milan Cupac closed SPARK-50740. --- > Addition of InsertLoops to replace recursive Union with UnionLoop and > CTERelationRef referencing it

[jira] [Created] (SPARK-50838) Implement checkRecursion to check if all the rules about recursive queries are fullfilled. Adjust optimizer with Unionloop cases.

Milan Cupac created SPARK-50838: --- Summary: Implement checkRecursion to check if all the rules about recursive queries are fullfilled. Adjust optimizer with Unionloop cases. Key: SPARK-50838 URL: https://issues.apach

[jira] [Resolved] (SPARK-50833) Update `buf` GitHub Action job to compare against branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50833. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49512 [https://

[jira] [Assigned] (SPARK-50833) Update `buf` GitHub Action job to compare against branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-50833: - Assignee: Dongjoon Hyun > Update `buf` GitHub Action job to compare against branch-4.0

[jira] [Assigned] (SPARK-50836) Add non-ANSI SQL Daily CI for branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-50836: - Assignee: Dongjoon Hyun > Add non-ANSI SQL Daily CI for branch-4.0 > --

[jira] [Updated] (SPARK-50836) Add non-ANSI SQL Daily CI for branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50836: -- Fix Version/s: 4.0.0 (was: 4.1.0) > Add non-ANSI SQL Daily CI for branc

[jira] [Created] (SPARK-50839) Spark Connect createDataFrame behavior does not match with classic Spark

Harsh Motwani created SPARK-50839: - Summary: Spark Connect createDataFrame behavior does not match with classic Spark Key: SPARK-50839 URL: https://issues.apache.org/jira/browse/SPARK-50839 Project: S

[jira] [Resolved] (SPARK-50836) Add non-ANSI SQL Daily CI for branch-4.0

[ https://issues.apache.org/jira/browse/SPARK-50836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-50836. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 49517 [https://

[jira] [Updated] (SPARK-50839) Spark Connect createDataFrame behavior does not match with classic Spark

[ https://issues.apache.org/jira/browse/SPARK-50839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh Motwani updated SPARK-50839: -- Description: In classic Spark shell (PySpark): {code:java} >>> spark.createDataFrame([1], "v i