[jira] [Created] (SPARK-52952) Add PySpark UDF Type Coercion Dev Script

2025-07-24 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-52952: -- Summary: Add PySpark UDF Type Coercion Dev Script Key: SPARK-52952 URL: https://issues.apache.org/jira/browse/SPARK-52952 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-52877) Improve Python UDF Arrow Serializer Performance

2025-07-18 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-52877: -- Summary: Improve Python UDF Arrow Serializer Performance Key: SPARK-52877 URL: https://issues.apache.org/jira/browse/SPARK-52877 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-52794) Performance improvement of arrow-optimized python UDTF with LocalDataToArrowConversion check

2025-07-14 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-52794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-52794: --- Summary: Performance improvement of arrow-optimized python UDTF with LocalDataToArrowConversion chec

[jira] [Updated] (SPARK-52794) 2x Performance Improvement of Arrow-Optimized Python UDTF

2025-07-14 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-52794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-52794: --- Summary: 2x Performance Improvement of Arrow-Optimized Python UDTF (was: Improve Performance of Arr

[jira] [Created] (SPARK-52794) Improve Performance of Arrow-Optimized Python UDTF

2025-07-14 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-52794: -- Summary: Improve Performance of Arrow-Optimized Python UDTF Key: SPARK-52794 URL: https://issues.apache.org/jira/browse/SPARK-52794 Project: Spark Issue Type: Ta

[jira] [Created] (SPARK-52365) PySpark User Guide Sections

2025-05-31 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-52365: -- Summary: PySpark User Guide Sections Key: SPARK-52365 URL: https://issues.apache.org/jira/browse/SPARK-52365 Project: Spark Issue Type: Epic Components

[jira] [Created] (SPARK-52368) Spark Connect Overview

2025-05-31 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-52368: -- Summary: Spark Connect Overview Key: SPARK-52368 URL: https://issues.apache.org/jira/browse/SPARK-52368 Project: Spark Issue Type: Task Components: PyS

[jira] [Updated] (SPARK-52365) PySpark User Guide Sections

2025-05-31 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-52365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-52365: --- Description: Add sections to the PySpark User Guide: [https://spark.apache.org/docs/latest/api/pyth

[jira] [Updated] (SPARK-52366) Eager vs. Lazy: Spark Connect vs. Spark Classic

2025-05-31 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-52366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-52366: --- Summary: Eager vs. Lazy: Spark Connect vs. Spark Classic (was: Eager vs. Lazy: Spark Connect) > Ea

[jira] [Created] (SPARK-52367) Structured Streaming Overview

2025-05-31 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-52367: -- Summary: Structured Streaming Overview Key: SPARK-52367 URL: https://issues.apache.org/jira/browse/SPARK-52367 Project: Spark Issue Type: Task Componen

[jira] [Created] (SPARK-52366) Eager vs. Lazy: Spark Connect

2025-05-31 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-52366: -- Summary: Eager vs. Lazy: Spark Connect Key: SPARK-52366 URL: https://issues.apache.org/jira/browse/SPARK-52366 Project: Spark Issue Type: Task Componen

[jira] [Updated] (SPARK-51802) OSS PySpark User Guide Docs

2025-04-14 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51802: --- Summary: OSS PySpark User Guide Docs (was: OSS PySpark User Guide) > OSS PySpark User Guide Docs >

[jira] [Created] (SPARK-51747) Data source cached plan should account for options

2025-04-10 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51747: -- Summary: Data source cached plan should account for options Key: SPARK-51747 URL: https://issues.apache.org/jira/browse/SPARK-51747 Project: Spark Issue Type: Ta

[jira] [Updated] (SPARK-51747) Data source cached plan should respect options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Summary: Data source cached plan should respect options (was: Data source cached plan should accoun

[jira] [Updated] (SPARK-51747) Data source cached plan should account for options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Description: Before this, DataSourceStrategy caches the first query plan and does not respect optio

[jira] [Updated] (SPARK-51747) Data source cached plan should account for options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Description: Before this, DataSourceStrategy does not take into account options.   Before: ``` sp

[jira] [Updated] (SPARK-51747) Data source cached plan should account for options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Description: Before this, DataSourceStrategy caches the first query plan and does not respect optio

[jira] [Created] (SPARK-51657) UTF8_BINARY default table collation shown by default in Desc As JSON (v1)

2025-03-28 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51657: -- Summary: UTF8_BINARY default table collation shown by default in Desc As JSON (v1) Key: SPARK-51657 URL: https://issues.apache.org/jira/browse/SPARK-51657 Project: Spark

[jira] [Updated] (SPARK-51612) Display Spark confs set at view creation in Desc As Json

2025-03-26 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51612: --- Summary: Display Spark confs set at view creation in Desc As Json (was: Display view creation confs

[jira] [Created] (SPARK-51612) Display view creation confs in Desc As Json

2025-03-26 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51612: -- Summary: Display view creation confs in Desc As Json Key: SPARK-51612 URL: https://issues.apache.org/jira/browse/SPARK-51612 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-51525) Add collation field in Desc As JSON type

2025-03-17 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51525: --- Description: Before: {"table_name":"table","catalog_name":"spark_catalog","namespace":["ns"],"schem

[jira] [Updated] (SPARK-51525) Add collation field in Desc As JSON type

2025-03-16 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51525: --- Description: Before: {"table_name":"table","catalog_name":"spark_catalog","namespace":["ns"],"schem

[jira] [Created] (SPARK-51525) Add collation field in Desc As JSON type

2025-03-16 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51525: -- Summary: Add collation field in Desc As JSON type Key: SPARK-51525 URL: https://issues.apache.org/jira/browse/SPARK-51525 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-51363) `Desc As JSON` clustering column names

2025-03-03 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51363: --- Summary: `Desc As JSON` clustering column names (was: Delegate `Desc As JSON` clustering info to re

[jira] [Created] (SPARK-51363) Delegate `Desc As JSON` clustering info to recursive jsonType struct

2025-03-02 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51363: -- Summary: Delegate `Desc As JSON` clustering info to recursive jsonType struct Key: SPARK-51363 URL: https://issues.apache.org/jira/browse/SPARK-51363 Project: Spark

[jira] [Updated] (SPARK-51084) Assign appropriate error class for negativeScaleNotAllowedError

2025-02-05 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51084: --- Description: Improve user-facing error message for `negativeScaleNotAllowedError`. Previously it wa

[jira] [Updated] (SPARK-51084) Assign appropriate error class for negativeScaleNotAllowedError

2025-02-05 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51084: --- Description: Improve user-facing error message for `negativeScaleNotAllowedError`. Previously was a

[jira] [Updated] (SPARK-51084) Assign appropriate error class for negativeScaleNotAllowedError

2025-02-05 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51084: --- Description: Improve user-facing error message for `negativeScaleNotAllowedError`. Previously was a

[jira] [Updated] (SPARK-51084) Assign appropriate error class for negativeScaleNotAllowedError

2025-02-04 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51084: --- Summary: Assign appropriate error class for negativeScaleNotAllowedError (was: Improve negativeScal

[jira] [Created] (SPARK-51084) Improve negativeScaleNotAllowedError message

2025-02-04 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51084: -- Summary: Improve negativeScaleNotAllowedError message Key: SPARK-51084 URL: https://issues.apache.org/jira/browse/SPARK-51084 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-51034) Reformat Describe As JSON statistics dict for parse-ability

2025-01-29 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51034: -- Summary: Reformat Describe As JSON statistics dict for parse-ability Key: SPARK-51034 URL: https://issues.apache.org/jira/browse/SPARK-51034 Project: Spark Issu

[jira] [Created] (SPARK-51032) Improve Describe As JSON Unsupported Table Error Message

2025-01-29 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51032: -- Summary: Improve Describe As JSON Unsupported Table Error Message Key: SPARK-51032 URL: https://issues.apache.org/jira/browse/SPARK-51032 Project: Spark Issue Ty

[jira] [Created] (SPARK-51007) Describe As JSON v2 Table Support

2025-01-27 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-51007: -- Summary: Describe As JSON v2 Table Support Key: SPARK-51007 URL: https://issues.apache.org/jira/browse/SPARK-51007 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-50541) Describe Table As JSON

2025-01-27 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-50541: --- Issue Type: Epic (was: Task) > Describe Table As JSON > -- > >

[jira] [Created] (SPARK-50795) Display all DESCRIBE AS JSON dates in ISO-8601 format

2025-01-12 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-50795: -- Summary: Display all DESCRIBE AS JSON dates in ISO-8601 format Key: SPARK-50795 URL: https://issues.apache.org/jira/browse/SPARK-50795 Project: Spark Issue Type:

[jira] [Created] (SPARK-50690) Fix discrepancy in DESCRIBE TABLE columns list quoting

2024-12-27 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-50690: -- Summary: Fix discrepancy in DESCRIBE TABLE columns list quoting Key: SPARK-50690 URL: https://issues.apache.org/jira/browse/SPARK-50690 Project: Spark Issue Type

[jira] [Created] (SPARK-50541) Describe Table As JSON

2024-12-10 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-50541: -- Summary: Describe Table As JSON Key: SPARK-50541 URL: https://issues.apache.org/jira/browse/SPARK-50541 Project: Spark Issue Type: Task Components: SQL

[jira] [Updated] (SPARK-50541) Describe Table As JSON

2024-12-10 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-50541: --- Description: Support DESCRIBE TABLE ...  [AS JSON] option to display table metadata in JSON format.

[jira] [Updated] (SPARK-50541) Describe Table As JSON

2024-12-10 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-50541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-50541: --- Description: Support DESCRIBE TABLE ...  [AS JSON] option to display table metadata in JSON format.

[jira] [Updated] (SPARK-49145) Improve readability of log4j console log output

2024-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-49145: --- Description: Prior to this update, the OSS Spark logs were difficult to interpret. The logs followe

[jira] [Updated] (SPARK-49145) Improve readability of log4j console log output

2024-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-49145: --- Description: Prior to this update, the OSS Spark logs were difficult to interpret. The logs followe

[jira] [Created] (SPARK-49145) Improve readability of log4j console log output

2024-08-07 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-49145: -- Summary: Improve readability of log4j console log output Key: SPARK-49145 URL: https://issues.apache.org/jira/browse/SPARK-49145 Project: Spark Issue Type: Task

[jira] [Updated] (SPARK-48759) Add migration doc for CREATE TABLE AS SELECT behavior change behavior change since Spark 3.4

2024-06-30 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-48759: --- Summary: Add migration doc for CREATE TABLE AS SELECT behavior change behavior change since Spark 3.

[jira] [Created] (SPARK-48759) Add migration doc for CREATE TABLE behavior change behavior change since Spark 3.4

2024-06-30 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48759: -- Summary: Add migration doc for CREATE TABLE behavior change behavior change since Spark 3.4 Key: SPARK-48759 URL: https://issues.apache.org/jira/browse/SPARK-48759 Projec

[jira] [Created] (SPARK-48740) Catch missing window specification error early

2024-06-27 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48740: -- Summary: Catch missing window specification error early Key: SPARK-48740 URL: https://issues.apache.org/jira/browse/SPARK-48740 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-48676) Structured Logging Framework Scala Style Migration [Part 2]

2024-06-20 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48676: -- Summary: Structured Logging Framework Scala Style Migration [Part 2] Key: SPARK-48676 URL: https://issues.apache.org/jira/browse/SPARK-48676 Project: Spark Issu

[jira] [Resolved] (SPARK-48632) Remove unused LogKeys

2024-06-14 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu resolved SPARK-48632. Resolution: Not A Problem > Remove unused LogKeys > - > > Key:

[jira] [Created] (SPARK-48632) Remove unused LogKeys

2024-06-14 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48632: -- Summary: Remove unused LogKeys Key: SPARK-48632 URL: https://issues.apache.org/jira/browse/SPARK-48632 Project: Spark Issue Type: Sub-task Components:

[jira] [Created] (SPARK-48623) Structured Logging Framework Scala Style Migration

2024-06-13 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48623: -- Summary: Structured Logging Framework Scala Style Migration Key: SPARK-48623 URL: https://issues.apache.org/jira/browse/SPARK-48623 Project: Spark Issue Type: Su

[jira] [Created] (SPARK-48592) Add scala style check for logging message inline variables

2024-06-11 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-48592: -- Summary: Add scala style check for logging message inline variables Key: SPARK-48592 URL: https://issues.apache.org/jira/browse/SPARK-48592 Project: Spark Issue

[jira] [Created] (SPARK-46910) Eliminate JDK Requirement in PySpark Installation

2024-01-29 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-46910: -- Summary: Eliminate JDK Requirement in PySpark Installation Key: SPARK-46910 URL: https://issues.apache.org/jira/browse/SPARK-46910 Project: Spark Issue Type: Imp

[jira] [Created] (SPARK-45729) Fix PySpark testing guide links

2023-10-30 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-45729: -- Summary: Fix PySpark testing guide links Key: SPARK-45729 URL: https://issues.apache.org/jira/browse/SPARK-45729 Project: Spark Issue Type: Sub-task Co

[jira] [Updated] (SPARK-44712) Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44712: --- Description: Migrate assert_eq to assertDataFrameEqual in this file: [‎python/pyspark/pandas/tests/d

[jira] [Created] (SPARK-44712) Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44712: -- Summary: Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual Key: SPARK-44712 URL: https://issues.apache.org/jira/browse/SPARK-44712 Project: Spark

[jira] [Updated] (SPARK-44711) Migrate test_series_conversion assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44711: --- Description: Migrate assert_eq to assertDataFrameEqual in this file:  [‎python/pyspark/pandas/tests/

[jira] [Created] (SPARK-44711) Migrate test_series_conversion assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44711: -- Summary: Migrate test_series_conversion assert_eq to use assertDataFrameEqual Key: SPARK-44711 URL: https://issues.apache.org/jira/browse/SPARK-44711 Project: Spark

[jira] [Created] (SPARK-44708) Migrate test_reset_index assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44708: -- Summary: Migrate test_reset_index assert_eq to use assertDataFrameEqual Key: SPARK-44708 URL: https://issues.apache.org/jira/browse/SPARK-44708 Project: Spark I

[jira] [Updated] (SPARK-44597) Migrate test_sql assert_eq to use assertDataFrameEqual

2023-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44597: --- Description: Migrate tests to new test utils in this file: python/pyspark/pandas/tests/test_sql.py

[jira] [Updated] (SPARK-44589) Migrate PySpark tests to use PySpark built-in test utils

2023-08-07 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44589: --- Description: The Jira ticket SPARK-44042 SPIP: PySpark Test Framework introduces a new PySpark test

[jira] [Created] (SPARK-44682) Make pandas error class message_parameters strings

2023-08-04 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44682: -- Summary: Make pandas error class message_parameters strings Key: SPARK-44682 URL: https://issues.apache.org/jira/browse/SPARK-44682 Project: Spark Issue Type: Su

[jira] [Updated] (SPARK-44548) Add support for pandas-on-Spark DataFrame assertDataFrameEqual

2023-08-03 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44548: --- Summary: Add support for pandas-on-Spark DataFrame assertDataFrameEqual (was: Add support for panda

[jira] [Created] (SPARK-44665) Add support for pandas DataFrame assertDataFrameEqual

2023-08-03 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44665: -- Summary: Add support for pandas DataFrame assertDataFrameEqual Key: SPARK-44665 URL: https://issues.apache.org/jira/browse/SPARK-44665 Project: Spark Issue Type:

[jira] [Created] (SPARK-44652) Raise error when only one df is None

2023-08-02 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44652: -- Summary: Raise error when only one df is None Key: SPARK-44652 URL: https://issues.apache.org/jira/browse/SPARK-44652 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-44645) Update assertDataFrameEqual docs error example output

2023-08-02 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44645: --- Summary: Update assertDataFrameEqual docs error example output (was: Update assertDataFrame docs er

[jira] [Created] (SPARK-44645) Update assertDataFrame docs error example output

2023-08-02 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44645: -- Summary: Update assertDataFrame docs error example output Key: SPARK-44645 URL: https://issues.apache.org/jira/browse/SPARK-44645 Project: Spark Issue Type: Sub-

[jira] [Created] (SPARK-44629) Publish PySpark Test Guidelines webpage

2023-08-01 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44629: -- Summary: Publish PySpark Test Guidelines webpage Key: SPARK-44629 URL: https://issues.apache.org/jira/browse/SPARK-44629 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-44617) Support comparison between list of Rows

2023-07-31 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44617: -- Summary: Support comparison between list of Rows Key: SPARK-44617 URL: https://issues.apache.org/jira/browse/SPARK-44617 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-44617) Support comparison between lists of Rows

2023-07-31 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44617: --- Summary: Support comparison between lists of Rows (was: Support comparison between list of Rows) >

[jira] [Updated] (SPARK-44597) Migrate test_sql assert_eq to use assertDataFrameEqual

2023-07-31 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44597: --- Description: The Jira ticket [[SPARK-44042] SPIP: PySpark Test Framework |https://issues.apache.org

[jira] [Updated] (SPARK-44597) Migrate test_sql assert_eq to use assertDataFrameEqual

2023-07-31 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44597: --- Description: The Jira ticket [SPARK-44042] SPIP: PySpark Test Framework introduces a new PySpark t

[jira] [Updated] (SPARK-44603) Add pyspark.testing to setup.py

2023-07-30 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44603: --- Summary: Add pyspark.testing to setup.py (was: Add pyspark.testing.utils to setup.py) > Add pyspar

[jira] [Created] (SPARK-44603) Add pyspark.testing.utils to Python Setup

2023-07-30 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44603: -- Summary: Add pyspark.testing.utils to Python Setup Key: SPARK-44603 URL: https://issues.apache.org/jira/browse/SPARK-44603 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-44603) Add pyspark.testing.utils to setup.py

2023-07-30 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44603: --- Summary: Add pyspark.testing.utils to setup.py (was: Add pyspark.testing.utils to Python Setup) >

[jira] [Created] (SPARK-44597) Migrate test_sql assert_eq to use assertDataFrameEqual

2023-07-29 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44597: -- Summary: Migrate test_sql assert_eq to use assertDataFrameEqual Key: SPARK-44597 URL: https://issues.apache.org/jira/browse/SPARK-44597 Project: Spark Issue Type

[jira] [Created] (SPARK-44596) Fix pandas-on-Spark type checks for assertDataFrameEqual

2023-07-29 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44596: -- Summary: Fix pandas-on-Spark type checks for assertDataFrameEqual Key: SPARK-44596 URL: https://issues.apache.org/jira/browse/SPARK-44596 Project: Spark Issue Ty

[jira] [Created] (SPARK-44589) Migrate PySpark tests to use PySpark built-in test utils

2023-07-28 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44589: -- Summary: Migrate PySpark tests to use PySpark built-in test utils Key: SPARK-44589 URL: https://issues.apache.org/jira/browse/SPARK-44589 Project: Spark Issue Ty

[jira] [Updated] (SPARK-44218) Customize diff log in assertDataFrameEqual error message format

2023-07-28 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44218: --- Summary: Customize diff log in assertDataFrameEqual error message format (was: Customize context_di

[jira] [Updated] (SPARK-44218) Customize context_diff in assertDataFrameEqual error message format

2023-07-28 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44218: --- Summary: Customize context_diff in assertDataFrameEqual error message format (was: Add improved err

[jira] [Updated] (SPARK-44546) Add a dev utility to generate PySpark tests with LLM

2023-07-25 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44546: --- Description: h2. Summary This ticket adds a dev utility script to help generate PySpark tests using

[jira] [Created] (SPARK-44548) Add support for pandas DataFrame assertDataFrameEqual

2023-07-25 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44548: -- Summary: Add support for pandas DataFrame assertDataFrameEqual Key: SPARK-44548 URL: https://issues.apache.org/jira/browse/SPARK-44548 Project: Spark Issue Type:

[jira] [Updated] (SPARK-44546) Add a dev utility to generate PySpark tests with LLM

2023-07-25 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44546: --- Description: h2. Summary This ticket adds a dev utility script to help generate PySpark tests using

[jira] [Updated] (SPARK-44546) Add a dev utility to generate PySpark tests with LLM

2023-07-25 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44546: --- Description: h2. Summary This ticket adds a dev utility script to help generate PySpark tests using

[jira] [Updated] (SPARK-44546) Add a dev utility to generate PySpark tests with LLM

2023-07-25 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44546: --- Description: h2. Summary This ticket adds a dev utility script to help generate PySpark tests using

[jira] [Updated] (SPARK-44546) Add a dev utility to generate PySpark tests with LLM

2023-07-25 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44546: --- Description: h2. Summary This ticket adds a dev utility script to help generate PySpark tests using

[jira] [Created] (SPARK-44546) Add a dev utility to generate PySpark tests with LLM

2023-07-25 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44546: -- Summary: Add a dev utility to generate PySpark tests with LLM Key: SPARK-44546 URL: https://issues.apache.org/jira/browse/SPARK-44546 Project: Spark Issue Type:

[jira] [Updated] (SPARK-44061) Add assertDataFrameEqual util function

2023-07-17 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44061: --- Summary: Add assertDataFrameEqual util function (was: Add assertDataFrameEquality util function) >

[jira] [Created] (SPARK-44453) Use difflib to display errors in assertDataFrameEqual

2023-07-16 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44453: -- Summary: Use difflib to display errors in assertDataFrameEqual Key: SPARK-44453 URL: https://issues.apache.org/jira/browse/SPARK-44453 Project: Spark Issue Type:

[jira] [Created] (SPARK-44446) Add checks for expected list type special cases

2023-07-16 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-6: -- Summary: Add checks for expected list type special cases Key: SPARK-6 URL: https://issues.apache.org/jira/browse/SPARK-6 Project: Spark Issue Type: Sub-t

[jira] [Created] (SPARK-44413) Clarify error for unsupported arg data type in assertDataFrameEqual

2023-07-13 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44413: -- Summary: Clarify error for unsupported arg data type in assertDataFrameEqual Key: SPARK-44413 URL: https://issues.apache.org/jira/browse/SPARK-44413 Project: Spark

[jira] [Updated] (SPARK-44216) Make assertSchemaEqual API public

2023-07-13 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44216: --- Summary: Make assertSchemaEqual API public (was: Make assertSchemaEqual API with ignore_nullable op

[jira] [Created] (SPARK-44397) Expose assertDataFrameEqual in pyspark.testing.utils

2023-07-12 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44397: -- Summary: Expose assertDataFrameEqual in pyspark.testing.utils Key: SPARK-44397 URL: https://issues.apache.org/jira/browse/SPARK-44397 Project: Spark Issue Type:

[jira] [Updated] (SPARK-44217) Allow custom precision for fp approx equality

2023-07-11 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44217: --- Summary: Allow custom precision for fp approx equality (was: Add assert_approx_df_equality util fun

[jira] [Updated] (SPARK-44216) Add assertSchemaEqual API with ignore_nullable optional flag

2023-07-10 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44216: --- Summary: Add assertSchemaEqual API with ignore_nullable optional flag (was: Add improved error mess

[jira] [Updated] (SPARK-44216) Make assertSchemaEqual API with ignore_nullable optional flag

2023-07-10 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44216: --- Summary: Make assertSchemaEqual API with ignore_nullable optional flag (was: Add assertSchemaEqual

[jira] [Updated] (SPARK-44363) Display percent of unequal rows in DataFrame comparison

2023-07-10 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44363: --- Summary: Display percent of unequal rows in DataFrame comparison (was: Display percent of unequal r

[jira] [Updated] (SPARK-44061) Add assertDataFrameEquality util function

2023-07-10 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44061: --- Summary: Add assertDataFrameEquality util function (was: Add assert_df_equality util function) > A

[jira] [Created] (SPARK-44364) Support List[Row] data type for expected DataFrame argument

2023-07-10 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44364: -- Summary: Support List[Row] data type for expected DataFrame argument Key: SPARK-44364 URL: https://issues.apache.org/jira/browse/SPARK-44364 Project: Spark Issu

[jira] [Created] (SPARK-44363) Display percent of unequal rows in dataframe comparison

2023-07-10 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44363: -- Summary: Display percent of unequal rows in dataframe comparison Key: SPARK-44363 URL: https://issues.apache.org/jira/browse/SPARK-44363 Project: Spark Issue Typ

[jira] [Created] (SPARK-44357) Add pyspark_testing module for GHA tests

2023-07-10 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44357: -- Summary: Add pyspark_testing module for GHA tests Key: SPARK-44357 URL: https://issues.apache.org/jira/browse/SPARK-44357 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-44218) Add improved error message formatting for assert_approx_df_equality

2023-06-27 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44218: -- Summary: Add improved error message formatting for assert_approx_df_equality Key: SPARK-44218 URL: https://issues.apache.org/jira/browse/SPARK-44218 Project: Spark

  1   2   >