Hyukjin Kwon created SPARK-58058:
------------------------------------
Summary: Split slow PySpark test suites into smaller modules
Key: SPARK-58058
URL: https://issues.apache.org/jira/browse/SPARK-58058
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Several PySpark test suites are slow in CI. This splits them into smaller
modules so they run as separate test targets and parallelize better, reducing
wall-clock time. No test logic is changed; test methods are only relocated,
following the existing convention already used in the repo (e.g.
test_split_apply_*, test_stat_*).
Suites split (each with a matching Spark Connect parity module):
- pyspark.sql.tests.pandas.test_pandas_cogrouped_map -> *_misc
- pyspark.sql.tests.arrow.test_arrow_cogrouped_map -> *_misc
- pyspark.pandas.tests.data_type_ops.test_num_ops -> *_integral_ext,
*_fractional_ext
- pyspark.pandas.tests.groupby.test_stat -> test_stat_median
- pyspark.pandas.tests.groupby.test_split_apply -> test_split_apply_mean
New modules are registered in dev/sparktestsupport/modules.py.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
