Hyukjin Kwon created SPARK-58059:
------------------------------------
Summary: Reduce runtime of pyspark.pandas.frame doctests
Key: SPARK-58059
URL: https://issues.apache.org/jira/browse/SPARK-58059
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
pyspark.pandas.frame runs as a single doctest module with ~800 doctest
statements spanning the public DataFrame API, making it one of the slower
PySpark test targets. Unlike class-based test suites, a doctest module cannot
be split by simply moving test classes, and bulk-skipping public-API doctests
would reduce user-facing documentation coverage.
This is a follow-up to SPARK-58058 (which split several class-based PySpark
suites). The goal here is to reduce the wall-clock runtime of the
pyspark.pandas.frame doctests without losing meaningful coverage, e.g. by
relocating the heaviest doctests into dedicated test cases that can run in
parallel, or trimming redundant/expensive doctest examples where appropriate.
Approach to be discussed given the doc-coverage tradeoff.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
