[
https://issues.apache.org/jira/browse/SPARK-58059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094963#comment-18094963
]
Hyukjin Kwon commented on SPARK-58059:
--------------------------------------
Investigated with per-doctest timing (ran the full pyspark.pandas.frame doctest
set with a DocTestFinder/DocTestRunner harness, timing each of the 133
docstrings individually).
Finding: the doctest *content* is not the cost.
- All 133 docstrings-with-examples execute in ~1.3s total locally (warm
session).
- Heaviest single docstring: ~0.29s; the rest are ~0.01-0.07s each.
- The CI doctest goal for pyspark.pandas.frame reports ~147s.
So ~145s of the ~147s is fixed per-goal overhead (JVM/SparkSession cold start,
bin/pyspark launch, local-cluster warmup, and per-example Spark-job dispatch
latency), not any expensive doctest.
Consequence: skipping or trimming individual doctests would not meaningfully
reduce the 147s (each is already ~10ms of real work). The effective levers are
(a) sharding the module's doctests across parallel workers -- which is not
feasible for an implementation file since doctest.testmod runs the whole module
as one runnable goal -- or (b) reducing the fixed per-goal startup overhead at
the test-runner/infra level.
Recommendation: this is not actionable as a "skip expensive doctests" change.
Closing unless we pursue a runner-level doctest sharding mechanism (broader
than this ticket). The class-based slow PySpark suites that *were* amenable to
splitting are handled in SPARK-58058.
> Reduce runtime of pyspark.pandas.frame doctests
> -----------------------------------------------
>
> Key: SPARK-58059
> URL: https://issues.apache.org/jira/browse/SPARK-58059
> Project: Spark
> Issue Type: Test
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> pyspark.pandas.frame runs as a single doctest module with ~800 doctest
> statements spanning the public DataFrame API, making it one of the slower
> PySpark test targets. Unlike class-based test suites, a doctest module cannot
> be split by simply moving test classes, and bulk-skipping public-API doctests
> would reduce user-facing documentation coverage.
> This is a follow-up to SPARK-58058 (which split several class-based PySpark
> suites). The goal here is to reduce the wall-clock runtime of the
> pyspark.pandas.frame doctests without losing meaningful coverage, e.g. by
> relocating the heaviest doctests into dedicated test cases that can run in
> parallel, or trimming redundant/expensive doctest examples where appropriate.
> Approach to be discussed given the doc-coverage tradeoff.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
