[
https://issues.apache.org/jira/browse/SPARK-60148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Lu updated SPARK-60148:
-------------------------------
Description:
Lines 41-42 of pyspark/traceback_utils.py (when first_spark_frame == 0 in
first_spark_call()) are not exercised by existing tests because the test runner
is normally at the bottom of the call stack.
This ticket adds test_shell_reports_where_the_active_context_was_created to
SparkSubmitTests in python/pyspark/tests/test_appsubmit.py to exercise the
first_spark_frame == 0 branch via the real PySpark shell entry point
(bin/pyspark -> shell.py) without mocks.
was:
{{pyspark/traceback_utils.py}} has no dedicated unit tests. It is only
exercised indirectly through RDD and DataFrame actions
({{{}core/context.py{}}}, {{{}core/rdd.py{}}},
{{{}sql/classic/dataframe.py{}}}), so several branches are never checked
directly.
This ticket adds {{python/pyspark/tests/test_traceback_utils.py}} and registers
it in {{{}dev/sparktestsupport/modules.py{}}}. It covers:
* {{{}first_spark_call(){}}}: empty stack returns {{{}None{}}}; returns the
user's file and line for the first Spark frame; only the first Spark frame is
used; Spark frame at the bottom of the stack.
* {{{}SCCallSiteSync{}}}: sets and clears the call site; fallback message when
the call site is unknown; nested usage sets the call site only at the outermost
level; the call site is cleared when the block raises.
> Add unit tests for pyspark.traceback_utils
> ------------------------------------------
>
> Key: SPARK-60148
> URL: https://issues.apache.org/jira/browse/SPARK-60148
> Project: Spark
> Issue Type: Test
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Nicolas Lu
> Priority: Minor
> Labels: pull-request-available
>
> Lines 41-42 of pyspark/traceback_utils.py (when first_spark_frame == 0 in
> first_spark_call()) are not exercised by existing tests because the test
> runner is normally at the bottom of the call stack.
> This ticket adds test_shell_reports_where_the_active_context_was_created to
> SparkSubmitTests in python/pyspark/tests/test_appsubmit.py to exercise the
> first_spark_frame == 0 branch via the real PySpark shell entry point
> (bin/pyspark -> shell.py) without mocks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
