Edward Lei created SPARK-60168:
----------------------------------
Summary: Remove type: ignore[union-attr] for the JVM and gateway
handles
Key: SPARK-60168
URL: https://issues.apache.org/jira/browse/SPARK-60168
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: Edward Lei
Five `# type: ignore[union-attr]` comments in PySpark suppress mypy errors that
arise only because `SparkContext._jvm` and `SparkContext._gateway` are typed
`Optional` (they are `None` until a `SparkContext` is created), even though the
surrounding code requires a live JVM.
This follows SPARK-58346, which removed similar ignores by making the non-None
assumption explicit with an `assert`, the same idiom `SparkContext` itself uses
(for example `assert self._jvm is not None` in `core/context.py`).
Proposed change:
- `sql/connect/session.py`, `sql/streaming/listener.py` (two places),
`ml/connect/io_utils.py`: assert the JVM or gateway handle is not None right
before using it.
- `ml/wrapper.py` (`JavaWrapper.__del__`): read
`SparkContext._active_spark_context` into a local variable and check `_gateway
is not None` in the existing condition. This method already swallows all
exceptions, so an assert would be pointless there.
No user-facing change. The only behavior difference is `AssertionError` instead
of `AttributeError` if the JVM handle is missing in the first group, which
already could not work in that case. Verified with mypy over `python/pyspark`
and the existing `test_wrapper` and `test_streaming_listener` tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
