Peter Parente created SPARK-20818:
-------------------------------------
Summary: Tab-to-autocomplete in IPython + Python3 results in job
execution
Key: SPARK-20818
URL: https://issues.apache.org/jira/browse/SPARK-20818
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.0.2
Reporter: Peter Parente
Using Spark in a Jupyter Notebook 5.0 with an IPython 6.0 kernel and Python 3,
when I press Tab to autocomplete the function names on an RDD, Spark executes
the job graph constructed thus far. This only appears to happen for certain
autocompletions, namely completions that resolve to functions having the
@ignore_unicode_prefix decorator applied to them, and only in Python 3 (never
in Python 2).
https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L146
Indeed, this function does have a special case for Python 3 in which it
rewrites function docstrings. Why IPython autocompletion has a bad interaction
with this logic is unknown (to me at least) at the moment.
I reproduced this bug on Spark 2.0.2. The code in the decorator hasn't changed
in 2.1.x, so the bug likely impacts that version as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
