[
https://issues.apache.org/jira/browse/SPARK-20818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Parente updated SPARK-20818:
----------------------------------
Description:
Using Spark in a Jupyter Notebook 5.0 with an IPython 6.0 kernel and Python 3,
when I press Tab to autocomplete the function names on a DataFrame or RDD,
Spark executes the job graph constructed thus far. This only appears to happen
for certain autocompletions, namely completions that resolve to functions
having the @ignore_unicode_prefix decorator applied to them, and only in Python
3 (never in Python 2).
https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L146
Indeed, this function does have a special case for Python 3 in which it
rewrites function docstrings. Why IPython autocompletion has a bad interaction
with this logic is unknown (to me at least) at the moment.
I reproduced this bug on Spark 2.0.2. The code in the decorator hasn't changed
in 2.1.x, so the bug likely impacts that version as well.
was:
Using Spark in a Jupyter Notebook 5.0 with an IPython 6.0 kernel and Python 3,
when I press Tab to autocomplete the function names on an RDD, Spark executes
the job graph constructed thus far. This only appears to happen for certain
autocompletions, namely completions that resolve to functions having the
@ignore_unicode_prefix decorator applied to them, and only in Python 3 (never
in Python 2).
https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L146
Indeed, this function does have a special case for Python 3 in which it
rewrites function docstrings. Why IPython autocompletion has a bad interaction
with this logic is unknown (to me at least) at the moment.
I reproduced this bug on Spark 2.0.2. The code in the decorator hasn't changed
in 2.1.x, so the bug likely impacts that version as well.
> Tab-to-autocomplete in IPython + Python3 results in job execution
> -----------------------------------------------------------------
>
> Key: SPARK-20818
> URL: https://issues.apache.org/jira/browse/SPARK-20818
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.0.2
> Reporter: Peter Parente
> Labels: ipython, jupyter
>
> Using Spark in a Jupyter Notebook 5.0 with an IPython 6.0 kernel and Python
> 3, when I press Tab to autocomplete the function names on a DataFrame or RDD,
> Spark executes the job graph constructed thus far. This only appears to
> happen for certain autocompletions, namely completions that resolve to
> functions having the @ignore_unicode_prefix decorator applied to them, and
> only in Python 3 (never in Python 2).
> https://github.com/apache/spark/blob/master/python/pyspark/rdd.py#L146
> Indeed, this function does have a special case for Python 3 in which it
> rewrites function docstrings. Why IPython autocompletion has a bad
> interaction with this logic is unknown (to me at least) at the moment.
> I reproduced this bug on Spark 2.0.2. The code in the decorator hasn't
> changed in 2.1.x, so the bug likely impacts that version as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
