Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2554#issuecomment-57533566
This looks great, but I noticed one minor problem when running some manual
tests:
If I run `IPYTHON=1 ./bin/pyspark test.py` (where test.py is just some
dummy file that prints "Hello World!"), then this produces an error:
```
WARNING: Running python applications through ./bin/pyspark is deprecated as
of Spark 1.0.
Use ./bin/spark-submit <python file>
[TerminalIPythonApp] CRITICAL | Bad config encountered during
initialization:
[TerminalIPythonApp] CRITICAL | Unrecognized flag: '-u'
```
The problem here is that `spark-submit`'s `PythonRunner` passes the `-u`
flag to configure Python to use unbuffered output, but IPython doesn't support
this flag. It looks like we can use the `PYTHONUNBUFFERED` environment
variable instead
([source](https://docs.python.org/2.6/using/cmdline.html#envvar-PYTHONUNBUFFERED)),
which I think should also work with IPython.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
