Hi there,
I wanted to ask whether or not anyone has successfully used Jython with the pyspark library. I wasn't sure if the C extension support was needed for pyspark itself or was just a bonus of using Cython. There was a claim ( http://apache-spark-developers-list.1001551.n3.nabble.com/PySpark-Driver-from-Jython-td7142.html#a7269 ) that using Jython would be better - if you didn't need C extension support - because the cost of serialization is lower. However, I have not been able to import pyspark into a Jython session. I'm using version 2.7b3 of Jython and version 1.1.0 of Spark for reference. Jython 2.7b3 (default:e81256215fb0, Aug 4 2014, 02:39:51) [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_51 Type "help", "copyright", "credits" or "license" for more information. >>> from pyspark import SparkContext, SparkConf Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pyspark/__init__.py", line 63, in <module> File "pyspark/context.py", line 25, in <module> File "pyspark/accumulators.py", line 94, in <module> File "pyspark/serializers.py", line 341, in <module> File "pyspark/serializers.py", line 328, in _hijack_namedtuple RuntimeError: maximum recursion depth exceeded (Java StackOverflowError) Is there something I am missing with this? Did Jython ever work for pyspark? The same error happens regardless of whether I use the Python files or compile them down to Java class files using Jython first. I know that previous documentation (0.9.1) indicated, "PySpark requires Python 2.6 or higher. PySpark applications are executed using a standard CPython interpreter in order to support Python modules that use C extensions. We have not tested PySpark with Python 3 or with alternative Python interpreters, such as PyPy or Jython." In later versions, it now reflects, "Spark 1.1.0 works with Python 2.6 or higher (but not Python 3). It uses the standard CPython interpreter, so C libraries like NumPy can be used." I'm assuming this means that attempts to use other interpreters failed. If so, are there any plans to support something like Jython in the future? Signed, Chip Senkbeil
