Hi, I think that the best option is to use the py4j which is either automatically installed with "pip install pyspark" or when we unzip the Spark download from its site, its in SPARK_HOME/python/lib folder.
Regards, Gourav Sengupta On Wed, Aug 29, 2018 at 8:00 AM Aakash Basu <aakash.spark....@gmail.com> wrote: > Hi, > > Which Py4J version goes with Spark 2.3.1? I have py4j-0.10.7 but throws > an error because of certain compatibility issues with the Spark 2.3.1. > > Error: > > [2018-08-29] [06:46:56] [ERROR] - Traceback (most recent call last): File > "<string>", line 120, in run File > "/data/spark-2.3.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/readwriter.py", > line 441, in csv return > self._df(self._jreader.csv(self._spark._sc._jvm.PythonUtils.toSeq(path))) > File > "/data/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", > line 1257, in __call__ answer, self.gateway_client, self.target_id, > self.name) File > "/data/spark-2.3.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/utils.py", > line 63, in deco return f(*a, **kw) File > "/data/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", > line 328, in get_return_value format(target_id, ".", name), value) > py4j.protocol.Py4JJavaError: An error occurred while calling o49.csv. > > Any help? > > Thanks, > Aakash. >
