Hi there,
I'm using Spark 1.1.0 and experimenting with trying to use the DataStax
Cassandra Connector (https://github.com/datastax/spark-cassandra-connector)
from within PySpark.
As a baby step, I'm simply trying to validate that I have access to classes
that I'd need via Py4J. Sample python program:
from py4j.java_gateway import java_import
from pyspark.conf import SparkConf
from pyspark import SparkContext
conf = SparkConf().set("spark.cassandra.connection.host", "127.0.0.1")
sc = SparkContext(appName="Spark + Cassandra Example", conf=conf)
java_import(sc._gateway.jvm, "com.datastax.spark.connector.*")
print sc._jvm.CassandraRow()
CassandraRow corresponds to
https://github.com/datastax/spark-cassandra-connector/blob/master/spark-cassandra-connector/src/main/scala/com/datastax/spark/connector/CassandraRow.scala
which is included in the JAR I submit. Feel free to download the JAR here
https://dl.dropboxusercontent.com/u/4385786/pyspark-cassandra-0.1.0-SNAPSHOT-standalone.jar
I'm currently running this Python example with:
spark-submit
--driver-class-path="/path/to/pyspark-cassandra-0.1.0-SNAPSHOT-standalone.jar"
--verbose src/python/cassandara_example.py
But continually get the following error indicating that the classes aren't
in fact on the classpath of the GatewayServer:
Traceback (most recent call last):
File
"/Users/mikesukmanowsky/Development/parsely/pyspark-cassandra/src/python/cassandara_example.py",
line 37, in <module>
main()
File
"/Users/mikesukmanowsky/Development/parsely/pyspark-cassandra/src/python/cassandara_example.py",
line 25, in main
print sc._jvm.CassandraRow()
File
"/Users/mikesukmanowsky/.opt/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
line 726, in __getattr__
py4j.protocol.Py4JError: Trying to call a package.
The correct response from the GatewayServer should be:
In [22]: gateway.jvm.CassandraRow()
Out[22]: JavaObject id=o0
Also tried using --jars option instead and that doesn't seem to work
either. Is there something I'm missing as to why the classes aren't
available?
--
Mike Sukmanowsky
Aspiring Digital Carpenter
*p*: +1 (416) 953-4248
*e*: [email protected]
facebook <http://facebook.com/mike.sukmanowsky> | twitter
<http://twitter.com/msukmanowsky> | LinkedIn
<http://www.linkedin.com/profile/view?id=10897143> | github
<https://github.com/msukmanowsky>
