Joris, One thing to check - since you are using a downloaded jar, are you using the Uber jar that contains all the dependencies? http://search.maven.org/remotecontent?filepath=com/basho/riak/spark-riak-connector_2.10/1.6.0/spark-riak-connector_2.10-1.6.0-uber.jar
Thanks, Alex On Tue, Sep 13, 2016 at 8:44 AM, Stephen Etheridge <setheri...@basho.com> wrote: > Hi Joris, > > I have looked at the tutorial you have been following but I confess I am > confused. In the example you are following I do not see where the spark > and sql contexts are created. I use PySpark through the Jupyter notebook > and I have to specify a path to the connector on invoking the jupyter > notebook. Is it possible for you to share all your code (and how you are > invoking zeppelin) with me so I can trace everything through? > > regards > Stephen > > On Mon, Sep 12, 2016 at 3:27 PM, Agtmaal, Joris van < > joris.vanagtm...@wartsila.com> wrote: > >> Hi >> >> >> >> I’m new to Riak and followed the installation instructions to get it >> working on an AWS cluster (3 nodes). >> >> >> >> So far ive been able to use Riak in pyspark (zeppelin) to >> create/read/write tables, but i would like to use the dataframes directly >> from spark, using the Spark-Riak Connector. >> >> When following the example found here: http://docs.basho.com/riak/ts/ >> 1.4.0/add-ons/spark-riak-connector/quick-start/#python >> >> But i run into trouble on this last part: >> >> >> >> host= my_ip_adress_of_riak_node >> >> pb_port = '8087' >> >> hostAndPort = ":".join([host, pb_port]) >> >> client = riak.RiakClient(host=host, pb_port=pb_port) >> >> >> >> df.write \ >> >> .format('org.apache.spark.sql.riak') \ >> >> .option('spark.riak.connection.host', hostAndPort) \ >> >> .mode('Append') \ >> >> .save('test') >> >> >> >> Important to note that i’m using a local download of the Jar file that is >> loaded into the pyspark interpreter in zeppeling through: >> >> %dep >> >> z.reset() >> >> z.load("/home/hadoop/spark-riak-connector_2.10-1.6.0.jar") >> >> >> >> Here is the error message i get back: >> >> Py4JJavaError: An error occurred while calling o569.save. : >> java.lang.NoClassDefFoundError: com/basho/riak/client/core/util/HostAndPort >> at >> com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:76) >> at >> com.basho.riak.spark.rdd.connector.RiakConnectorConf$.apply(RiakConnectorConf.scala:89) >> at org.apache.spark.sql.riak.RiakRelation$.apply(RiakRelation.scala:115) >> at >> org.apache.spark.sql.riak.DefaultSource.createRelation(DefaultSource.scala:51) >> at org.apache.spark.sql.execution.datasources.ResolvedDataSourc >> e$.apply(ResolvedDataSource.scala:222) at org.apache.spark.sql.DataFrame >> Writer.save(DataFrameWriter.scala:148) at org.apache.spark.sql.DataFrame >> Writer.save(DataFrameWriter.scala:139) at >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) at >> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at >> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at >> py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand. >> invokeMethod(AbstractCommand.java:133) at >> py4j.commands.CallCommand.execute(CallCommand.java:79) >> at py4j.GatewayConnection.run(GatewayConnection.java:209) at >> java.lang.Thread.run(Thread.java:745) (<class >> 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error occurred while >> calling o569.save.\n', JavaObject id=o570), <traceback object at >> 0x7f7021bb0200>) >> >> >> >> Hope somebody can help out. >> >> thanks, joris >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > > -- > { "name" : "Stephen Etheridge", > "title" : "Solution Architect, EMEA", > "Organisation" : "Basho Technologies, Inc", > "Telephone" : "07814 406662", > "email" : "mailto:setheri...@basho.com";, > "github" : "http://github.com/datalemming";, > "twitter" : "@datalemming"} > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
