Hey guys,
I am trying to get zeppelin to work with my HDP cluster with spark,
cassandra, and yarn. Unfortunately, for the last three days I have
been trying multiple compilation options, configuration settings,
adjusting pom.xml, etc. I just cannot seem to get it working. Have any
of you been able to get Zeppelin to work with yarn/spark/cassandra?
Any help would be very much appreciated.
install command:
mvn clean package -Pbuild-distr -Pcassandra-spark-1.5
-Dhadoop.version=2.6.0 -Phadoop-2.6 -DskipTests
Environment setup:
hdp-select status hadoop-client | sed 's/hadoop-client - \(.*\)/\1/' =
2.4.0.0-169
hadoop version = Hadoop 2.7.1.2.4.0.0-169
spark = 1.6.0
HDP has 9 nodes. I am only configuring the master node(which hosts
zeppelin) and not using the ambari zeppelin service.
config:
zeppelin-env.sh
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_HOME=/usr/hdp/current/spark-client
export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.4.0.0-169"
interpreter config:
"id": "2BGDFG2NR",
"name": "spark",
"group": "spark",
"properties": {
"spark.executor.memory": "",
"args": "",
"zeppelin.spark.printREPLOutput": "true",
"spark.cores.max": "",
"zeppelin.dep.additionalRemoteRepository":
"spark-packages,http://dl.bintray.com/spark-packages/maven,false;";,
"zeppelin.spark.sql.stacktrace": "false",
"zeppelin.spark.concurrentSQL": "false",
"zeppelin.spark.useHiveContext": "true",
"zeppelin.pyspark.python": "python",
"zeppelin.dep.localrepo": "local-repo",
"spark.cassandra.auth.password": "[password]",
"zeppelin.interpreter.localRepo":
"/home/zeppelin/zeppelin_live/local-repo/2BGDFG2NR",
"spark.cassandra.connection.host": "10.0.4.80",
"spark.yarn.am.extraJavaOptions": "-Dhdp.version\u003d2.4.0.0-169",
"zeppelin.spark.maxResult": "1000",
"master": "yarn-client",
"spark.app.name": "Zeppelin",
"spark.cassandra.auth.username": "zeppelin",
"spark.driver.extraJavaOptions": "-Dhdp.version\u003d2.4.0.0-169"
},
"interpreterGroup": [
{
"class": "org.apache.zeppelin.spark.SparkInterpreter",
"name": "spark"
},
{
"class": "org.apache.zeppelin.spark.PySparkInterpreter",
"name": "pyspark"
},
{
"class": "org.apache.zeppelin.spark.SparkSqlInterpreter",
"name": "sql"
},
{
"class": "org.apache.zeppelin.spark.DepInterpreter",
"name": "dep"
}
],
"dependencies": [
{
"groupArtifactVersion":
"com.datastax.spark:spark-cassandra-connector_2.10:1.6.0-M2",
"local": false
},
{
"groupArtifactVersion": "com.google.guava:guava:16.0.1",
"local": false
}
],
"option": {
"remote": true,
"perNoteSession": false
}
},
Notebook:
I have the following paragraphs:
1) import com.datastax.spark.connector._
----result----
import com.datastax.spark.connector._
2) val rdd = sc.cassandraTable("testing", "results")
----result----
rdd:
com.datastax.spark.connector.rdd.CassandraTableScanRDD[com.datastax.spark.connector.CassandraRow]
= CassandraTableScanRDD[0] at RDD at CassandraRDD.scala:15
3) rdd.count
----result----
org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3
in stage 1.0 (TID 21, vm-10-155-208-69.cloud.mwn.de):
java.io.IOException: Failed to open native connection to Cassandra at
{10.0.4.80, 10.0.4.81, 10.0.4.82}:9042
at
com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at
com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at
com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
at
com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
at
com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
at
com.datastax.spark.connector.rdd.CassandraTableScanRDD.compute(CassandraTableScanRDD.scala:319)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError:
com.google.common.util.concurrent.Futures.withFallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureFallback;Ljava/util/concurrent/Executor;)Lcom/google/common/util/concurrent/ListenableFuture;
at com.datastax.driver.core.Connection.initAsync(Connection.java:177)
at com.datastax.driver.core.Connection$Factory.open(Connection.java:731)
at
com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:251)
at
com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:199)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1414)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:393)
at
com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
looking at the logs:
INFO [2016-04-19 15:39:34,612] ({dispatcher-event-loop-5}
Logging.scala[logInfo]:58) - Starting task 1.1 in stage 0.0 (TID 3,
[master], partition 1,RACK_LOCAL, 4863 bytes)
INFO [2016-04-19 15:39:34,616] ({task-result-getter-1}
Logging.scala[logInfo]:58) - Lost task 0.0 in stage 0.0 (TID 0) on
executor [master]: java.io.IOException (Failed to open native
connection to Cassandra at {10.0.4.80, 10.0.4.81, 10.0.4.82}:9042)
[duplicate 1]
INFO [2016-04-19 15:39:36,279] ({dispatcher-event-loop-3}
Logging.scala[logInfo]:58) - Starting task 0.1 in stage 0.0 (TID 4,
[worker 1], partition 0,RACK_LOCAL, 4863 bytes)
INFO [2016-04-19 15:39:36,281] ({task-result-getter-2}
Logging.scala[logInfo]:58) - Lost task 2.0 in stage 0.0 (TID 2) on
executor [worker 1]: java.io.IOException (Failed to open native
connection to Cassandra at {10.0.4.80, 10.0.4.81, 10.0.4.82}:9042)
[duplicate 2]
INFO [2016-04-19 15:39:36,862] ({dispatcher-event-lop-2}
Logging.scala[logInfo]:58) - Starting task 2.1 in stage 0.0 (TID 5,
[master], partition 2,RACK_LOCAL, 4863 bytes)
INFO [2016-04-19 15:39:36,863] ({task-result-getter-3}
Logging.scala[logInfo]:58) - Lost task 1.1 in stage 0.0 (TID 3) on
executor [master]: java.io.IOException (Failed to open native
connection to Cassandra at {10.0.4.80, 10.0.4.81, 10.0.4.82}:9042)
[duplicate 3]
and it just keeps going
