Hello. I think you missed the SPARK_HOME in zeppelin-env.sh and you can refer to [1].
[1] http://zeppelin.incubator.apache.org/docs/0.5.6-incubating/interpreter/spark.html I hope this is help. 2016년 4월 17일 일요일, John Omernik<j...@omernik.com>님이 작성한 메시지: > I am trying to use the databricks csv reader and have tried multiple ways > to get this package available to pyspark. I have modified both > spark-defaults.conf and zeppelin-env.sh (as stated below). I've included > the spark-interpreter log from Zeppelin which seems to show it adding the > jar properly. Funny thing is, running pyspark at the command line works > properly. I will say this, I am running Zeppelin (and thus Spark) in > Docker, however, to ensure I did proper troubleshooting, I connected to the > docker container (that was throwing the error in Zeppelin) and ran pyspark > from within the container and it worked fine. The error only exists in > Zeppelin. > > I would welcome any assistance. > > John > > > > *Error in Zeppelin:* > Py4JJavaError: An error occurred while calling o82.load. > : java.lang.ClassNotFoundException: Failed to find data source: > com.databricks.spark.csv. Please find packages at > http://spark-packages.org > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) > at py4j.Gateway.invoke(Gateway.java:259) > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) > at py4j.commands.CallCommand.execute(CallCommand.java:79) > at py4j.GatewayConnection.run(GatewayConnection.java:209) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassNotFoundException: > com.databricks.spark.csv.DefaultSource > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62) > at scala.util.Try$.apply(Try.scala:161) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62) > at scala.util.Try.orElse(Try.scala:82) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:62) > ... 14 more > (<class 'py4j.protocol.Py4JJavaError'>, Py4JJavaError(u'An error occurred > while calling o82.load.\n', JavaObject id=o83), <traceback object at > 0x7f3776b36320>) > > > *zeppelin-env.sh* > > export SPARK_SUBMIT_OPTIONS="--packages > com.databricks:spark-csv_2.10:1.2.0" > > *spark-defaults.conf* > > spark.jars.packages com.databricks:spark-csv_2.10:1.2.0 > *Command I am running:* > > df = sqlContext.read.format('com.databricks.spark.csv').option('header', > 'true').option('inferschema', 'true').option('mode', > 'DROPMALFORMED').load('/user/test/airline/2016_ONTIME.csv') > > > > *spark interpreter log:* > > INFO [2016-04-17 11:45:59,335] ({pool-2-thread-2} > Logging.scala[logInfo]:58) - Added JAR > file:/home/test/.ivy2/jars/com.databricks_spark-csv_2.10-1.2.0.jar at > http://192.168.0.95:59483/jars/com.databricks_spark-csv_2.10-1.2.0.jar > with timestamp 1460893559334 > > INFO [2016-04-17 11:45:59,335] ({pool-2-thread-2} > Logging.scala[logInfo]:58) - Added JAR > file:/home/test/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar at > http://192.168.0.95:59483/jars/org.apache.commons_commons-csv-1.1.jar > with timestamp 1460893559335 > > INFO [2016-04-17 11:45:59,336] ({pool-2-thread-2} > Logging.scala[logInfo]:58) - Added JAR > file:/home/test/.ivy2/jars/com.univocity_univocity-parsers-1.5.1.jar at > http://192.168.0.95:59483/jars/com.univocity_univocity-parsers-1.5.1.jar > with timestamp 1460893559336 > > INFO [2016-04-17 11:45:59,348] ({pool-2-thread-2} > Logging.scala[logInfo]:58) - Added JAR > file:/zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar > at > http://192.168.0.95:59483/jars/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar > with timestamp 1460893559348 > > INFO [2016-04-17 11:45:59,470] ({pool-2-thread-2} > Logging.scala[logInfo]:58) - Created default pool default, schedulingMode: > FIFO, minShare: 0, weight: 1 > > INFO [2016-04-17 11:45:59,551] ({Thread-38} Logging.scala[logInfo]:58) - > Registered as framework ID e996d06e-4a8b-4647-9d07-02a7517c1453-0025 > > INFO [2016-04-17 11:45:59,556] ({pool-2-thread-2} > Logging.scala[logInfo]:58) - Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37373. >
