I looked at this again, and when I use the Scala spark-shell and load a CSV using the same package it works just fine, so this seems specific to pyspark.
I've created the following JIRA: https://issues.apache.org/jira/browse/SPARK-8365 -Don On Sat, Jun 13, 2015 at 11:46 AM, Don Drake <[email protected]> wrote: > I downloaded the pre-compiled Spark 1.4.0 and attempted to run an existing > Python Spark application against it and got the following error: > > py4j.protocol.Py4JJavaError: An error occurred while calling o90.save. > : java.lang.RuntimeException: Failed to load class for data source: > com.databricks.spark.csv > > I pass the following on the command-line to my spark-submit: > --packages com.databricks:spark-csv_2.10:1.0.3 > > This worked fine on 1.3.1, but not in 1.4. > > I was able to replicate it with the following pyspark: > > a = {'a':1.0, 'b':'asdf'} > rdd = sc.parallelize([a]) > df = sqlContext.createDataFrame(rdd) > df.save("/tmp/d.csv", "com.databricks.spark.csv") > > > Even using the new > df.write.format('com.databricks.spark.csv').save('/tmp/d.csv') gives the > same error. > > I see it was added in the web UI: > file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jarAdded > By > Userfile:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jarAdded > By User > http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jarAdded > By > Userhttp://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jarAdded > By User > Thoughts? > > -Don > > > > Gory details: > > $ pyspark --packages "com.databricks:spark-csv_2.10:1.0.3" > Python 2.7.6 (default, Sep 9 2014, 15:04:36) > [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > Ivy Default Cache set to: /Users/drake/.ivy2/cache > The jars for the packages stored in: /Users/drake/.ivy2/jars > :: loading settings :: url = > jar:file:/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/lib/spark-assembly-1.4.0-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml > com.databricks#spark-csv_2.10 added as a dependency > :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 > confs: [default] > found com.databricks#spark-csv_2.10;1.0.3 in central > found org.apache.commons#commons-csv;1.1 in central > :: resolution report :: resolve 590ms :: artifacts dl 17ms > :: modules in use: > com.databricks#spark-csv_2.10;1.0.3 from central in [default] > org.apache.commons#commons-csv;1.1 from central in [default] > --------------------------------------------------------------------- > | | modules || artifacts | > | conf | number| search|dwnlded|evicted|| number|dwnlded| > --------------------------------------------------------------------- > | default | 2 | 0 | 0 | 0 || 2 | 0 | > --------------------------------------------------------------------- > :: retrieving :: org.apache.spark#spark-submit-parent > confs: [default] > 0 artifacts copied, 2 already retrieved (0kB/15ms) > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 15/06/13 11:06:08 INFO SparkContext: Running Spark version 1.4.0 > 2015-06-13 11:06:08.921 java[19233:2145789] Unable to load realm info from > SCDynamicStore > 15/06/13 11:06:09 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 15/06/13 11:06:09 WARN Utils: Your hostname, Dons-MacBook-Pro-2.local > resolves to a loopback address: 127.0.0.1; using 10.0.0.222 instead (on > interface en0) > 15/06/13 11:06:09 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to > another address > 15/06/13 11:06:09 INFO SecurityManager: Changing view acls to: drake > 15/06/13 11:06:09 INFO SecurityManager: Changing modify acls to: drake > 15/06/13 11:06:09 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(drake); users > with modify permissions: Set(drake) > 15/06/13 11:06:10 INFO Slf4jLogger: Slf4jLogger started > 15/06/13 11:06:10 INFO Remoting: Starting remoting > 15/06/13 11:06:10 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://[email protected]:56870] > 15/06/13 11:06:10 INFO Utils: Successfully started service 'sparkDriver' > on port 56870. > 15/06/13 11:06:10 INFO SparkEnv: Registering MapOutputTracker > 15/06/13 11:06:10 INFO SparkEnv: Registering BlockManagerMaster > 15/06/13 11:06:10 INFO DiskBlockManager: Created local directory at > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/blockmgr-a1412b71-fe56-429c-a193-ce3fb95d2ffd > 15/06/13 11:06:10 INFO MemoryStore: MemoryStore started with capacity > 265.4 MB > 15/06/13 11:06:10 INFO HttpFileServer: HTTP File server directory is > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/httpd-84d178da-7e60-4eed-8031-e6a0c465bd4c > 15/06/13 11:06:10 INFO HttpServer: Starting HTTP Server > 15/06/13 11:06:10 INFO Utils: Successfully started service 'HTTP file > server' on port 56871. > 15/06/13 11:06:10 INFO SparkEnv: Registering OutputCommitCoordinator > 15/06/13 11:06:11 WARN Utils: Service 'SparkUI' could not bind on port > 4040. Attempting port 4041. > 15/06/13 11:06:11 INFO Utils: Successfully started service 'SparkUI' on > port 4041. > 15/06/13 11:06:11 INFO SparkUI: Started SparkUI at http://10.0.0.222:4041 > 15/06/13 11:06:11 INFO SparkContext: Added JAR > file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar at > http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jar with > timestamp 1434211571303 > 15/06/13 11:06:11 INFO SparkContext: Added JAR > file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar at > http://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jar with > timestamp 1434211571326 > 15/06/13 11:06:11 INFO Utils: Copying > /Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar > 15/06/13 11:06:11 INFO SparkContext: Added file > file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar at > file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar with > timestamp 1434211571468 > 15/06/13 11:06:11 INFO Utils: Copying > /Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar > 15/06/13 11:06:11 INFO SparkContext: Added file > file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar at > file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar with > timestamp 1434211571502 > 15/06/13 11:06:11 INFO Executor: Starting executor ID driver on host > localhost > 15/06/13 11:06:11 INFO Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56872. > 15/06/13 11:06:11 INFO NettyBlockTransferService: Server created on 56872 > 15/06/13 11:06:11 INFO BlockManagerMaster: Trying to register BlockManager > 15/06/13 11:06:11 INFO BlockManagerMasterEndpoint: Registering block > manager localhost:56872 with 265.4 MB RAM, BlockManagerId(driver, > localhost, 56872) > 15/06/13 11:06:11 INFO BlockManagerMaster: Registered BlockManager > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /__ / .__/\_,_/_/ /_/\_\ version 1.4.0 > /_/ > > Using Python version 2.7.6 (default, Sep 9 2014 15:04:36) > SparkContext available as sc, HiveContext available as sqlContext. > >>> a = {'a':1.0, 'b':'asdf'} > >>> rdd = sc.parallelize([a]) > >>> df = sqlContext.createDataFrame(rdd) > 15/06/13 11:06:50 INFO SparkContext: Starting job: runJob at > PythonRDD.scala:366 > 15/06/13 11:06:50 INFO DAGScheduler: Got job 0 (runJob at > PythonRDD.scala:366) with 1 output partitions (allowLocal=true) > 15/06/13 11:06:50 INFO DAGScheduler: Final stage: ResultStage 0(runJob at > PythonRDD.scala:366) > 15/06/13 11:06:50 INFO DAGScheduler: Parents of final stage: List() > 15/06/13 11:06:50 INFO DAGScheduler: Missing parents: List() > 15/06/13 11:06:50 INFO DAGScheduler: Submitting ResultStage 0 > (PythonRDD[1] at RDD at PythonRDD.scala:43), which has no missing parents > 15/06/13 11:06:51 INFO MemoryStore: ensureFreeSpace(3672) called with > curMem=0, maxMem=278302556 > 15/06/13 11:06:51 INFO MemoryStore: Block broadcast_0 stored as values in > memory (estimated size 3.6 KB, free 265.4 MB) > 15/06/13 11:06:51 INFO MemoryStore: ensureFreeSpace(2328) called with > curMem=3672, maxMem=278302556 > 15/06/13 11:06:51 INFO MemoryStore: Block broadcast_0_piece0 stored as > bytes in memory (estimated size 2.3 KB, free 265.4 MB) > 15/06/13 11:06:51 INFO BlockManagerInfo: Added broadcast_0_piece0 in > memory on localhost:56872 (size: 2.3 KB, free: 265.4 MB) > 15/06/13 11:06:51 INFO SparkContext: Created broadcast 0 from broadcast at > DAGScheduler.scala:874 > 15/06/13 11:06:51 INFO DAGScheduler: Submitting 1 missing tasks from > ResultStage 0 (PythonRDD[1] at RDD at PythonRDD.scala:43) > 15/06/13 11:06:51 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks > 15/06/13 11:06:51 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID > 0, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:51 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) > 15/06/13 11:06:51 INFO Executor: Fetching > file:/Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar with > timestamp 1434211571502 > 15/06/13 11:06:51 INFO Utils: > /Users/drake/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar has been > previously copied to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar > 15/06/13 11:06:51 INFO Executor: Fetching > file:/Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar with > timestamp 1434211571468 > 15/06/13 11:06:51 INFO Utils: > /Users/drake/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar has been > previously copied to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar > 15/06/13 11:06:51 INFO Executor: Fetching > http://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jar with > timestamp 1434211571326 > 15/06/13 11:06:51 INFO Utils: Fetching > http://10.0.0.222:56871/jars/org.apache.commons_commons-csv-1.1.jar to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp2449082240048543653.tmp > 15/06/13 11:06:51 INFO Utils: > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp2449082240048543653.tmp > has been previously copied to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar > 15/06/13 11:06:51 INFO Executor: Adding > file:/private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/org.apache.commons_commons-csv-1.1.jar > to class loader > 15/06/13 11:06:51 INFO Executor: Fetching > http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jar with > timestamp 1434211571303 > 15/06/13 11:06:51 INFO Utils: Fetching > http://10.0.0.222:56871/jars/com.databricks_spark-csv_2.10-1.0.3.jar to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp1396931258018379545.tmp > 15/06/13 11:06:51 INFO Utils: > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/fetchFileTemp1396931258018379545.tmp > has been previously copied to > /private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar > 15/06/13 11:06:51 INFO Executor: Adding > file:/private/var/folders/7_/k5h82ws97b95v5f5h8wf9j0h0000gn/T/spark-f36f39f5-7f82-42e0-b3e0-9eb1e1cc0816/userFiles-1cab505b-7e88-4f9c-82d9-d6b361689d9d/com.databricks_spark-csv_2.10-1.0.3.jar > to class loader > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 3165, boot = 3155, init = > 10, finish = 0 > 15/06/13 11:06:54 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). > 666 bytes result sent to driver > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID > 0) in 3505 ms on localhost (1/1) > 15/06/13 11:06:54 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks > have all completed, from pool > 15/06/13 11:06:54 INFO DAGScheduler: ResultStage 0 (runJob at > PythonRDD.scala:366) finished in 3.525 s > 15/06/13 11:06:54 INFO DAGScheduler: Job 0 finished: runJob at > PythonRDD.scala:366, took 3.852112 s > 15/06/13 11:06:54 INFO SparkContext: Starting job: runJob at > PythonRDD.scala:366 > 15/06/13 11:06:54 INFO DAGScheduler: Got job 1 (runJob at > PythonRDD.scala:366) with 4 output partitions (allowLocal=true) > 15/06/13 11:06:54 INFO DAGScheduler: Final stage: ResultStage 1(runJob at > PythonRDD.scala:366) > 15/06/13 11:06:54 INFO DAGScheduler: Parents of final stage: List() > 15/06/13 11:06:54 INFO DAGScheduler: Missing parents: List() > 15/06/13 11:06:54 INFO DAGScheduler: Submitting ResultStage 1 > (PythonRDD[2] at RDD at PythonRDD.scala:43), which has no missing parents > 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(3672) called with > curMem=6000, maxMem=278302556 > 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_1 stored as values in > memory (estimated size 3.6 KB, free 265.4 MB) > 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(2330) called with > curMem=9672, maxMem=278302556 > 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_1_piece0 stored as > bytes in memory (estimated size 2.3 KB, free 265.4 MB) > 15/06/13 11:06:54 INFO BlockManagerInfo: Added broadcast_1_piece0 in > memory on localhost:56872 (size: 2.3 KB, free: 265.4 MB) > 15/06/13 11:06:54 INFO SparkContext: Created broadcast 1 from broadcast at > DAGScheduler.scala:874 > 15/06/13 11:06:54 INFO DAGScheduler: Submitting 4 missing tasks from > ResultStage 1 (PythonRDD[2] at RDD at PythonRDD.scala:43) > 15/06/13 11:06:54 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID > 1, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID > 2, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID > 3, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID > 4, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:54 INFO Executor: Running task 0.0 in stage 1.0 (TID 1) > 15/06/13 11:06:54 INFO Executor: Running task 1.0 in stage 1.0 (TID 2) > 15/06/13 11:06:54 INFO Executor: Running task 2.0 in stage 1.0 (TID 3) > 15/06/13 11:06:54 INFO Executor: Running task 3.0 in stage 1.0 (TID 4) > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -15, init = 17, > finish = 0 > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 9, boot = 6, init = 2, > finish = 1 > 15/06/13 11:06:54 INFO Executor: Finished task 3.0 in stage 1.0 (TID 4). > 666 bytes result sent to driver > 15/06/13 11:06:54 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). > 666 bytes result sent to driver > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 13, boot = 9, init = 4, > finish = 0 > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID > 4) in 24 ms on localhost (1/4) > 15/06/13 11:06:54 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). > 666 bytes result sent to driver > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID > 1) in 28 ms on localhost (2/4) > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID > 3) in 27 ms on localhost (3/4) > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 28, boot = 28, init = 0, > finish = 0 > 15/06/13 11:06:54 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). > 666 bytes result sent to driver > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID > 2) in 42 ms on localhost (4/4) > 15/06/13 11:06:54 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks > have all completed, from pool > 15/06/13 11:06:54 INFO DAGScheduler: ResultStage 1 (runJob at > PythonRDD.scala:366) finished in 0.044 s > 15/06/13 11:06:54 INFO DAGScheduler: Job 1 finished: runJob at > PythonRDD.scala:366, took 0.063304 s > 15/06/13 11:06:54 INFO SparkContext: Starting job: runJob at > PythonRDD.scala:366 > 15/06/13 11:06:54 INFO DAGScheduler: Got job 2 (runJob at > PythonRDD.scala:366) with 3 output partitions (allowLocal=true) > 15/06/13 11:06:54 INFO DAGScheduler: Final stage: ResultStage 2(runJob at > PythonRDD.scala:366) > 15/06/13 11:06:54 INFO DAGScheduler: Parents of final stage: List() > 15/06/13 11:06:54 INFO DAGScheduler: Missing parents: List() > 15/06/13 11:06:54 INFO DAGScheduler: Submitting ResultStage 2 > (PythonRDD[3] at RDD at PythonRDD.scala:43), which has no missing parents > 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(3672) called with > curMem=12002, maxMem=278302556 > 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_2 stored as values in > memory (estimated size 3.6 KB, free 265.4 MB) > 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(2330) called with > curMem=15674, maxMem=278302556 > 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_2_piece0 stored as > bytes in memory (estimated size 2.3 KB, free 265.4 MB) > 15/06/13 11:06:54 INFO BlockManagerInfo: Added broadcast_2_piece0 in > memory on localhost:56872 (size: 2.3 KB, free: 265.4 MB) > 15/06/13 11:06:54 INFO SparkContext: Created broadcast 2 from broadcast at > DAGScheduler.scala:874 > 15/06/13 11:06:54 INFO DAGScheduler: Submitting 3 missing tasks from > ResultStage 2 (PythonRDD[3] at RDD at PythonRDD.scala:43) > 15/06/13 11:06:54 INFO TaskSchedulerImpl: Adding task set 2.0 with 3 tasks > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID > 5, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID > 6, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID > 7, localhost, PROCESS_LOCAL, 1708 bytes) > 15/06/13 11:06:54 INFO Executor: Running task 0.0 in stage 2.0 (TID 5) > 15/06/13 11:06:54 INFO Executor: Running task 1.0 in stage 2.0 (TID 6) > 15/06/13 11:06:54 INFO Executor: Running task 2.0 in stage 2.0 (TID 7) > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -41, init = 43, > finish = 0 > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -38, init = 40, > finish = 0 > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 1, boot = -77, init = 78, > finish = 0 > 15/06/13 11:06:54 INFO Executor: Finished task 0.0 in stage 2.0 (TID 5). > 666 bytes result sent to driver > 15/06/13 11:06:54 INFO Executor: Finished task 1.0 in stage 2.0 (TID 6). > 666 bytes result sent to driver > 15/06/13 11:06:54 INFO Executor: Finished task 2.0 in stage 2.0 (TID 7). > 722 bytes result sent to driver > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID > 5) in 14 ms on localhost (1/3) > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID > 6) in 13 ms on localhost (2/3) > 15/06/13 11:06:54 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID > 7) in 13 ms on localhost (3/3) > 15/06/13 11:06:54 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks > have all completed, from pool > 15/06/13 11:06:54 INFO DAGScheduler: ResultStage 2 (runJob at > PythonRDD.scala:366) finished in 0.016 s > 15/06/13 11:06:54 INFO DAGScheduler: Job 2 finished: runJob at > PythonRDD.scala:366, took 0.087522 s > /Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/context.py:198: > UserWarning: Using RDD of dict to inferSchema is deprecated,please use > pyspark.sql.Row instead > warnings.warn("Using RDD of dict to inferSchema is deprecated," > 15/06/13 11:06:54 INFO BlockManagerInfo: Removed broadcast_1_piece0 on > localhost:56872 in memory (size: 2.3 KB, free: 265.4 MB) > 15/06/13 11:06:54 INFO BlockManagerInfo: Removed broadcast_0_piece0 on > localhost:56872 in memory (size: 2.3 KB, free: 265.4 MB) > 15/06/13 11:06:54 INFO SparkContext: Starting job: runJob at > PythonRDD.scala:366 > 15/06/13 11:06:54 INFO DAGScheduler: Got job 3 (runJob at > PythonRDD.scala:366) with 1 output partitions (allowLocal=true) > 15/06/13 11:06:54 INFO DAGScheduler: Final stage: ResultStage 3(runJob at > PythonRDD.scala:366) > 15/06/13 11:06:54 INFO DAGScheduler: Parents of final stage: List() > 15/06/13 11:06:54 INFO DAGScheduler: Missing parents: List() > 15/06/13 11:06:54 INFO DAGScheduler: Submitting ResultStage 3 > (PythonRDD[4] at RDD at PythonRDD.scala:43), which has no missing parents > 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(5120) called with > curMem=6002, maxMem=278302556 > 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_3 stored as values in > memory (estimated size 5.0 KB, free 265.4 MB) > 15/06/13 11:06:54 INFO MemoryStore: ensureFreeSpace(3338) called with > curMem=11122, maxMem=278302556 > 15/06/13 11:06:54 INFO MemoryStore: Block broadcast_3_piece0 stored as > bytes in memory (estimated size 3.3 KB, free 265.4 MB) > 15/06/13 11:06:54 INFO BlockManagerInfo: Added broadcast_3_piece0 in > memory on localhost:56872 (size: 3.3 KB, free: 265.4 MB) > 15/06/13 11:06:54 INFO SparkContext: Created broadcast 3 from broadcast at > DAGScheduler.scala:874 > 15/06/13 11:06:54 INFO DAGScheduler: Submitting 1 missing tasks from > ResultStage 3 (PythonRDD[4] at RDD at PythonRDD.scala:43) > 15/06/13 11:06:54 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks > 15/06/13 11:06:54 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID > 8, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:54 INFO Executor: Running task 0.0 in stage 3.0 (TID 8) > 15/06/13 11:06:54 INFO PythonRDD: Times: total = 2, boot = -29, init = 31, > finish = 0 > 15/06/13 11:06:55 INFO Executor: Finished task 0.0 in stage 3.0 (TID 8). > 666 bytes result sent to driver > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID > 8) in 10 ms on localhost (1/1) > 15/06/13 11:06:55 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks > have all completed, from pool > 15/06/13 11:06:55 INFO DAGScheduler: ResultStage 3 (runJob at > PythonRDD.scala:366) finished in 0.011 s > 15/06/13 11:06:55 INFO DAGScheduler: Job 3 finished: runJob at > PythonRDD.scala:366, took 0.035088 s > 15/06/13 11:06:55 INFO SparkContext: Starting job: runJob at > PythonRDD.scala:366 > 15/06/13 11:06:55 INFO DAGScheduler: Got job 4 (runJob at > PythonRDD.scala:366) with 4 output partitions (allowLocal=true) > 15/06/13 11:06:55 INFO DAGScheduler: Final stage: ResultStage 4(runJob at > PythonRDD.scala:366) > 15/06/13 11:06:55 INFO DAGScheduler: Parents of final stage: List() > 15/06/13 11:06:55 INFO DAGScheduler: Missing parents: List() > 15/06/13 11:06:55 INFO DAGScheduler: Submitting ResultStage 4 > (PythonRDD[5] at RDD at PythonRDD.scala:43), which has no missing parents > 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(5120) called with > curMem=14460, maxMem=278302556 > 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_4 stored as values in > memory (estimated size 5.0 KB, free 265.4 MB) > 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(3337) called with > curMem=19580, maxMem=278302556 > 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_4_piece0 stored as > bytes in memory (estimated size 3.3 KB, free 265.4 MB) > 15/06/13 11:06:55 INFO BlockManagerInfo: Added broadcast_4_piece0 in > memory on localhost:56872 (size: 3.3 KB, free: 265.4 MB) > 15/06/13 11:06:55 INFO SparkContext: Created broadcast 4 from broadcast at > DAGScheduler.scala:874 > 15/06/13 11:06:55 INFO DAGScheduler: Submitting 4 missing tasks from > ResultStage 4 (PythonRDD[5] at RDD at PythonRDD.scala:43) > 15/06/13 11:06:55 INFO TaskSchedulerImpl: Adding task set 4.0 with 4 tasks > 15/06/13 11:06:55 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID > 9, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:55 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID > 10, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:55 INFO TaskSetManager: Starting task 2.0 in stage 4.0 (TID > 11, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:55 INFO TaskSetManager: Starting task 3.0 in stage 4.0 (TID > 12, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:55 INFO Executor: Running task 3.0 in stage 4.0 (TID 12) > 15/06/13 11:06:55 INFO Executor: Running task 1.0 in stage 4.0 (TID 10) > 15/06/13 11:06:55 INFO Executor: Running task 2.0 in stage 4.0 (TID 11) > 15/06/13 11:06:55 INFO Executor: Running task 0.0 in stage 4.0 (TID 9) > 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -79, init = 80, > finish = 0 > 15/06/13 11:06:55 INFO Executor: Finished task 3.0 in stage 4.0 (TID 12). > 666 bytes result sent to driver > 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -27, init = 28, > finish = 0 > 15/06/13 11:06:55 INFO Executor: Finished task 1.0 in stage 4.0 (TID 10). > 666 bytes result sent to driver > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 3.0 in stage 4.0 (TID > 12) in 11 ms on localhost (1/4) > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID > 10) in 14 ms on localhost (2/4) > 15/06/13 11:06:55 INFO PythonRDD: Times: total = 22, boot = 22, init = 0, > finish = 0 > 15/06/13 11:06:55 INFO PythonRDD: Times: total = 21, boot = 21, init = 0, > finish = 0 > 15/06/13 11:06:55 INFO Executor: Finished task 2.0 in stage 4.0 (TID 11). > 666 bytes result sent to driver > 15/06/13 11:06:55 INFO Executor: Finished task 0.0 in stage 4.0 (TID 9). > 666 bytes result sent to driver > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 2.0 in stage 4.0 (TID > 11) in 37 ms on localhost (3/4) > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID > 9) in 43 ms on localhost (4/4) > 15/06/13 11:06:55 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks > have all completed, from pool > 15/06/13 11:06:55 INFO DAGScheduler: ResultStage 4 (runJob at > PythonRDD.scala:366) finished in 0.044 s > 15/06/13 11:06:55 INFO DAGScheduler: Job 4 finished: runJob at > PythonRDD.scala:366, took 0.059163 s > 15/06/13 11:06:55 INFO SparkContext: Starting job: runJob at > PythonRDD.scala:366 > 15/06/13 11:06:55 INFO DAGScheduler: Got job 5 (runJob at > PythonRDD.scala:366) with 3 output partitions (allowLocal=true) > 15/06/13 11:06:55 INFO DAGScheduler: Final stage: ResultStage 5(runJob at > PythonRDD.scala:366) > 15/06/13 11:06:55 INFO DAGScheduler: Parents of final stage: List() > 15/06/13 11:06:55 INFO DAGScheduler: Missing parents: List() > 15/06/13 11:06:55 INFO DAGScheduler: Submitting ResultStage 5 > (PythonRDD[6] at RDD at PythonRDD.scala:43), which has no missing parents > 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(5120) called with > curMem=22917, maxMem=278302556 > 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_5 stored as values in > memory (estimated size 5.0 KB, free 265.4 MB) > 15/06/13 11:06:55 INFO MemoryStore: ensureFreeSpace(3338) called with > curMem=28037, maxMem=278302556 > 15/06/13 11:06:55 INFO MemoryStore: Block broadcast_5_piece0 stored as > bytes in memory (estimated size 3.3 KB, free 265.4 MB) > 15/06/13 11:06:55 INFO BlockManagerInfo: Added broadcast_5_piece0 in > memory on localhost:56872 (size: 3.3 KB, free: 265.4 MB) > 15/06/13 11:06:55 INFO SparkContext: Created broadcast 5 from broadcast at > DAGScheduler.scala:874 > 15/06/13 11:06:55 INFO DAGScheduler: Submitting 3 missing tasks from > ResultStage 5 (PythonRDD[6] at RDD at PythonRDD.scala:43) > 15/06/13 11:06:55 INFO TaskSchedulerImpl: Adding task set 5.0 with 3 tasks > 15/06/13 11:06:55 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID > 13, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:55 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID > 14, localhost, PROCESS_LOCAL, 1665 bytes) > 15/06/13 11:06:55 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID > 15, localhost, PROCESS_LOCAL, 1708 bytes) > 15/06/13 11:06:55 INFO Executor: Running task 0.0 in stage 5.0 (TID 13) > 15/06/13 11:06:55 INFO Executor: Running task 1.0 in stage 5.0 (TID 14) > 15/06/13 11:06:55 INFO Executor: Running task 2.0 in stage 5.0 (TID 15) > 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -24, init = 25, > finish = 0 > 15/06/13 11:06:55 INFO PythonRDD: Times: total = 1, boot = -24, init = 25, > finish = 0 > 15/06/13 11:06:55 INFO Executor: Finished task 0.0 in stage 5.0 (TID 13). > 666 bytes result sent to driver > 15/06/13 11:06:55 INFO Executor: Finished task 2.0 in stage 5.0 (TID 15). > 716 bytes result sent to driver > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID > 13) in 12 ms on localhost (1/3) > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID > 15) in 11 ms on localhost (2/3) > 15/06/13 11:06:55 INFO PythonRDD: Times: total = 21, boot = 20, init = 1, > finish = 0 > 15/06/13 11:06:55 INFO Executor: Finished task 1.0 in stage 5.0 (TID 14). > 666 bytes result sent to driver > 15/06/13 11:06:55 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID > 14) in 36 ms on localhost (3/3) > 15/06/13 11:06:55 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks > have all completed, from pool > 15/06/13 11:06:55 INFO DAGScheduler: ResultStage 5 (runJob at > PythonRDD.scala:366) finished in 0.038 s > 15/06/13 11:06:55 INFO DAGScheduler: Job 5 finished: runJob at > PythonRDD.scala:366, took 0.052978 s > 15/06/13 11:06:56 INFO HiveContext: Initializing execution hive, version > 0.13.1 > 15/06/13 11:06:56 INFO HiveMetaStore: 0: Opening raw store with > implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > 15/06/13 11:06:56 INFO ObjectStore: ObjectStore, initialize called > 15/06/13 11:06:57 INFO Persistence: Property datanucleus.cache.level2 > unknown - will be ignored > 15/06/13 11:06:57 INFO Persistence: Property > hive.metastore.integral.jdo.pushdown unknown - will be ignored > 15/06/13 11:06:57 WARN Connection: BoneCP specified but not present in > CLASSPATH (or one of dependencies) > 15/06/13 11:06:58 WARN Connection: BoneCP specified but not present in > CLASSPATH (or one of dependencies) > 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_5_piece0 on > localhost:56872 in memory (size: 3.3 KB, free: 265.4 MB) > 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_4_piece0 on > localhost:56872 in memory (size: 3.3 KB, free: 265.4 MB) > 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on > localhost:56872 in memory (size: 3.3 KB, free: 265.4 MB) > 15/06/13 11:06:59 INFO BlockManagerInfo: Removed broadcast_2_piece0 on > localhost:56872 in memory (size: 2.3 KB, free: 265.4 MB) > 15/06/13 11:07:00 INFO ObjectStore: Setting MetaStore object pin classes > with > hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" > 15/06/13 11:07:00 INFO MetaStoreDirectSql: MySQL check failed, assuming we > are not on mysql: Lexical error at line 1, column 5. Encountered: "@" > (64), after : "". > 15/06/13 11:07:01 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:01 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:04 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:04 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:04 INFO ObjectStore: Initialized ObjectStore > 15/06/13 11:07:04 WARN ObjectStore: Version information not found in > metastore. hive.metastore.schema.verification is not enabled so recording > the schema version 0.13.1aa > 15/06/13 11:07:05 INFO HiveMetaStore: Added admin role in metastore > 15/06/13 11:07:05 INFO HiveMetaStore: Added public role in metastore > 15/06/13 11:07:05 INFO HiveMetaStore: No user is added in admin role, > since config is empty > 15/06/13 11:07:05 INFO SessionState: No Tez session required at this > point. hive.execution.engine=mr. > 15/06/13 11:07:06 INFO HiveContext: Initializing HiveMetastoreConnection > version 0.13.1 using Spark classes. > 15/06/13 11:07:07 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 15/06/13 11:07:08 INFO HiveMetaStore: 0: Opening raw store with > implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > 15/06/13 11:07:08 INFO ObjectStore: ObjectStore, initialize called > 15/06/13 11:07:08 INFO Persistence: Property datanucleus.cache.level2 > unknown - will be ignored > 15/06/13 11:07:08 INFO Persistence: Property > hive.metastore.integral.jdo.pushdown unknown - will be ignored > 15/06/13 11:07:08 WARN Connection: BoneCP specified but not present in > CLASSPATH (or one of dependencies) > 15/06/13 11:07:09 WARN Connection: BoneCP specified but not present in > CLASSPATH (or one of dependencies) > 15/06/13 11:07:11 INFO ObjectStore: Setting MetaStore object pin classes > with > hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" > 15/06/13 11:07:11 INFO MetaStoreDirectSql: MySQL check failed, assuming we > are not on mysql: Lexical error at line 1, column 5. Encountered: "@" > (64), after : "". > 15/06/13 11:07:12 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:12 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:13 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:13 INFO Datastore: The class > "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > "embedded-only" so does not have its own datastore table. > 15/06/13 11:07:14 INFO ObjectStore: Initialized ObjectStore > 15/06/13 11:07:14 WARN ObjectStore: Version information not found in > metastore. hive.metastore.schema.verification is not enabled so recording > the schema version 0.13.1aa > 15/06/13 11:07:14 INFO HiveMetaStore: Added admin role in metastore > 15/06/13 11:07:15 INFO HiveMetaStore: Added public role in metastore > 15/06/13 11:07:15 INFO HiveMetaStore: No user is added in admin role, > since config is empty > 15/06/13 11:07:15 INFO SessionState: No Tez session required at this > point. hive.execution.engine=mr. > >>> > >>> df.save("/tmp/d.csv", "com.databricks.spark.csv") > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File > "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/dataframe.py", > line 202, in save > return self.write.save(path, source, mode, **options) > File > "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/readwriter.py", > line 295, in save > self._jwrite.save(path) > File > "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", > line 538, in __call__ > File > "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", > line 300, in get_return_value > py4j.protocol.Py4JJavaError: An error occurred while calling o86.save. > : java.lang.RuntimeException: Failed to load class for data source: > com.databricks.spark.csv > at scala.sys.package$.error(package.scala:27) > at > org.apache.spark.sql.sources.ResolvedDataSource$.lookupDataSource(ddl.scala:216) > at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:302) > at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:144) > at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:135) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) > at py4j.Gateway.invoke(Gateway.java:259) > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) > at py4j.commands.CallCommand.execute(CallCommand.java:79) > at py4j.GatewayConnection.run(GatewayConnection.java:207) > at java.lang.Thread.run(Thread.java:744) > > >>> > ... > >>> df.write.format("com.databricks.spark.csv").save("/tmp/d.csv") > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File > "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/readwriter.py", > line 295, in save > self._jwrite.save(path) > File > "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", > line 538, in __call__ > File > "/Users/drake/spark/spark-1.4.0-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", > line 300, in get_return_value > py4j.protocol.Py4JJavaError: An error occurred while calling o90.save. > : java.lang.RuntimeException: Failed to load class for data source: > com.databricks.spark.csv > at scala.sys.package$.error(package.scala:27) > at > org.apache.spark.sql.sources.ResolvedDataSource$.lookupDataSource(ddl.scala:216) > at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:302) > at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:144) > at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:135) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) > at py4j.Gateway.invoke(Gateway.java:259) > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) > at py4j.commands.CallCommand.execute(CallCommand.java:79) > at py4j.GatewayConnection.run(GatewayConnection.java:207) > at java.lang.Thread.run(Thread.java:744) > > >>> > > -- > Donald Drake > Drake Consulting > http://www.drakeconsulting.com/ > http://www.MailLaunder.com/ > 800-733-2143 > -- Donald Drake Drake Consulting http://www.drakeconsulting.com/ http://www.MailLaunder.com/ 800-733-2143
