Followup question: the docs to make a new SparkContext require that I know where $SPARK_HOME is. However, I have no idea. Any idea where that might be?
On Sun, Jun 1, 2014 at 10:28 AM, Aaron Davidson <ilike...@gmail.com> wrote: > Gotcha. The easiest way to get your dependencies to your Executors would > probably be to construct your SparkContext with all necessary jars passed > in (as the "jars" parameter), or inside a SparkConf with setJars(). Avro is > a "necessary jar", but it's possible your application also needs to > distribute other ones to the cluster. > > An easy way to make sure all your dependencies get shipped to the cluster > is to create an assembly jar of your application, and then you just need to > tell Spark about that jar, which includes all your application's transitive > dependencies. Maven and sbt both have pretty straightforward ways of > producing assembly jars. > > > On Sat, May 31, 2014 at 11:23 PM, Russell Jurney <russell.jur...@gmail.com > > wrote: > >> Thanks for the fast reply. >> >> I am running CDH 4.4 with the Cloudera Parcel of Spark 0.9.0, in >> standalone mode. >> >> >> On Saturday, May 31, 2014, Aaron Davidson <ilike...@gmail.com> wrote: >> >>> First issue was because your cluster was configured incorrectly. You >>> could probably read 1 file because that was done on the driver node, but >>> when it tried to run a job on the cluster, it failed. >>> >>> Second issue, it seems that the jar containing avro is not getting >>> propagated to the Executors. What version of Spark are you running on? What >>> deployment mode (YARN, standalone, Mesos)? >>> >>> >>> On Sat, May 31, 2014 at 9:37 PM, Russell Jurney < >>> russell.jur...@gmail.com> wrote: >>> >>> Now I get this: >>> >>> scala> rdd.first >>> >>> 14/05/31 21:36:28 INFO spark.SparkContext: Starting job: first at >>> <console>:41 >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Got job 4 (first at >>> <console>:41) with 1 output partitions (allowLocal=true) >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Final stage: Stage 4 >>> (first at <console>:41) >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Parents of final stage: >>> List() >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Missing parents: List() >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Computing the requested >>> partition locally >>> >>> 14/05/31 21:36:28 INFO rdd.HadoopRDD: Input split: >>> hdfs://hivecluster2/securityx/web_proxy_mef/2014/05/29/22/part-m-00000.avro:0+3864 >>> >>> 14/05/31 21:36:28 INFO spark.SparkContext: Job finished: first at >>> <console>:41, took 0.037371256 s >>> >>> 14/05/31 21:36:28 INFO spark.SparkContext: Starting job: first at >>> <console>:41 >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Got job 5 (first at >>> <console>:41) with 16 output partitions (allowLocal=true) >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Final stage: Stage 5 >>> (first at <console>:41) >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Parents of final stage: >>> List() >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Missing parents: List() >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Submitting Stage 5 >>> (HadoopRDD[0] at hadoopRDD at <console>:37), which has no missing parents >>> >>> 14/05/31 21:36:28 INFO scheduler.DAGScheduler: Submitting 16 missing >>> tasks from Stage 5 (HadoopRDD[0] at hadoopRDD at <console>:37) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 >>> with 16 tasks >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:0 as >>> TID 92 on executor 2: hivecluster3 (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:0 >>> as 1294 bytes in 1 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:3 as >>> TID 93 on executor 1: hivecluster5.labs.lan (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:3 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:1 as >>> TID 94 on executor 4: hivecluster4 (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:1 >>> as 1294 bytes in 1 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:2 as >>> TID 95 on executor 0: hivecluster6.labs.lan (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:2 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:4 as >>> TID 96 on executor 3: hivecluster1.labs.lan (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:4 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:6 as >>> TID 97 on executor 2: hivecluster3 (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:6 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:5 as >>> TID 98 on executor 1: hivecluster5.labs.lan (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:5 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:8 as >>> TID 99 on executor 4: hivecluster4 (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:8 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:7 as >>> TID 100 on executor 0: hivecluster6.labs.lan (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:7 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:10 as >>> TID 101 on executor 3: hivecluster1.labs.lan (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:10 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:14 as >>> TID 102 on executor 2: hivecluster3 (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:14 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:9 as >>> TID 103 on executor 1: hivecluster5.labs.lan (NODE_LOCAL) >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Serialized task 5.0:9 >>> as 1294 bytes in 0 ms >>> >>> 14/05/31 21:36:28 INFO scheduler.TaskSetManager: Starting task 5.0:11 as >>> TID 104 on executor 4: hivecluster4 (N >>> >>> >> >> -- >> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome. >> com >> > > -- Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome.com
