Does the spark UI show your program running? (http://spark-masterIP:8118). If the program is listed as running you should be able to see details via the UI. In my experience there are 3 sets of logs -- the log where you're running your program (the driver), the log on the master node, and the log on each executor. The master log often has very useful details when one of your slave executors has an issue. Then you can go and read the logs on that machine. Of course, if you have a small number of workers in your cluster you can just read all the logs. That's just general debugging advice... (I also find it useful to do rdd.partitions.size before anything else to check how many partitions the RDD is actually partitioned to...)
On Tue, May 27, 2014 at 2:48 PM, Min Li <limin...@gmail.com> wrote: > Hi all, > > I've a single machine with 8 cores and 8g mem. I've deployed the > standalone spark on the machine and successfully run the examples. > > Now I'm trying to write some simple java codes. I just read a local file > (23M) into string list and use JavaRDD<String> rdds = > sparkContext.paralellize() method to get the corresponding rdd. And I asked > to run rdds.count(). But the program just stopped on the count(). The last > log info is: > > 14/05/27 14:13:16 INFO SparkContext: Starting job: count at RDDTest.java:40 > 14/05/27 14:13:16 INFO DAGScheduler: Got job 0 (count at RDDTest.java:40) > with 2 output partitions (allowLocal=false) > 14/05/27 14:13:16 INFO DAGScheduler: Final stage: Stage 0 (count at > RDDTest.java:40) > 14/05/27 14:13:16 INFO DAGScheduler: Parents of final stage: List() > 14/05/27 14:13:16 INFO DAGScheduler: Missing parents: List() > 14/05/27 14:13:16 INFO DAGScheduler: Submitting Stage 0 > (ParallelCollectionRDD[0] at parallelize at RDDTest.java:37), which has no > missing parents > 14/05/27 14:13:16 INFO SparkDeploySchedulerBackend: Connected to Spark > cluster with app ID app-20140527141316-0003 > 14/05/27 14:13:16 INFO AppClient$ClientActor: Executor added: > app-20140527141316-0003/0 on worker-20140526221107-spark-35303 > (spark:35303) with 8 cores > 14/05/27 14:13:16 INFO SparkDeploySchedulerBackend: Granted executor ID > app-20140527141316-0003/0 on hostPort spark:35303 with 8 cores, 1024.0 MB > RAM > 14/05/27 14:13:16 INFO AppClient$ClientActor: Executor updated: > app-20140527141316-0003/0 is now RUNNING > 14/05/27 14:13:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage > 0 (ParallelCollectionRDD[0] at parallelize at RDDTest.java:37) > 14/05/27 14:13:16 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks > 14/05/27 14:13:17 INFO SparkDeploySchedulerBackend: Registered executor: > Actor[akka.tcp://sparkExecutor@spark:34279/user/Executor#196489168] with > ID 0 > 14/05/27 14:13:17 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on > executor 0: spark (PROCESS_LOCAL) > 14/05/27 14:13:17 INFO TaskSetManager: Serialized task 0.0:0 as 12993529 > bytes in 127 ms > 14/05/27 14:13:17 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on > executor 0: spark (PROCESS_LOCAL) > 14/05/27 14:13:17 INFO TaskSetManager: Serialized task 0.0:1 as 13006417 > bytes in 74 ms > 14/05/27 14:13:17 INFO BlockManagerMasterActor$BlockManagerInfo: > Registering block manager spark:37617 with 589.2 MB RAM > > I tried to figure out what's going on, but just can't. Could any please > give me some suggestions and point out some possible issues? > > Best Regards, > Min >
