Re: spark-submit stuck and no output in console

Our hadoop NFS Gateway seems to be malfunctioning. I basically restart it. Now spark jobs have resumed successfully. Problem solved.

Re: spark-submit stuck and no output in console

I would suggest a couple of things to try A. Try running the example program with master as local[*]. See if spark can run locally or not. B. Check spark master and worker logs. C. Check if normal hadoop jobs can be run properly on the cluster. D. Check spark master webui and see health of cluster

Re: spark-submit stuck and no output in console

Sonal, SparkPi couldn't run as well. Stuck to the screen with no output hadoop-user@yks-hadoop-m01:/usr/local/spark$ ./bin/run-example SparkPi On Tue, Nov 17, 2015 at 12:22 PM, Steve Loughran wrote: > 48 hours is one of those kerberos warning times (as is 24h, 72h and 7 > days) Does this

Re: spark-submit stuck and no output in console

On 17 Nov 2015, at 09:54, Kayode Odeyemi mailto:drey...@gmail.com>> wrote: Initially, I submitted 2 jobs to the YARN cluster which was running for 2 days and suddenly stops. Nothing in the logs shows the root cause. 48 hours is one of those kerberos warning times (as is 24h, 72h and 7 days) ..

Re: spark-submit stuck and no output in console

How did the example spark jobs go? SparkPI etc..? Best Regards, Sonal Founder, Nube Technologies Reifier at Strata Hadoop World Reifier at Spark Summit 2015

Re: spark-submit stuck and no output in console

Thanks for the reply Sonal. I'm on JDK 7 (/usr/lib/jvm/java-7-oracle) My env is a YARN cluster made of 7 nodes (6 datanodes/ node manager, 1 namenode/resource manager). On the namenode, is where I executed the spark-submit job while on one of the datanodes, I executed 'hadoop fs -put /binstore

Re: spark-submit stuck and no output in console

Could it be jdk related ? Which version are you on? Best Regards, Sonal Founder, Nube Technologies Reifier at Strata Hadoop World Reifier at Spark Summit 2015

Re: spark-submit stuck and no output in console

Anyone experienced this issue as well? On Mon, Nov 16, 2015 at 8:06 PM, Kayode Odeyemi wrote: > > Or are you saying that the Java process never even starts? > > > Exactly. > > Here's what I got back from jstack as expected: > > hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 > 3131

Re: spark-submit stuck and no output in console

> Or are you saying that the Java process never even starts? Exactly. Here's what I got back from jstack as expected: hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 31316: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used

Re: spark-submit stuck and no output in console

He means for you to use jstack to obtain a stacktrace of all of the threads. Or are you saying that the Java process never even starts? On Mon, Nov 16, 2015 at 7:48 AM, Kayode Odeyemi wrote: > Spark 1.5.1 > > The fact is that there's no stack trace. No output from that command at > all to the co

Re: spark-submit stuck and no output in console

Spark 1.5.1 The fact is that there's no stack trace. No output from that command at all to the console. This is all I get: hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ tail -1 /tmp/spark-profile-job.log nohup: ignoring input /usr/local/spark/bin/spark-class: line 76: 29516 Killed "$RUNN

Re: spark-submit stuck and no output in console

Which release of Spark are you using ? Can you take stack trace and pastebin it ? Thanks On Mon, Nov 16, 2015 at 5:50 AM, Kayode Odeyemi wrote: > ./spark-submit --class com.migration.UpdateProfiles --executor-memory 8g > ~/migration-profiles-0.1-SNAPSHOT.jar > > is stuck and outputs nothing to