Thanks for the reply Sonal. I'm on JDK 7 (/usr/lib/jvm/java-7-oracle)
My env is a YARN cluster made of 7 nodes (6 datanodes/ node manager, 1 namenode/resource manager). On the namenode, is where I executed the spark-submit job while on one of the datanodes, I executed 'hadoop fs -put /binstore /user/hadoop-user/' to dump 1TB of data into all the datanodes. That process is still running without hassle and it's only using 1.3 GB of 1.7g heap space. Initially, I submitted 2 jobs to the YARN cluster which was running for 2 days and suddenly stops. Nothing in the logs shows the root cause. On Tue, Nov 17, 2015 at 11:42 AM, Sonal Goyal <sonalgoy...@gmail.com> wrote: > Could it be jdk related ? Which version are you on? > > Best Regards, > Sonal > Founder, Nube Technologies <http://www.nubetech.co> > Reifier at Strata Hadoop World > <http://strataconf.com/big-data-conference-sg-2015/public/schedule/detail/44606> > Reifier at Spark Summit 2015 > <https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/> > > <http://in.linkedin.com/in/sonalgoyal> > > > > On Tue, Nov 17, 2015 at 2:48 PM, Kayode Odeyemi <drey...@gmail.com> wrote: > >> Anyone experienced this issue as well? >> >> On Mon, Nov 16, 2015 at 8:06 PM, Kayode Odeyemi <drey...@gmail.com> >> wrote: >> >>> >>> Or are you saying that the Java process never even starts? >>> >>> >>> Exactly. >>> >>> Here's what I got back from jstack as expected: >>> >>> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 >>> 31316: Unable to open socket file: target process not responding or >>> HotSpot VM not loaded >>> The -F option can be used when the target process is not responding >>> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 -F >>> Attaching to core -F from executable 31316, please wait... >>> Error attaching to core file: Can't attach to the core file >>> >>> >>> >> >> >> -- >> Odeyemi 'Kayode O. >> http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde >> > > -- Odeyemi 'Kayode O. http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde
