Can you not use the spark jobserver instead? Just submit your job to the job server who already has the sparkcontext initialized in it, it would make it much easier i think.
Thanks Best Regards On Mon, Aug 31, 2015 at 2:16 PM, Pranay Tonpay <pranay.ton...@impetus.co.in> wrote: > thx for the inputs Igor,, i am actually building an Analytics layer ( 'As > a service model' using Spark as the backend engine ) and hence i am > implementing it this way... Initially, i was opening the spark-contenxt in > the JVM that i had spawned ( without even using Spark-submit ) and adding > all the jars etc in it ( as you have described )... It worked like a charm, > when i used Spark cluster as the master, but then when i tried to pass in > "yarn-client" instead, it started giving some issues ( hung jobs etc etc > )... On reading some comments on forum, i read that we should use > spark-submit and so i changed my approach... let me check more on the 2nd > point and see... > > > thx again for your prompt responses... BTW, do you see any issues in > invoking spark-submit in the fashion i am attempting ? ( via Runtime-exec > thing as i described ) > > > thx > > pranay > > > ------------------------------ > *From:* Igor Berman <igor.ber...@gmail.com> > *Sent:* Monday, August 31, 2015 1:39 PM > *To:* Pranay Tonpay > *Cc:* user@spark.apache.org > *Subject:* Re: spark-submit issue > > 1. think once again if you want to call spark submit in such way...I'm not > sure why you do it, but please consider just opening spark context inside > your jvm(you need to add spark jars to classpath..) > 2. use https://commons.apache.org/proper/commons-exec/ with > PumpStreamHandler > > On 31 August 2015 at 10:42, Pranay Tonpay <pranay.ton...@impetus.co.in> > wrote: > >> Igor,, this seems to be the cause, however i am not sure at the moment >> how to resolve it ... what i tried just now was that after " >> >> SparkSubmitDriverBootstrapper" process reaches the hung stage... i went >> inside /proc/<pid>/fd .... and just tailed "2" (stderr) and the process >> immediately exits . >> >> >> *From:* Igor Berman <igor.ber...@gmail.com> >> *Sent:* Monday, August 31, 2015 12:41 PM >> *To:* Pranay Tonpay >> *Cc:* user >> *Subject:* Re: spark-submit issue >> >> might be you need to drain stdout/stderr of subprocess...otherwise >> subprocess can deadlock >> http://stackoverflow.com/questions/3054531/correct-usage-of-processbuilder >> >> On 27 August 2015 at 16:11, pranay <pranay.ton...@impetus.co.in> wrote: >> >>> I have a java program that does this - (using Spark 1.3.1 ) Create a >>> command >>> string that uses "spark-submit" in it ( with my Class file etc ), and i >>> store this string in a temp file somewhere as a shell script Using >>> Runtime.exec, i execute this script and wait for its completion, using >>> process.waitFor Doing ps -ef shows me SparkSubmitDriverBootstrapper , the >>> script running my class ... parent child relationship.. >>> >>> The job gets triggered on spark-cluster and gets over but >>> SparkSubmitDriverBootstrapper still shows up, due to this the >>> process.waitFor never comes out and i can't detect the execution end... >>> >>> If i run the /temp file independently. things work file... only when i >>> trigger /temp scrict inside Runtime.exec , this issue occurs... Any >>> comments >>> ? >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-issue-tp24474.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> ------------------------------ >> >> >> >> >> >> >> NOTE: This message may contain information that is confidential, >> proprietary, privileged or otherwise protected by law. The message is >> intended solely for the named addressee. If received in error, please >> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, interception or interference. >> > > > ------------------------------ > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. >
