1. think once again if you want to call spark submit in such way...I'm not sure why you do it, but please consider just opening spark context inside your jvm(you need to add spark jars to classpath..) 2. use https://commons.apache.org/proper/commons-exec/ with PumpStreamHandler
On 31 August 2015 at 10:42, Pranay Tonpay <pranay.ton...@impetus.co.in> wrote: > Igor,, this seems to be the cause, however i am not sure at the moment how > to resolve it ... what i tried just now was that after " > > SparkSubmitDriverBootstrapper" process reaches the hung stage... i went > inside /proc/<pid>/fd .... and just tailed "2" (stderr) and the process > immediately exits . > > > *From:* Igor Berman <igor.ber...@gmail.com> > *Sent:* Monday, August 31, 2015 12:41 PM > *To:* Pranay Tonpay > *Cc:* user > *Subject:* Re: spark-submit issue > > might be you need to drain stdout/stderr of subprocess...otherwise > subprocess can deadlock > http://stackoverflow.com/questions/3054531/correct-usage-of-processbuilder > > On 27 August 2015 at 16:11, pranay <pranay.ton...@impetus.co.in> wrote: > >> I have a java program that does this - (using Spark 1.3.1 ) Create a >> command >> string that uses "spark-submit" in it ( with my Class file etc ), and i >> store this string in a temp file somewhere as a shell script Using >> Runtime.exec, i execute this script and wait for its completion, using >> process.waitFor Doing ps -ef shows me SparkSubmitDriverBootstrapper , the >> script running my class ... parent child relationship.. >> >> The job gets triggered on spark-cluster and gets over but >> SparkSubmitDriverBootstrapper still shows up, due to this the >> process.waitFor never comes out and i can't detect the execution end... >> >> If i run the /temp file independently. things work file... only when i >> trigger /temp scrict inside Runtime.exec , this issue occurs... Any >> comments >> ? >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-issue-tp24474.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > ------------------------------ > > > > > > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. >
