On 18 Dec 2015, at 21:39, Andrew Or <and...@databricks.com<mailto:and...@databricks.com>> wrote:
Hi Roy, I believe Spark just gets its application ID from YARN, so you can just do `sc.applicationId`. If you listen for a spark start event you get the app ID, but not the real spark attempt ID; SPARK-11314 adds an extension point to the process in charge of the yarn app which will get both (with an appID of None for client-managed). to find a running app., you can use yarn list --appTypes SPARK to list all spark apps, then filter by name you can also set the conf option "spark.yarn.tags" to set one or more comma separated tags (e.g "production" which you can use for production code) Looking @ the yarn command, you can't currently filter the listing by tag (or by user & queue), even though those (and start time) are things you can filter by. you could create a YARN JIRA to add those features —but they could also be added to the spark CLI so you don't have to wait for Hadoop 2.9+ to use them. FWIW, I added something similarish to the slider app, where exporting the listings and app reports as parseable JSON turns out be invaluable for testing and other scripted operations. Your shell scripts and test runners can find the app, get its ID and then manage it without human intervention. So add that to whatever SPARK JIRA you create. for now though, set the tags in the config and then look for them in the "yarn list" output -Andrew 2015-12-18 0:14 GMT-08:00 Deepak Sharma <deepakmc...@gmail.com<mailto:deepakmc...@gmail.com>>: I have never tried this but there is yarn client api's that you can use in your spark program to get the application id. Here is the link to the yarn client java doc: http://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/yarn/client/api/YarnClient.html getApplications() is the method for your purpose here. Thanks Deepak On Fri, Dec 18, 2015 at 1:31 PM, Kyle Lin <kylelin2...@gmail.com<mailto:kylelin2...@gmail.com>> wrote: Hello there I have the same requirement. I submit a streaming job with yarn-cluster mode. If I want to shutdown this endless YARN application, I should find out the application id by myself and use "yarn appplication -kill <app_id>" to kill the application. Therefore, if I can get returned application id in my client program, it will be easy for me to kill YARN application from my client program. Kyle 2015-06-24 13:02 GMT+08:00 canan chen <ccn...@gmail.com<mailto:ccn...@gmail.com>>: I don't think there is yarn related stuff to access in spark. Spark don't depend on yarn. BTW, why do you want the yarn application id ? On Mon, Jun 22, 2015 at 11:45 PM, roy <rp...@njit.edu<mailto:rp...@njit.edu>> wrote: Hi, Is there a way to get Yarn application ID inside spark application, when running spark Job on YARN ? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-application-ID-for-Spark-job-on-Yarn-tp23429.html Sent from the Apache Spark User List mailing list archive at Nabble.com<http://nabble.com>. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> -- Thanks Deepak www.bigdatabig.com<http://www.bigdatabig.com/> www.keosha.net<http://www.keosha.net/>
