I'm also interested in the solution to this. Thanks, Mike
On Mon, Dec 29, 2014 at 12:01 PM, hutashan [via Apache Spark User List] < [email protected]> wrote: > > Hello All, > > I need to clean up app folder(include app downloaded jar) in spark under > work folder. > I have tried to set below configuration in spark env. but it is not > working as expected. > > SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true > -Dspark.worker.cleanup.interval=10" > > I cannot clear work folder blindly because of some other job may run on > the same time. > > Is there any way to find out below :- > 1) Clear only my app folder under work folder > 2) In executor level can we set work directories. So, that i will clean > that folder after my job completed. > 3) Is there way to find of app id of my application. I tried to get master > object in code but it is throwing actor error. > 4) Is there any correction required on below configuration > SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true > -Dspark.worker.cleanup.interval=10" > > > Thanks in advance > Hutashan. > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/Clean-up-app-folders-in-worker-nodes-tp20889.html > To unsubscribe from Apache Spark User List, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bXEwMDFrQGdtYWlsLmNvbXwxfDgxMTQwOTE5Nw==> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Clean-up-app-folders-in-worker-nodes-tp20889p20890.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
