I would be cautious regarding use of spark.cleaner.ttl, as it can lead to confusing error messages if time-based cleaning deletes resources that are still needed. See my comment at https://issues.apache.org/jira/browse/SPARK-5594?focusedCommentId=14486034&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14486034
On Mon, May 11, 2015 at 10:17 AM, Ganelin, Ilya <[email protected] > wrote: > Also check out the spark.cleaner.ttl property. Otherwise, you will > accumulate shuffle metadata in the memory of the driver. > > > > Sent with Good (www.good.com) > > > > -----Original Message----- > *From: *Silvio Fiorito [[email protected]] > *Sent: *Monday, May 11, 2015 01:03 PM Eastern Standard Time > *To: *stanley; [email protected] > *Subject: *Re: Does long-lived SparkContext hold on to executor resources? > > You want to look at dynamic resource allocation, here: > http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation > > > > > > On 5/11/15, 11:23 AM, "stanley" <[email protected]> wrote: > > >I am building an analytics app with Spark. I plan to use long-lived > >SparkContexts to minimize the overhead for creating Spark contexts, which > in > >turn reduces the analytics query response time. > > > >The number of queries that are run in the system is relatively small each > >day. Would long lived contexts hold on to the executor resources when > there > >is no queries running? Is there a way to free executor resources in this > >type of use cases? > > > > > > > >-- > >View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Does-long-lived-SparkContext-hold-on-to-executor-resources-tp22848.html > >Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > >--------------------------------------------------------------------- > >To unsubscribe, e-mail: [email protected] > >For additional commands, e-mail: [email protected] > > > > ------------------------------ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the > intended recipient, you are hereby notified that any review, > retransmission, dissemination, distribution, copying or other use of, or > taking of any action in reliance upon this information is strictly > prohibited. If you have received this communication in error, please > contact the sender and delete the material from your computer. >
