Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 From: Pavan Kotikalapudi Sent: Thursday, January 18, 2024 4:19:32 AM To: Spark dev list Subject: Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815] CAUTION: This email originated from outside of the organisation. Do not click lin

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for proposing and voting for the feature Mich. adding some references to the thread. - Jira ticket - SPARK-24815 - Design Doc

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 for me (non binding) *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monet

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

Hi, We are internally exploring adding support for dynamically changing the resource profile of a stage based on runtime characteristics. This includes failures due to OOM and the like, slowness due to excessive GC, resource wastage due to excessive overprovisioning, etc. Essentially handles sca

Re: [Spark-Core] Improving Reliability of spark when Executors OOM

It is interesting. I think there are definitely some discussion points around this.  reliability vs performance is always a trade off and its great it doesn't fail but if it doesn't meet someone's SLA now that could be as bad if its hard to figure out why.   I think if something like this kicks

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for the +1, I will propose voting in a new thread now. - Pavan On Wed, Jan 17, 2024 at 5:28 PM Mich Talebzadeh wrote: > I think we have discussed this enough and I consider it as a useful > feature.. I propose a vote on it. > > + 1 for me > > Mich Talebzadeh, > Dad | Technologist | Solut

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

I think we have discussed this enough and I consider it as a useful feature.. I propose a vote on it. + 1 for me Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile h