Fwd: Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Spark Dev, I have extended traditional DRA to work for structured streaming use-case. Here is an initial Implementation draft PR https://github.com/apache/spark/pull/42352 and design doc: https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing Please

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damag

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

sponsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. &g

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Aug 2023 at 14:58, Martin Andersson > wrote: > >> IMO, using any kind of machine learning or AI for DRA is overkill. The >> effort involved would be considerable and likely counterproductive, >> compared to a more conventional approach of comparing the rate of incoming >

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

lB4wR3SukiIw$> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Dev community, Just bumping to see if there are more reviews to evaluate this idea of adding auto-scaling to structured streaming. Thanks again, Pavan On Wed, Aug 23, 2023 at 2:49 PM Pavan Kotikalapudi wrote: > Thanks for the review Mich. > > I have updated the Q4 with a

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Here is an initial Implementation draft PR https://github.com/apache/spark/pull/42352 and design doc: https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing On Sun, Nov 12, 2023 at 5:24 PM Pavan Kotikalapudi wrote: > Hi Dev community, > > Jus

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

=sharing Thank you, Pavan On Mon, Nov 13, 2023 at 6:57 AM Pavan Kotikalapudi wrote: > > > Here is an initial Implementation draft PR > https://github.com/apache/spark/pull/42352 and design doc: > https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/e

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

esponsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruc

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

damages arising from > such loss, damage or destruction. > > > > > On Tue, 8 Aug 2023 at 01:30, Pavan Kotikalapudi > wrote: > >> Hi Spark Dev, >> >> I have extended traditional DRA to work for structured streaming >> use-case. >> >> Her

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for proposing and voting for the feature Mich. adding some references to the thread. - Jira ticket - SPARK-24815 - Design Doc

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 If my vote counts. Does only spark PMC votes count? Thanks, Pavan On Thu, Jan 18, 2024 at 3:19 AM Adam Hobbs wrote: > +1 > -- > *From:* Pavan Kotikalapudi > *Sent:* Thursday, January 18, 2024 4:19:32 AM > *To:* Spark dev list > *Subject:* R

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Here is the link to the voting thread https://lists.apache.org/thread/rlwqrw6ddxdkbvkp78kpd0zgvglgbbp8. Thank you, Pavan On Wed, Jan 17, 2024 at 7:15 PM Pavan Kotikalapudi wrote: > Thanks for the +1, I will propose voting in a new thread now. > > - Pavan > > On Wed, Jan 17,

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

/owner re-open the PR? ( I guess people only with admin access to the repository can do that). Thank you, Pavan On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell wrote: > +1 > > On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: > > Thanks for proposing and voting for the feature

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

; Braun > <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> > )". > > > On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi > wrote: > >&g

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Mich, We have five +1s till now. Mich Talebzadeh Adam Hobbs Pavan Kotikalapudi Krystal Mitchell Sona Torosyan (few more in github pr) +0: None -1: None Does it pass the required condition as approved? Not sure of that though, nothing about minimum required is mentioned in the past

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for the pointers Mich, will wait for Jungtaek Lee or any other PMC members to respond. aggregating upvotes to this email thread +6 Mich Talebzadeh Adam Hobbs Pavan Kotikalapudi Krystal Mitchell Sona Torosyan Aaron Kern Thank you, Pavan On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh

Re: Please unlock Jira ticket for SPARK-24815, Dynamic resource allocation for structured streaming

Thanks Yuming. On Mon, Feb 26, 2024 at 9:55 PM Yuming Wang wrote: > Unlocked. > > On Tue, Feb 27, 2024 at 11:47 AM Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> >> Hi, >> >> Can a committer please unlock this SPIP? It is for Dynamic resource >> allocation for structured streaming th

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

is worth one-thousand >> expert opinions (Werner >> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >> Braun >> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von > Braun > <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDn

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

cording to the WIP PR > <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrA6JBQ8S0$> > . > > @Pavan Kotikalapudi > I see you've added the copyright of

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

gt;> Pavan >> >> >> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> I'm happy to, but it looks like I need to check one more thing about the >>> license, according to the WIP PR >>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8f

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

nd also clearly describe the limitation. > (Also a warning in the codebase that this must not be enabled by default.) > > > On Tue, Mar 26, 2024 at 7:02 PM Pavan Kotikalapudi < > pkotikalap...@twilio.com> wrote: > >> Hi Bhuwan, >> >> Glad to hear back from you! Ver

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

hugely benefit the community utilizing structured streaming applications in their data pipelines. Looking forward to hear back from you. Thank you, Pavan On Thu, Mar 28, 2024 at 3:38 PM Pavan Kotikalapudi wrote: > Hi Jungtaek, > > Sorry for the late reply. > > I understand the

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

k if adding such kinds of configs to automate the process of calculating the thresholds and their respective configs makes sense? Thank you, Pavan On Thu, Mar 28, 2024 at 3:38 PM Pavan Kotikalapudi wrote: > Hi Jungtaek, > > Sorry for the late reply. > > I understand the concern

Re: [DISCUSS] Auto scaling support for structured streaming

Hi, I have taken up the responsibility for the development of that feature right now. Here is the current work https://github.com/apache/spark/pull/42352 last active email thread (maybe you want to reply to this): Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Re: [DISCUSS] Auto scaling support for structured streaming

the community experiment with it to decide if it's good enough for > production use? > > Thanks! > Nimrod > > On Mon, Jul 8, 2024 at 4:10 PM Pavan Kotikalapudi < > pkotikalap...@twilio.com> wrote: > >> Hi, >> >> I have taken up the responsibili

Re: ASF board report draft for February 2025

Pavan On Tue, Feb 11, 2025 at 4:46 AM Mich Talebzadeh wrote: > Hi all, > > ok this DRA has already had a thread here > > Vote on Dynamic resource allocation for structured streaming [SPARK-24815] > > I recall I asked a committer to open the PR and it was opened and > c