Re: [DISCUSS] Auto scaling support for structured streaming

Hi, Anyone? Scaling for different loads in a structured streaming app should be a trivial requirement for users... Thanks! Nimrod בתאריך יום ג׳, 9 ביולי 2024, 10:20, מאת Nimrod Ofek ‏: > PMC members, can someone please push this thing forward? > > Thanks! > Nimrod > > בתאריך יום ג׳, 9 ביולי 202

Re: [DISCUSS] Auto scaling support for structured streaming

PMC members, can someone please push this thing forward? Thanks! Nimrod בתאריך יום ג׳, 9 ביולי 2024, 01:57, מאת Pavan Kotikalapudi ‏< pkotikalap...@twilio.com>: > Definitely!. We internally use it extensively in all our apps and would > love to get community feedback. > > I think we have enough

Re: [DISCUSS] Auto scaling support for structured streaming

Definitely!. We internally use it extensively in all our apps and would love to get community feedback. I think we have enough work done to move this feature forward. We had discussion and vote threads already published in the past, but we need enough backing/votes of the PMC members to take it to

Re: [DISCUSS] Auto scaling support for structured streaming

Hi, Thanks Pavan. I think that the change is very important due to the amount of Spark structured streaming apps running today out there... IMHO this should be introduced in the upcoming Spark 4.0.0 version as an experimental feature for evaluation by the community... What should be the next ste

Re: [DISCUSS] Auto scaling support for structured streaming

Hi, I have taken up the responsibility for the development of that feature right now. Here is the current work https://github.com/apache/spark/pull/42352 last active email thread (maybe you want to reply to this): Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[DISCUSS] Auto scaling support for structured streaming

Hi, I remember there was a discussion about better supporting auto scaling for structured streaming. Is there anything happening with that for the upcoming Spark 4.0 release? Will there be support for auto scaling (at least on K8s) spark structured streaming apps? Thanks, Nimrod