Hi, Thanks Pavan.
I think that the change is very important due to the amount of Spark structured streaming apps running today out there... IMHO this should be introduced in the upcoming Spark 4.0.0 version as an experimental feature for evaluation by the community... What should be the next steps to make sure the community gets this important feature, at least to evaluate? How can the community experiment with it to decide if it's good enough for production use? Thanks! Nimrod On Mon, Jul 8, 2024 at 4:10 PM Pavan Kotikalapudi <pkotikalap...@twilio.com> wrote: > Hi, > > I have taken up the responsibility for the development of that feature > right now. > > Here is the current work https://github.com/apache/spark/pull/42352 > > last active email thread (maybe you want to reply to this): Re: Vote on > Dynamic resource allocation for structured streaming [SPARK-24815] > <https://lists.apache.org/thread/wpvtvf4w3zygtkfgq4sthbf00y5pqxvr> > > doc: Dynamic resource allocation for structured streaming > <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit> > . > > This still needs to be reviewed/approved by PMC members, so not sure about > the timeline at this point. > > Thanks, > > Pavan > > > > On Thu, Jul 4, 2024 at 10:46 AM Nimrod Ofek <ofek.nim...@gmail.com> wrote: > >> Hi, >> >> I remember there was a discussion about better supporting auto scaling >> for structured streaming. >> Is there anything happening with that for the upcoming Spark 4.0 release? >> Will there be support for auto scaling (at least on K8s) spark structured >> streaming apps? >> >> Thanks, >> Nimrod >> >
