Hi, I have taken up the responsibility for the development of that feature right now.
Here is the current work https://github.com/apache/spark/pull/42352 last active email thread (maybe you want to reply to this): Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815] <https://lists.apache.org/thread/wpvtvf4w3zygtkfgq4sthbf00y5pqxvr> doc: Dynamic resource allocation for structured streaming <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit> . This still needs to be reviewed/approved by PMC members, so not sure about the timeline at this point. Thanks, Pavan On Thu, Jul 4, 2024 at 10:46 AM Nimrod Ofek <ofek.nim...@gmail.com> wrote: > Hi, > > I remember there was a discussion about better supporting auto scaling for > structured streaming. > Is there anything happening with that for the upcoming Spark 4.0 release? > Will there be support for auto scaling (at least on K8s) spark structured > streaming apps? > > Thanks, > Nimrod >
