Definitely!. We internally use it extensively in all our apps and would love to get community feedback.
I think we have enough work done to move this feature forward. We had discussion and vote threads already published in the past, but we need enough backing/votes of the PMC members to take it to completion. cc: @Jungtaek Lim <kabhwan.opensou...@gmail.com>, @Mich Talebzadeh <mich.talebza...@gmail.com> mentors of this effort. Cheers, Pavan On Mon, Jul 8, 2024 at 10:33 AM Nimrod Ofek <ofek.nim...@gmail.com> wrote: > Hi, > > Thanks Pavan. > > I think that the change is very important due to the amount of Spark > structured streaming apps running today out there... > IMHO this should be introduced in the upcoming Spark 4.0.0 version as an > experimental feature for evaluation by the community... > > What should be the next steps to make sure the community gets this > important feature, at least to evaluate? > How can the community experiment with it to decide if it's good enough for > production use? > > Thanks! > Nimrod > > On Mon, Jul 8, 2024 at 4:10 PM Pavan Kotikalapudi < > pkotikalap...@twilio.com> wrote: > >> Hi, >> >> I have taken up the responsibility for the development of that feature >> right now. >> >> Here is the current work https://github.com/apache/spark/pull/42352 >> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIGKj5coZ$> >> >> last active email thread (maybe you want to reply to this): Re: Vote on >> Dynamic resource allocation for structured streaming [SPARK-24815] >> <https://urldefense.com/v3/__https://lists.apache.org/thread/wpvtvf4w3zygtkfgq4sthbf00y5pqxvr__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIPycqhCt$> >> >> doc: Dynamic resource allocation for structured streaming >> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIN-NBSBO$> >> . >> >> This still needs to be reviewed/approved by PMC members, so not sure >> about the timeline at this point. >> >> Thanks, >> >> Pavan >> >> >> >> On Thu, Jul 4, 2024 at 10:46 AM Nimrod Ofek <ofek.nim...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I remember there was a discussion about better supporting auto scaling >>> for structured streaming. >>> Is there anything happening with that for the upcoming Spark 4.0 >>> release? >>> Will there be support for auto scaling (at least on K8s) spark >>> structured streaming apps? >>> >>> Thanks, >>> Nimrod >>> >>
