Hi, Anyone? Scaling for different loads in a structured streaming app should be a trivial requirement for users...
Thanks! Nimrod בתאריך יום ג׳, 9 ביולי 2024, 10:20, מאת Nimrod Ofek <ofek.nim...@gmail.com >: > PMC members, can someone please push this thing forward? > > Thanks! > Nimrod > > בתאריך יום ג׳, 9 ביולי 2024, 01:57, מאת Pavan Kotikalapudi < > pkotikalap...@twilio.com>: > >> Definitely!. We internally use it extensively in all our apps and would >> love to get community feedback. >> >> I think we have enough work done to move this feature forward. >> We had discussion and vote threads already published in the past, but we >> need enough backing/votes of the PMC members to take it to completion. >> >> >> cc: @Jungtaek Lim <kabhwan.opensou...@gmail.com>, @Mich Talebzadeh >> <mich.talebza...@gmail.com> mentors of this effort. >> >> Cheers, >> >> Pavan >> >> >> >> >> On Mon, Jul 8, 2024 at 10:33 AM Nimrod Ofek <ofek.nim...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Thanks Pavan. >>> >>> I think that the change is very important due to the amount of Spark >>> structured streaming apps running today out there... >>> IMHO this should be introduced in the upcoming Spark 4.0.0 version as an >>> experimental feature for evaluation by the community... >>> >>> What should be the next steps to make sure the community gets this >>> important feature, at least to evaluate? >>> How can the community experiment with it to decide if it's good enough >>> for production use? >>> >>> Thanks! >>> Nimrod >>> >>> On Mon, Jul 8, 2024 at 4:10 PM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>>> Hi, >>>> >>>> I have taken up the responsibility for the development of that feature >>>> right now. >>>> >>>> Here is the current work https://github.com/apache/spark/pull/42352 >>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIGKj5coZ$> >>>> >>>> last active email thread (maybe you want to reply to this): Re: Vote >>>> on Dynamic resource allocation for structured streaming [SPARK-24815] >>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/wpvtvf4w3zygtkfgq4sthbf00y5pqxvr__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIPycqhCt$> >>>> >>>> doc: Dynamic resource allocation for structured streaming >>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIN-NBSBO$> >>>> . >>>> >>>> This still needs to be reviewed/approved by PMC members, so not sure >>>> about the timeline at this point. >>>> >>>> Thanks, >>>> >>>> Pavan >>>> >>>> >>>> >>>> On Thu, Jul 4, 2024 at 10:46 AM Nimrod Ofek <ofek.nim...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I remember there was a discussion about better supporting auto scaling >>>>> for structured streaming. >>>>> Is there anything happening with that for the upcoming Spark 4.0 >>>>> release? >>>>> Will there be support for auto scaling (at least on K8s) spark >>>>> structured streaming apps? >>>>> >>>>> Thanks, >>>>> Nimrod >>>>> >>>>
