Re: [VOTE] SPIP: Real-Time Mode in Apache Spark Structured Streaming

gt;> Hi all, >>>>> >>>>>> >>>>> >>>>>> I would like to start a vote on the new real-time mode in >>>>> Apache Spark >>>>> >>>>>> Structured Streaming. >>>>> >>>>>> >>>>> >>>>>> Discussion thread: >>>>> >>>>>> >>>>> https://lists.apache.org/thread/ovmfbzfkc3t9odvv5gs75fhpvdckn90f >>>>> >>>>>> SPIP: >>>>> https://docs.google.com/document/d/1CvJvtlTGP6TwQIT4kW6GFT1JbdziAYOBvt60ybb7Dw8/edit?tab=t.0#heading=h.ulas5788cm9t >>>>> >>>>>> JIRA: https://issues.apache.org/jira/browse/SPARK-52330 >>>>> >>>>>> >>>>> >>>>>> Please vote on the SPIP for the next 72 hours: >>>>> >>>>>> >>>>> >>>>>> [ ] +1: Accept the proposal as an official SPIP >>>>> >>>>>> [ ] +0 >>>>> >>>>>> [ ] -1: I don’t think this is a good idea because … >>>>> >>>>>> >>>>> >>>>>> >>>>> - >>>>> >>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>> >>>>>> >>>>> >>>>> - >>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>> >>>>> -- Liu Cao

Re: [VOTE] Release Apache Spark K8s Operator 0.3.0 (RC1)

gt; https://dist.apache.org/repos/dist/dev/spark/spark-kubernetes-operator-0.3.0-rc.1/ >> >>>> >> >>>> LIST OF ISSUES: >> >>>> https://issues.apache.org/jira/projects/SPARK/versions/12355949 >> >>>> >> >>>> Thanks, >> >>>> Dongjoon. >> >>> >> >>> - >> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>> >> >>> >> >> >> > >> > - >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> -- Liu Cao

Re: [VOTE] Release Apache Spark K8s Operator 0.2.0 (RC1)

/github.com/apache/spark-kubernetes-operator/releases/tag/0.2.0-rc.1 >>>> > (commit: 85f17fbe5290db5fda90d7d4324381833d0eefa8) >>>> > >>>> > RELEASE FILES: >>>> > >>>> https://dist.apache.org/repos/dist/dev/spark/spark-kubernetes-operator-0.2.0-rc.1/ >>>> > >>>> > LIST OF ISSUES: >>>> > https://issues.apache.org/jira/projects/SPARK/versions/12355908 >>>> > >>>> > Thanks, >>>> > Dongjoon. >>>> >>>> - >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> > -- Liu Cao

Re: [VOTE] SPIP: Declarative Pipelines

27 >>>> >> > > >>>> >> > > Please vote on the SPIP for the next 72 hours: >>>> >> > > >>>> >> > > [ ] +1: Accept the proposal as an official SPIP >>>> >> > > [ ] +0 >>>> >> > > [ ] -1: I don’t think this is a good idea because … >>>> >> > > >>>> >> > > -Sandy >>>> >> > > >>>> >> > >>>> >> > >>>> >> >>>> >> - >>>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >> >>>> >>>> - >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> >>> >>> -- >>> John Zhuge >>> >> -- Liu Cao

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

t;>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>> expert opinions (Werner >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>> >>>>>> >>>>>> On Wed, 9 Oct 2024 at 16:43, Nicholas Chammas < >>>>>> nicholas.cham...@gmail.com> wrote: >>>>>> >>>>>>> Mich, >>>>>>> >>>>>>> Can you please share with the list where *exactly* you are citing >>>>>>> these configs from? >>>>>>> >>>>>>> As far as I can tell, these two configs don’t exist and have never >>>>>>> existed in the Spark codebase: >>>>>>> >>>>>>> spark.executor.decommission.enabled=true >>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>> >>>>>>> Where exactly are you getting this information from (and then >>>>>>> posting it to the list as advice)? Please be clear and provide specific >>>>>>> references. >>>>>>> >>>>>>> Nick >>>>>>> >>>>>>> >>>>>>> On Oct 9, 2024, at 1:20 PM, Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>> Before responding, what configuration parameters are you using to >>>>>>> make this work? >>>>>>> >>>>>>> spark.executor.decommission.enabled=true >>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>> spark.executor.decommission.forceKillTimeout=100s >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> >>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>> College London >>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>> London, United Kingdom >>>>>>> >>>>>>>view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>> >>>>>>> >>>>>>> On Wed, 9 Oct 2024 at 11:05, Jay Han wrote: >>>>>>> >>>>>>>> Hi spark community, >>>>>>>> I have such a question: Why driver doesn't shutdown executors >>>>>>>> gracefully on k8s. For instance, >>>>>>>> kubernetesClient.pods().withGracePeriod(100).delete(). >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best, >>>>>>>> Jay >>>>>>>> >>>>>>> >>>>>>> -- Liu Cao