Unsubscribe

Unsubscribe

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Hi Jay As far as I am aware in Spark 2.4.4, there is no feature to enable executor decommissioning with graceful shutdown, nor is there a way to specify a timeout for forcefully killing executors. These were introduced in Spark 3.0. HTH Mich Talebzadeh, Architect | Data Engineer | Data Scienc

Re: [DISCUSS] Support spark.ml on Spark Connect

Thank you for working on this! Xiao Martin Grund 于2024年10月10日周四 03:01写道： > > Hi Bobby, > > Awesome to see the proposal! I'm very much looking forward to the > contributions! > > Martin > > On Thu, Oct 10, 2024 at 4:16 AM Ángel > wrote: > >> You have my vote (btw, great idea, ML is so sexy now

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Thank you for your prompt reply. By the way, I am utilizing Spark 2.4.4. My question is as the following: When the Spark driver tries to remove executors from Kubernetes, it invokes pods().delete() without specifying a grace period, regardless of whether it's due to job failure or success. If this

Unsubscribe

Unsubscribe

Unsubscribe

Unsubscribe Von meinem iPhone gesendet - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS?] Adding some empty string check in to_date built-in function + warning in documentation

Didn't know about that. I'll have a look at it and check whether fix the issue or not. Thanks El jue, 10 oct 2024, 13:29, Wenchen Fan escribió: > There is a `try_to_timestamp` function but not `try_to_date`, we should > probably add it for users who don't want to get runtime errors when > proces

Re: [DISCUSS?] Adding some empty string check in to_date built-in function + warning in documentation

Oh sorry, I misunderstood the issue. It's not about the user-facing error, but the performance issue when the `to_date` function deals with invalid date strings. This is unfortunately not easy to fix, as Spark relies on the JDK library to parse datetime strings, and we can only know if a string is

Re: [DISCUSS?] Adding some empty string check in to_date built-in function + warning in documentation

There is a `try_to_timestamp` function but not `try_to_date`, we should probably add it for users who don't want to get runtime errors when processing big dataset. On Thu, Oct 10, 2024 at 11:05 AM Ángel wrote: > Hi, > > I opened a Jira ticket back in August, but it seems to have been > overlooke

Re: [DISCUSS] Support spark.ml on Spark Connect

Hi Bobby, Awesome to see the proposal! I'm very much looking forward to the contributions! Martin On Thu, Oct 10, 2024 at 4:16 AM Ángel wrote: > You have my vote (btw, great idea, ML is so sexy nowadays 😉) > > El jue, 10 oct 2024 a las 3:19, Bobby () escribió: > >> Hi, >> >> I'd like to start

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

to be clear are you referring to these spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true thanks Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD Imperial College London <

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Do you know by any chance if that config also applies to Databricks? El jue, 10 oct 2024 a las 10:02, Ángel () escribió: > Thanks a lot for the clarification. Interesting... I've never needed it, > even though I've been using Spark for over 8 years. > > El jue, 10 oct 2024 a las 9:21, Liu Cao ()

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

Thanks a lot for the clarification. Interesting... I've never needed it, even though I've been using Spark for over 8 years. El jue, 10 oct 2024 a las 9:21, Liu Cao () escribió: > I’m unclear on what the exact issue the OP ran into. > > But if we are talking about decommission, just one side note

Re: [Question] Why driver doesn't shutdown executors gracefully on k8s?

I’m unclear on what the exact issue the OP ran into. But if we are talking about decommission, just one side note: The decommission feature has been in spark for a while, and deco