Spark on K8s : property simillar to yarn.max.application.attempt

[Pralabh Kumar]

Hi Spark Team

I am running spark on K8s and looking for a
property/mechanism similar to  yarn.max.application.attempt . I know this
is not really a spark question , but i thought if anyone have faced the
similar issue,

Basically I want if my driver pod fails , it should be retried on a
different machine . Is there a way to do the same .

Regards
Pralabh Kumar

Re: Spark on K8s : property simillar to yarn.max.application.attempt

[Mich Talebzadeh]

Not as far as I know. If your driver pod fails, then you need to resubmit
the job. I cannot see what else can be done?


HTH


   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar  wrote:

> Hi Spark Team
>
> I am running spark on K8s and looking for a
> property/mechanism similar to  yarn.max.application.attempt . I know this
> is not really a spark question , but i thought if anyone have faced the
> similar issue,
>
> Basically I want if my driver pod fails , it should be retried on a
> different machine . Is there a way to do the same .
>
> Regards
> Pralabh Kumar
>

Re: Spark on K8s : property simillar to yarn.max.application.attempt

[Pablo Langa Blanco]

Hi Pralabh,

I understand that this is a question more related to K8s than to Spark
itself. In K8s you have a set of objects to control the state of your pods,
which allow you, among other things, to redeploy pods that have failed.
This would apply to the pod containing the driver container.
https://kubernetes.io/docs/concepts/workloads/

I hope it helps you!

Regards

On Fri, Feb 4, 2022 at 7:31 AM Mich Talebzadeh 
wrote:

> Not as far as I know. If your driver pod fails, then you need to resubmit
> the job. I cannot see what else can be done?
>
>
> HTH
>
>
>view my Linkedin profile
> 
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Fri, 4 Feb 2022 at 10:22, Pralabh Kumar  wrote:
>
>> Hi Spark Team
>>
>> I am running spark on K8s and looking for a
>> property/mechanism similar to  yarn.max.application.attempt . I know this
>> is not really a spark question , but i thought if anyone have faced the
>> similar issue,
>>
>> Basically I want if my driver pod fails , it should be retried on a
>> different machine . Is there a way to do the same .
>>
>> Regards
>> Pralabh Kumar
>>
>

Re: [VOTE] SPIP: Catalog API for view metadata

[L. C. Hsieh]

+1

On Thu, Feb 3, 2022 at 7:25 PM Chao Sun  wrote:
>
> +1 (non-binding). Looking forward to this feature!
>
> On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue  wrote:
>>
>> +1 for the SPIP. I think it's well designed and it has worked quite well at 
>> Netflix for a long time.
>>
>> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge  wrote:
>>>
>>> Hi Spark community,
>>>
>>> I’d like to restart the vote for the ViewCatalog design proposal (SPIP).
>>>
>>> The proposal is to add a ViewCatalog interface that can be used to load, 
>>> create, alter, and drop views in DataSourceV2.
>>>
>>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don’t think this is a good idea because …
>>>
>>> Thanks!
>>
>>
>>
>> --
>> Ryan Blue
>> Tabular

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPIP: Catalog API for view metadata

[huaxin gao]

+1 (non-binding)

On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh  wrote:

> +1
>
> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun  wrote:
> >
> > +1 (non-binding). Looking forward to this feature!
> >
> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue  wrote:
> >>
> >> +1 for the SPIP. I think it's well designed and it has worked quite
> well at Netflix for a long time.
> >>
> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge  wrote:
> >>>
> >>> Hi Spark community,
> >>>
> >>> I’d like to restart the vote for the ViewCatalog design proposal
> (SPIP).
> >>>
> >>> The proposal is to add a ViewCatalog interface that can be used to
> load, create, alter, and drop views in DataSourceV2.
> >>>
> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
> >>>
> >>> [ ] +1: Accept the proposal as an official SPIP
> >>> [ ] +0
> >>> [ ] -1: I don’t think this is a good idea because …
> >>>
> >>> Thanks!
> >>
> >>
> >>
> >> --
> >> Ryan Blue
> >> Tabular
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] SPIP: Catalog API for view metadata

[Yufei Gu]

+1 (non-binding)
Best,

Yufei

`This is not a contribution`


On Fri, Feb 4, 2022 at 11:54 AM huaxin gao  wrote:

> +1 (non-binding)
>
> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh  wrote:
>
>> +1
>>
>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun  wrote:
>> >
>> > +1 (non-binding). Looking forward to this feature!
>> >
>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue  wrote:
>> >>
>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>> well at Netflix for a long time.
>> >>
>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge  wrote:
>> >>>
>> >>> Hi Spark community,
>> >>>
>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>> (SPIP).
>> >>>
>> >>> The proposal is to add a ViewCatalog interface that can be used to
>> load, create, alter, and drop views in DataSourceV2.
>> >>>
>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>> >>>
>> >>> [ ] +1: Accept the proposal as an official SPIP
>> >>> [ ] +0
>> >>> [ ] -1: I don’t think this is a good idea because …
>> >>>
>> >>> Thanks!
>> >>
>> >>
>> >>
>> >> --
>> >> Ryan Blue
>> >> Tabular
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] SPIP: Catalog API for view metadata

[Terry Kim]

+1 (non-binding). Thanks John!

Terry

On Fri, Feb 4, 2022 at 4:13 PM Yufei Gu  wrote:

> +1 (non-binding)
> Best,
>
> Yufei
>
> `This is not a contribution`
>
>
> On Fri, Feb 4, 2022 at 11:54 AM huaxin gao  wrote:
>
>> +1 (non-binding)
>>
>> On Fri, Feb 4, 2022 at 11:40 AM L. C. Hsieh  wrote:
>>
>>> +1
>>>
>>> On Thu, Feb 3, 2022 at 7:25 PM Chao Sun  wrote:
>>> >
>>> > +1 (non-binding). Looking forward to this feature!
>>> >
>>> > On Thu, Feb 3, 2022 at 2:32 PM Ryan Blue  wrote:
>>> >>
>>> >> +1 for the SPIP. I think it's well designed and it has worked quite
>>> well at Netflix for a long time.
>>> >>
>>> >> On Thu, Feb 3, 2022 at 2:04 PM John Zhuge  wrote:
>>> >>>
>>> >>> Hi Spark community,
>>> >>>
>>> >>> I’d like to restart the vote for the ViewCatalog design proposal
>>> (SPIP).
>>> >>>
>>> >>> The proposal is to add a ViewCatalog interface that can be used to
>>> load, create, alter, and drop views in DataSourceV2.
>>> >>>
>>> >>> Please vote on the SPIP until Feb. 9th (Wednesday).
>>> >>>
>>> >>> [ ] +1: Accept the proposal as an official SPIP
>>> >>> [ ] +0
>>> >>> [ ] -1: I don’t think this is a good idea because …
>>> >>>
>>> >>> Thanks!
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Ryan Blue
>>> >> Tabular
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>