+1 to rethink
Best,
Jingsong Lee
--
From:Stephan Ewen
Send Time:2019年9月10日(星期二) 15:40
To:dev
Cc:JingsongLee
Subject:Re: [DISCUSS] Support notifyOnMaster for notifyCheckpointComplete
Hi all!
I think it would be time to rethink
下午1:33写道：
> >
> >> Thanks jark and dian:
> >> 1.jark's approach: do the work in task-0. Simple way.
> >> 2.dian's approach: use StreamingRuntimeContext#getGlobalAggregateManager
> >> Can do more operation. But these accumulators are not fault-toler
more operation. But these accumulators are not fault-tolerant?
>>
>> Best,
>> Jingsong Lee
>>
>>
>> ------------------
>> From:shimin yang
>> Send Time:2019年9月6日(星期五) 15:21
>> To:dev
>> Subject:Re: [DISCUSS] Support notifyOnMaster for notifyCheckpointCom
t;
> Best,
> Jingsong Lee
>
>
> --
> From:shimin yang
> Send Time:2019年9月6日(星期五) 15:21
> To:dev
> Subject:Re: [DISCUSS] Support notifyOnMaster for notifyCheckpointComplete
>
> Hi Fu,
>
> That'll be ni
ngsong Lee
--
From:shimin yang
Send Time:2019年9月6日(星期五) 15:21
To:dev
Subject:Re: [DISCUSS] Support notifyOnMaster for notifyCheckpointComplete
Hi Fu,
That'll be nice.
Thanks.
Best,
Shimin
Dian Fu 于2019年9月6日周五 下午3:17写道：
> Hi Shimin,
>
> It can be guaranteed to be an a
Hi Fu,
That'll be nice.
Thanks.
Best,
Shimin
Dian Fu 于2019年9月6日周五 下午3:17写道：
> Hi Shimin,
>
> It can be guaranteed to be an atomic operation. This is ensured by the RPC
> framework. You could take a look at RpcEndpoint for more details.
>
> Regards,
> Dian
>
> > 在 2019年9月6日，下午2:35，shimin yang
Hi Shimin,
It can be guaranteed to be an atomic operation. This is ensured by the RPC
framework. You could take a look at RpcEndpoint for more details.
Regards,
Dian
> 在 2019年9月6日，下午2:35，shimin yang 写道：
>
> Hi Fu,
>
> Thank you for the remind. I think it would work in my case as long as it's
Hi Fu,
Thank you for the remind. I think it would work in my case as long as it's
an atomic operation.
Dian Fu 于2019年9月6日周五 下午2:22写道：
> Hi Jingsong,
>
> Thanks for bring up this discussion. You can try to look at the
> GlobalAggregateManager to see if it can meet your requirements. It can be
>
Hi Jingsong,
Thanks for bring up this discussion. You can try to look at the
GlobalAggregateManager to see if it can meet your requirements. It can be got
via StreamingRuntimeContext#getGlobalAggregateManager().
Regards,
Dian
> 在 2019年9月6日，下午1:39，shimin yang 写道：
>
> Hi Jingsong,
>
> Big fan
I think before we have such interface, maybe we can make task-0 to do the
global finalize work.
Best,
Jark
> 在 2019年9月6日，13:39，shimin yang 写道：
>
> Hi Jingsong,
>
> Big fan of this idea. We faced the same problem and resolved by adding a
> distributed lock. It would be nice to have this fea
Hi Jingsong,
Big fan of this idea. We faced the same problem and resolved by adding a
distributed lock. It would be nice to have this feature in JobMaster, which
can replace the lock.
Best,
Shimin
JingsongLee 于2019年9月6日周五 下午12:20写道：
> Hi devs:
>
> I try to implement streaming file sink for tab
Hi devs:
I try to implement streaming file sink for table[1] like StreamingFileSink.
If the underlying is a HiveFormat, or a format that updates visibility
through a metaStore, I have to update the metaStore in the
notifyCheckpointComplete, but this operation occurs on the task side,
which will
12 matches
Mail list logo
