+1 to rethink
Best,
Jingsong Lee
--
From:Stephan Ewen
Send Time:2019年9月10日(星期二) 15:40
To:dev
Cc:JingsongLee
Subject:Re: [DISCUSS] Support notifyOnMaster for notifyCheckpointComplete
Hi all!
I think it would be time to rethink t
Hi all!
I think it would be time to rethink the Sink API as a whole, like we did
with the Source API in FLIP-27.
It would be nice to have proper design that handles all this consistently,
rather than adding one more hook.
For example:
- For batch, you can already use the existing "finalize on m
Hi Jingsong,
Good point!
1. If it doesn't matter which task performs the finalize work, then I think
task-0 suggested by Jark is a very good solution.
2. If it requires the last finished task to perform the finalize work, then we
have to consider other solutions.
WRT fault-tolerant of Stream
Hi Jingsong,
Although it would be nice if the accumulators in GlobalAggregateManager is
fault-tolerant, we could still take advantage of managed state to guarantee
the semantic and use the accumulators to implement distributed barrier or
lock to solve the distributed access problem.
Best,
Shimin
Thanks jark and dian:
1.jark's approach: do the work in task-0. Simple way.
2.dian's approach: use StreamingRuntimeContext#getGlobalAggregateManager Can do
more operation. But these accumulators are not fault-tolerant?
Best,
Jingsong Lee
-
Hi Fu,
That'll be nice.
Thanks.
Best,
Shimin
Dian Fu 于2019年9月6日周五 下午3:17写道：
> Hi Shimin,
>
> It can be guaranteed to be an atomic operation. This is ensured by the RPC
> framework. You could take a look at RpcEndpoint for more details.
>
> Regards,
> Dian
>
> > 在 2019年9月6日，下午2:35，shimin yang
Hi Shimin,
It can be guaranteed to be an atomic operation. This is ensured by the RPC
framework. You could take a look at RpcEndpoint for more details.
Regards,
Dian
> 在 2019年9月6日，下午2:35，shimin yang 写道：
>
> Hi Fu,
>
> Thank you for the remind. I think it would work in my case as long as it's
Hi Fu,
Thank you for the remind. I think it would work in my case as long as it's
an atomic operation.
Dian Fu 于2019年9月6日周五 下午2:22写道：
> Hi Jingsong,
>
> Thanks for bring up this discussion. You can try to look at the
> GlobalAggregateManager to see if it can meet your requirements. It can be
>
Hi Jingsong,
Thanks for bring up this discussion. You can try to look at the
GlobalAggregateManager to see if it can meet your requirements. It can be got
via StreamingRuntimeContext#getGlobalAggregateManager().
Regards,
Dian
> 在 2019年9月6日，下午1:39，shimin yang 写道：
>
> Hi Jingsong,
>
> Big fan
I think before we have such interface, maybe we can make task-0 to do the
global finalize work.
Best,
Jark
> 在 2019年9月6日，13:39，shimin yang 写道：
>
> Hi Jingsong,
>
> Big fan of this idea. We faced the same problem and resolved by adding a
> distributed lock. It would be nice to have this fea
Hi Jingsong,
Big fan of this idea. We faced the same problem and resolved by adding a
distributed lock. It would be nice to have this feature in JobMaster, which
can replace the lock.
Best,
Shimin
JingsongLee 于2019年9月6日周五 下午12:20写道：
> Hi devs:
>
> I try to implement streaming file sink for tab
11 matches
Mail list logo
