Re: [DISCUSS] FLIP-67: Global partitions lifecycle

PRC call on TE#releasePartitions is also triggered not via > >> > ShuffleMaster in some cases, and it can be regareded as a shortcut > >> > way. Of course I am also in favour of via ShuffleMaster to call > >> > the actual release partition always, an

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

artition always, and the form seems elegant. >> > >> > I do not expect my inconsequential thought would block this feature >> > ongoing and disturb your previous conclusion. Moreover, Till's recent >> > reply already dispels my previous concern. :) >> >

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

#x27;s recent > > reply already dispels my previous concern. :) > > > > Best, > > Zhijiang > > > > -- > > From:Chesnay Schepler > > Send Time:2019年10月14日(星期一) 07:00 > > To:dev

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

s recent reply already dispels my previous concern. :) Best, Zhijiang -- From:Chesnay Schepler Send Time:2019年10月14日(星期一) 07:00 To:dev ; Till Rohrmann ; zhijiang Subject:Re: [DISCUSS] FLIP-67: Global parti

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

bal released partitions via payloads in heartbeat >> with TE to reduce additional explict RPC call, but this would bring some >> delays for releasing partition based on heartbeat interval. >> >> Best, >> Zhijiang >> -

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

. Best, Zhijiang -- From:Till Rohrmann Send Time:2019年10月13日(星期日) 17:04 To:zhijiang Cc:dev Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle I think we won't necessarily run multiple ShuffleMasters. I think it wou

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

th TE to reduce additional explict RPC call, but this would bring some delays for releasing partition based on heartbeat interval. Best, Zhijiang ---------- From:Chesnay Schepler Send Time:2019年10月11日(星期五) 10:21 To:dev ; Till Rohrmann Subject:

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

some > delays for releasing partition based on heartbeat interval. > > Best, > Zhijiang > -- > From:Chesnay Schepler > Send Time:2019年10月11日(星期五) 10:21 > To:dev ; Till Rohrmann > Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle > > h I like job-/clust

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Zhijiang -- From:Chesnay Schepler Send Time:2019年10月11日(星期五) 10:21 To:dev ; Till Rohrmann Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle h I like job-/cluster partitions. On 10/10/2019 16:27, Till Rohrmann wrote: > I think we should introduc

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

h I like job-/cluster partitions. On 10/10/2019 16:27, Till Rohrmann wrote: I think we should introduce a separate interface for the ResourceManager so that it can list and delete global result partitions from the shuffle service implementation. As long as the JM and RM run in the same proce

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

I think we should introduce a separate interface for the ResourceManager so that it can list and delete global result partitions from the shuffle service implementation. As long as the JM and RM run in the same process, this interface could be implemented by the ShuffleMaster implementations. Howev

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Are there any other opinions in regards to the naming scheme? (local/global, promote) On 06/09/2019 15:16, Chesnay Schepler wrote: Hello, FLIP-36 (interactive programming) proposes a new p

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

inside JM and expected to interactive with JM before. Now the RM also needs to interactive with ShuffleMaster to release global partitions. Then it might be better to move ShuffleMaster outside of JM, and the lifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

with ShuffleMaster to release global partitions. Then it might be better to move ShuffleMaster outside of JM, and the lifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best, Zhijiang --

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

design, RM should be able to release result >> partitions using ShuffleService. Will RM do this by sending RPC to the >> >> TEs? >> >> Or will the RM do it by itself? >> >> 4. How do we plan to handle the case when there are different shuffle >> services in the same Flink cluster? For exa

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

fleMaster to release global partitions. Then it might be better to move ShuffleMaster outside of JM, and the lifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best,

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

. I think it is better to unify the term in different components for > > for > > better understanding the semantic. Concering whether to use global or > persistent, I prefer the "global" term personally. Because it > > describes the > > scope of partition clear

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

ive with ShuffleMaster to release global partitions. Then it might be better to move ShuffleMaster outside of JM, and the lifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Bes

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

k it is better to unify the term in different components for > for > >>> better understanding the semantic. Concering whether to use global or > >>> persistent, I prefer the "global" term personally. Because it > describes the > >>> scope of partition cl

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

fleMaster outside of JM, and the lifecycle of ShuffleMaster should be consistent with RM. 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global partitions for successful jobs" Best, Zhijiang -- From:Till Rohrmann

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

al partitions. >> >> 4. Considering ShuffleMaster, it was built inside JM and expected to >> interactive with JM before. Now the RM also needs to interactive with >> ShuffleMaster to release global partitions. Then it might be better to move >> ShuffleMaster outside of

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

consistent with RM. > > 5. Nit: TM->TE in the section of Proposed Changes: "TMs retain global > partitions for successful jobs" > > Best, > Zhijiang > > > ---------- > From:Till Rohrmann > Send Time:20

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

s for successful jobs" Best, Zhijiang -- From:Till Rohrmann Send Time:2019年9月10日(星期二) 10:10 To:dev Subject:Re: [DISCUSS] FLIP-67: Global partitions lifecycle Thanks Chesnay for drafting the FLIP and starting this discussion. I have a couple of comments

Re: [DISCUSS] FLIP-67: Global partitions lifecycle

Thanks Chesnay for drafting the FLIP and starting this discussion. I have a couple of comments: * I know that I've also coined the terms global/local result partition but maybe it is not the perfect name. Maybe we could rethink the terminology and call them persistent result partitions? * Nit: I

[DISCUSS] FLIP-67: Global partitions lifecycle

Hello, FLIP-36 (interactive programming) proposes a new programming paradigm where jobs are built incrementally by the user. To support this in an efficient manner I propose to extend part