ooooh I like job-/cluster partitions.
On 10/10/2019 16:27, Till Rohrmann wrote:
I think we should introduce a separate interface for the ResourceManager so
that it can list and delete global result partitions from the shuffle
service implementation. As long as the JM and RM run in the same process,
this interface could be implemented by the ShuffleMaster implementations.
However, we should make sure that we don't introduce unnecessary
concurrency. If that should be the case, then it might be simpler to have
two separate components.
Some ideas for the naming problem:
local/global: job/cluster, intra/inter
Cheers,
Till
On Wed, Oct 9, 2019 at 1:35 PM Chesnay Schepler <ches...@apache.org> wrote:
Are there any other opinions in regards to the naming scheme?
(local/global, promote)
On 06/09/2019 15:16, Chesnay Schepler wrote:
Hello,
FLIP-36 (interactive programming)
<
https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink>
proposes a new programming paradigm where jobs are built incrementally
by the user.
To support this in an efficient manner I propose to extend partition
life-cycle to support the notion of /global partitions/, which are
partitions that can exist beyond the life-time of a job.
These partitions could then be re-used by subsequent jobs in a fairly
efficient manner, as they don't have to persisted to an external
storage first and consuming tasks could be scheduled to exploit
data-locality.
The FLIP outlines the required changes on the JobMaster, TaskExecutor
and ResourceManager to support this from a life-cycle perspective.
This FLIP does /not/ concern itself with the /usage/ of global
partitions, including client-side APIs, job-submission, scheduling and
reading said partitions; these are all follow-ups that will either be
part of FLIP-36 or spliced out into separate FLIPs.
