[DISCUSS] FLIP-67: Global partitions lifecycle

[Chesnay Schepler]

Hello,

FLIP-36 (interactive programming) 
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink> 
proposes a new programming paradigm where jobs are built incrementally 
by the user.

To support this in an efficient manner I propose to extend partition 
life-cycle to support the notion of /global partitions/, which are 
partitions that can exist beyond the life-time of a job.

These partitions could then be re-used by subsequent jobs in a fairly 
efficient manner, as they don't have to persisted to an external storage 
first and consuming tasks could be scheduled to exploit data-locality.

The FLIP outlines the required changes on the JobMaster, TaskExecutor 
and ResourceManager to support this from a life-cycle perspective.

This FLIP does /not/ concern itself with the /usage/ of global 
partitions, including client-side APIs, job-submission, scheduling and 
reading said partitions; these are all follow-ups that will either be 
part of FLIP-36 or spliced out into separate FLIPs.