Ilya, Thanks for the detailed explanation. Everything you suggested makes sense. Needless to say, PME effect on the grid should be minimized. Let's start with the simpler and less risky changes first.
D. On Thu, Sep 13, 2018 at 5:58 AM Ilya Lantukh <[email protected]> wrote: > Igniters, > > As most of you know, Ignite has a concept of AffinityTopologyVersion, which > is associated with nodes that are currently present in topology and a > global cluster state (active/inactive, baseline topology, started caches). > Modification of either of them involves process called Partition Map > Exchange (PME) and results in new AffinityTopologyVersion. At that moment > all new cache and compute grid operations are globally "frozen". This might > lead to indeterminate cache downtimes. > > However, our recent changes (esp. introduction of Baseline Topology) caused > me to re-think those concept. Currently there are many cases when we > trigger PME, but it isn't necessary. For example, adding/removing client > node or server node not in BLT should never cause partition map > modifications. Those events modify the *topology*, but *affinity* in > unaffected. On the other hand, there are events that affect only *affinity* > - most straightforward example is CacheAffinityChange event, which is > triggered after rebalance is finished to assign new primary/backup nodes. > So the term *AffinityTopologyVersion* now looks weird - it tries to "merge" > two entities that aren't always related. To me it makes sense to introduce > separate *AffinityVersion *and *TopologyVersion*, review all events that > currently modify AffinityTopologyVersion and split them into 3 categories: > those that modify only AffinityVersion, only TopologyVersion and both. It > will allow us to process such events using different mechanics and avoid > redundant steps, and also reconsider mapping of operations - some will be > mapped to topology, others - to affinity. > > Here is my view about how different event types theoretically can be > optimized: > 1. Client node start / stop: as stated above, no PME is needed, ticket > https://issues.apache.org/jira/browse/IGNITE-9558 is already in progress. > 2. Server node start / stop not from baseline: should be similar to the > previous case, since nodes outside of baseline cannot be partition owners. > 3. Start node in baseline: both affinity and topology versions should be > incremented, but it might be possible to optimize PME for such case and > avoid cluster-wide freeze. Partition assignments for such node are already > calculated, so we can simply put them all into MOVING state. However, it > might take significant effort to avoid race conditions and redesign our > architecture. > 4. Cache start / stop: starting or stopping one cache doesn't modify > partition maps for other caches. It should be possible to change this > procedure to skip PME and perform all necessary actions (compute affinity, > start/stop cache contexts on each node) in background, but it looks like a > very complex modification too. > 5. Rebalance finish: it seems possible to design a "lightweight" PME for > this case as well. If there were no node failures (and if there were, PME > should be triggered and rebalance should be cancelled anyways) all > partition states are already known by coordinator. Furthermore, no new > MOVING or OWNING node for any partition is introduced, so all previous > mappings should still be valid. > > For the latter complex cases in might be necessary to introduce "is > compatible" relationship between affinity versions. Operation needs to be > remapped only if new version isn't compatible with the previous one. > > Please share your thoughts. > > -- > Best regards, > Ilya >
