Ilya,
> 3. Start node in baseline: both affinity and topology versions should be incremented, but it might be possible to optimize PME for such case and avoid cluster-wide freeze. Partition assignments for such node are already calculated, so we can simply put them all into MOVING state. However, it might take significant effort to avoid race conditions and redesign our architecture. As you mentioned all assignments are already calculated. So as another suggestion, we can introduce a new `intermediate` state of such joined nodes. Being in this state node recovers all data from their local storage, preloads whole missed partition data from the cluster (probably on some point in time), creates and preloads missed in-memory and persistent caches. And only after these recovery such node will fire discovery join event and affinity\topology version may be incremented. I think this approach can help to reduce the further rebalance time. WDYT? On Tue, 18 Sep 2018 at 16:31 Alexey Goncharuk <alexey.goncha...@gmail.com> wrote: > Ilya, > > This is a great idea, but before we can ultimately decouple the affinity > version from the topology version, we need to fix a few things with > baseline topology first. Currently the in-memory caches are not using the > baseline topology. We are going to fix this as a part of IEP-4 Phase II > (baseline auto-adjust). Once fixed, we can safely assume that > out-of-baseline node does not affect affinity distribution. > > Agree with Dmitriy that we should start with simpler optimizations first. > > чт, 13 сент. 2018 г. в 15:58, Ilya Lantukh <ilant...@gridgain.com>: > > > Igniters, > > > > As most of you know, Ignite has a concept of AffinityTopologyVersion, > which > > is associated with nodes that are currently present in topology and a > > global cluster state (active/inactive, baseline topology, started > caches). > > Modification of either of them involves process called Partition Map > > Exchange (PME) and results in new AffinityTopologyVersion. At that moment > > all new cache and compute grid operations are globally "frozen". This > might > > lead to indeterminate cache downtimes. > > > > However, our recent changes (esp. introduction of Baseline Topology) > caused > > me to re-think those concept. Currently there are many cases when we > > trigger PME, but it isn't necessary. For example, adding/removing client > > node or server node not in BLT should never cause partition map > > modifications. Those events modify the *topology*, but *affinity* in > > unaffected. On the other hand, there are events that affect only > *affinity* > > - most straightforward example is CacheAffinityChange event, which is > > triggered after rebalance is finished to assign new primary/backup nodes. > > So the term *AffinityTopologyVersion* now looks weird - it tries to > "merge" > > two entities that aren't always related. To me it makes sense to > introduce > > separate *AffinityVersion *and *TopologyVersion*, review all events that > > currently modify AffinityTopologyVersion and split them into 3 > categories: > > those that modify only AffinityVersion, only TopologyVersion and both. It > > will allow us to process such events using different mechanics and avoid > > redundant steps, and also reconsider mapping of operations - some will be > > mapped to topology, others - to affinity. > > > > Here is my view about how different event types theoretically can be > > optimized: > > 1. Client node start / stop: as stated above, no PME is needed, ticket > > https://issues.apache.org/jira/browse/IGNITE-9558 is already in > progress. > > 2. Server node start / stop not from baseline: should be similar to the > > previous case, since nodes outside of baseline cannot be partition > owners. > > 3. Start node in baseline: both affinity and topology versions should be > > incremented, but it might be possible to optimize PME for such case and > > avoid cluster-wide freeze. Partition assignments for such node are > already > > calculated, so we can simply put them all into MOVING state. However, it > > might take significant effort to avoid race conditions and redesign our > > architecture. > > 4. Cache start / stop: starting or stopping one cache doesn't modify > > partition maps for other caches. It should be possible to change this > > procedure to skip PME and perform all necessary actions (compute > affinity, > > start/stop cache contexts on each node) in background, but it looks like > a > > very complex modification too. > > 5. Rebalance finish: it seems possible to design a "lightweight" PME for > > this case as well. If there were no node failures (and if there were, PME > > should be triggered and rebalance should be cancelled anyways) all > > partition states are already known by coordinator. Furthermore, no new > > MOVING or OWNING node for any partition is introduced, so all previous > > mappings should still be valid. > > > > For the latter complex cases in might be necessary to introduce "is > > compatible" relationship between affinity versions. Operation needs to be > > remapped only if new version isn't compatible with the previous one. > > > > Please share your thoughts. > > > > -- > > Best regards, > > Ilya > > > -- -- Maxim Muzafarov
