Anton, Thank you for the clarification!
вт, 29 янв. 2019 г. в 15:15, Anton Kalashnikov <kaa....@yandex.ru>: > > Ivan, I glad you interested in this feature. Some answers are below. > > Yes, it is correct about properties it is consistent across the cluster cause > it based on distributed metastore(we have other topic for discuss it). > > Some details of implementation: When event happened we added task to > GridTimeoutProcessor(old mechanism for task execution with delay). Task is > added only for coordinator. There is not required do it on non coordinator > nodes because if coordinator will be failed new event will be appear and we > will generate new task on new coordinator. > > -- > Best regards, > Anton Kalashnikov > > > 28.01.2019, 13:17, "Павлухин Иван" <vololo...@gmail.com>: > > Anton, > > > > Great feature! > > > > Could you please clarify a bit about implementation details? As I > > understood auto-adjust properties is meant to be consistent across the > > cluster. And baseline adjustment is put into some delay queue. Do we > > put event into a queue on each node? Or is there some dedicated node > > driving baseline adjustment? > > > > пт, 25 янв. 2019 г. в 16:31, Anton Kalashnikov <kaa....@yandex.ru>: > >> Initially, hard timeout should protect grid from constantly changing > >> topology(constantly blinking node). But in fact if we have constantly > >> changing topology, baseline adjust operation is failed in most cases. As > >> result hard timeout only added complexity but it don't give any new > >> guarantee. So I think we can skip it in first implementation. > >> > >> First of all timeout protect us from unnecessary adjust of baseline . If > >> node left the grid and immediately(or after some time less than us > >> timeout) it join back to grid. Also timeout is helpful in other cases when > >> some events happened one after another. > >> > >> This feature doesn't have any complex heuristic to react, except of > >> described in restrictions section. > >> > >> Also I want to notes that this feature isn't protect us from constantly > >> blinking node. We need one more heuristic mechanism for detect this > >> situation and doing some actions like removing this node from grid. > >> > >> -- > >> Best regards, > >> Anton Kalashnikov > >> > >> 25.01.2019, 15:43, "Sergey Chugunov" <sergey.chugu...@gmail.com>: > >> > Anton, > >> > > >> > As I understand from the IEP document policy was supposed to support two > >> > timeouts: soft and hard, so here you're proposing a bit simpler > >> > functionality. > >> > > >> > Just to clarify, do I understand correctly that this feature when > >> enabled > >> > will auto-adjust blt on each node join/node left event, and timeout is > >> > necessary to protect us from blinking nodes? > >> > So no complexities with taking into account number of alive backups or > >> > something like that? > >> > > >> > On Fri, Jan 25, 2019 at 1:11 PM Vladimir Ozerov <voze...@gridgain.com> > >> > wrote: > >> > > >> >> Got it, makes sense. > >> >> > >> >> On Fri, Jan 25, 2019 at 11:06 AM Anton Kalashnikov <kaa....@yandex.ru> > >> >> wrote: > >> >> > >> >> > Vladimir, thanks for your notes, both of them looks good enough but I > >> >> > have two different thoughts about it. > >> >> > > >> >> > I think I agree about enabling only one of manual/auto adjustment. > >> It is > >> >> > easier than current solution and in fact as extra feature we can > >> allow > >> >> > user to force task to execute(if they doesn't want to wait until > >> timeout > >> >> > expired). > >> >> > But about second one I don't sure that one parameters instead of two > >> >> would > >> >> > be more convenient. For example: in case when user changed timeout > >> and > >> >> then > >> >> > disable auto-adjust after then when someone will want to enable it > >> they > >> >> > should know what value of timeout was before auto-adjust was > >> disabled. I > >> >> > think "negative value" pattern good choice for always usable > >> parameters > >> >> > like timeout of connection (ex. -1 equal to endless waiting) and so > >> on, > >> >> but > >> >> > in our case we want to disable whole functionality rather than change > >> >> > parameter value. > >> >> > > >> >> > -- > >> >> > Best regards, > >> >> > Anton Kalashnikov > >> >> > > >> >> > > >> >> > 24.01.2019, 22:03, "Vladimir Ozerov" <voze...@gridgain.com>: > >> >> > > Hi Anton, > >> >> > > > >> >> > > This is great feature, but I am a bit confused about automatic > >> >> disabling > >> >> > of > >> >> > > a feature during manual baseline adjustment. This may lead to > >> >> unpleasant > >> >> > > situations when a user enabled auto-adjustment, then re-adjusted it > >> >> > > manually somehow (e.g. from some previously created script) so that > >> >> > > auto-adjustment disabling went unnoticed, then added more nodes > >> hoping > >> >> > that > >> >> > > auto-baseline is still active, etc. > >> >> > > > >> >> > > Instead, I would rather make manual and auto adjustment mutually > >> >> > exclusive > >> >> > > - baseline cannot be adjusted manually when auto mode is set, and > >> vice > >> >> > > versa. If exception is thrown in that cases, administrators will > >> always > >> >> > > know current behavior of the system. > >> >> > > > >> >> > > As far as configuration, wouldn’t it be enough to have a single > >> long > >> >> > value > >> >> > > as opposed to Boolean + long? Say, 0 - immediate auto adjustment, > >> >> > negative > >> >> > > - disabled, positive - auto adjustment after timeout. > >> >> > > > >> >> > > Thoughts? > >> >> > > > >> >> > > чт, 24 янв. 2019 г. в 18:33, Anton Kalashnikov <kaa....@yandex.ru>: > >> >> > > > >> >> > >> Hello, Igniters! > >> >> > >> > >> >> > >> Work on the Phase II of IEP-4 (Baseline topology) [1] has > >> started. I > >> >> > want > >> >> > >> to start to discuss of implementation of "Baseline auto-adjust" > >> [2]. > >> >> > >> > >> >> > >> "Baseline auto-adjust" feature implements mechanism of auto-adjust > >> >> > >> baseline corresponding to current topology after event join/left > >> was > >> >> > >> appeared. It is required because when a node left the grid and > >> nobody > >> >> > would > >> >> > >> change baseline manually it can lead to lost data(when some more > >> >> nodes > >> >> > left > >> >> > >> the grid on depends in backup factor) but permanent tracking of > >> grid > >> >> > is not > >> >> > >> always possible/desirible. Looks like in many cases auto-adjust > >> >> > baseline > >> >> > >> after some timeout is very helpfull. > >> >> > >> > >> >> > >> Distributed metastore[3](it is already done): > >> >> > >> > >> >> > >> First of all it is required the ability to store configuration > >> data > >> >> > >> consistently and cluster-wide. Ignite doesn't have any specific > >> API > >> >> for > >> >> > >> such configurations and we don't want to have many similar > >> >> > implementations > >> >> > >> of the same feature in our code. After some thoughts is was > >> proposed > >> >> to > >> >> > >> implement it as some kind of distributed metastorage that gives > >> the > >> >> > ability > >> >> > >> to store any data in it. > >> >> > >> First implementation is based on existing local metastorage API > >> for > >> >> > >> persistent clusters (in-memory clusters will store data in > >> memory). > >> >> > >> Write/remove operation use Discovery SPI to send updates to the > >> >> > cluster, it > >> >> > >> guarantees updates order and the fact that all existing (alive) > >> nodes > >> >> > have > >> >> > >> handled the update message. As a way to find out which node has > >> the > >> >> > latest > >> >> > >> data there is a "version" value of distributed metastorage, which > >> is > >> >> > >> basically <number of all updates, hash of updates>. All updates > >> >> history > >> >> > >> until some point in the past is stored along with the data, so > >> when > >> >> an > >> >> > >> outdated node connects to the cluster it will receive all the > >> missing > >> >> > data > >> >> > >> and apply it locally. If there's not enough history stored or > >> joining > >> >> > node > >> >> > >> is clear then it'll receive shapshot of distributed metastorage so > >> >> > there > >> >> > >> won't be inconsistencies. > >> >> > >> > >> >> > >> Baseline auto-adjust: > >> >> > >> > >> >> > >> Main scenario: > >> >> > >> - There is grid with the baseline is equal to the current > >> >> > topology > >> >> > >> - New node joins to grid or some node left(failed) the grid > >> >> > >> - New mechanism detects this event and it add task for > >> >> changing > >> >> > >> baseline to queue with configured timeout > >> >> > >> - If new event are happened before baseline would be changed > >> >> > task > >> >> > >> would be removed from queue and new task will be added > >> >> > >> - When timeout are expired the task would try to set new > >> >> > baseline > >> >> > >> corresponded to current topology > >> >> > >> > >> >> > >> First of all we need to add two parameters[4]: > >> >> > >> - baselineAutoAdjustEnabled - enable/disable "Baseline > >> >> > >> auto-adjust" feature. > >> >> > >> - baselineAutoAdjustTimeout - timeout after which baseline > >> >> > should > >> >> > >> be changed. > >> >> > >> > >> >> > >> This parameters are cluster wide and can be changed in real time > >> >> > because > >> >> > >> it is based on "Distributed metastore". On first time this > >> parameters > >> >> > would > >> >> > >> be initiated by corresponded > >> >> parameters(initBaselineAutoAdjustEnabled, > >> >> > >> initBaselineAutoAdjustTimeout) from "Ignite Configuration". Init > >> >> value > >> >> > >> valid only before first changing of it after value would be > >> changed > >> >> it > >> >> > is > >> >> > >> stored in "Distributed metastore". > >> >> > >> > >> >> > >> Restrictions: > >> >> > >> - This mechanism handling events only on active grid > >> >> > >> - If baselineNodes != gridNodes on activate this feature > >> >> would > >> >> > be > >> >> > >> disabled > >> >> > >> - If lost partitions was detected this feature would be > >> >> > disabled > >> >> > >> - If baseline was adjusted manually on baselineNodes != > >> >> > gridNodes > >> >> > >> this feature would be disabled > >> >> > >> > >> >> > >> Draft implementation you can find here[5]. Feel free to ask more > >> >> > details > >> >> > >> and make suggestions. > >> >> > >> > >> >> > >> [1] > >> >> > >> > >> >> > > >> >> > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-4+Baseline+topology+for+caches > >> >> > >> [2] https://issues.apache.org/jira/browse/IGNITE-8571 > >> >> > >> [3] https://issues.apache.org/jira/browse/IGNITE-10640 > >> >> > >> [4] https://issues.apache.org/jira/browse/IGNITE-8573 > >> >> > >> [5] https://github.com/apache/ignite/pull/5907 > >> >> > >> > >> >> > >> -- > >> >> > >> Best regards, > >> >> > >> Anton Kalashnikov > >> >> > > > > > -- > > Best regards, > > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
