Anton,

Great feature!

Could you please clarify a bit about implementation details? As I
understood auto-adjust properties is meant to be consistent across the
cluster. And baseline adjustment is put into some delay queue. Do we
put event into a queue on each node? Or is there some dedicated node
driving baseline adjustment?

пт, 25 янв. 2019 г. в 16:31, Anton Kalashnikov <kaa....@yandex.ru>:
>
> Initially, hard timeout should protect grid from constantly changing 
> topology(constantly blinking node). But in fact if we have constantly 
> changing topology, baseline adjust operation is failed in most cases. As 
> result hard timeout only added complexity but it don't give any new 
> guarantee. So I think we can skip it in first implementation.
>
> First of all timeout protect us from unnecessary adjust of baseline . If node 
> left the grid and immediately(or after some time less than us timeout) it 
> join back to grid. Also timeout is helpful in other cases when some events 
> happened one after another.
>
> This feature doesn't have any complex heuristic to react, except of described 
> in restrictions section.
>
> Also I want to notes that this feature isn't protect us from constantly 
> blinking node. We need one more heuristic mechanism for detect this situation 
> and doing some actions like removing this node from grid.
>
> --
> Best regards,
> Anton Kalashnikov
>
>
> 25.01.2019, 15:43, "Sergey Chugunov" <sergey.chugu...@gmail.com>:
> > Anton,
> >
> > As I understand from the IEP document policy was supposed to support two
> > timeouts: soft and hard, so here you're proposing a bit simpler
> > functionality.
> >
> > Just to clarify, do I understand correctly that this feature when enabled
> > will auto-adjust blt on each node join/node left event, and timeout is
> > necessary to protect us from blinking nodes?
> > So no complexities with taking into account number of alive backups or
> > something like that?
> >
> > On Fri, Jan 25, 2019 at 1:11 PM Vladimir Ozerov <voze...@gridgain.com>
> > wrote:
> >
> >>  Got it, makes sense.
> >>
> >>  On Fri, Jan 25, 2019 at 11:06 AM Anton Kalashnikov <kaa....@yandex.ru>
> >>  wrote:
> >>
> >>  > Vladimir, thanks for your notes, both of them looks good enough but I
> >>  > have two different thoughts about it.
> >>  >
> >>  > I think I agree about enabling only one of manual/auto adjustment. It is
> >>  > easier than current solution and in fact as extra feature we can allow
> >>  > user to force task to execute(if they doesn't want to wait until timeout
> >>  > expired).
> >>  > But about second one I don't sure that one parameters instead of two
> >>  would
> >>  > be more convenient. For example: in case when user changed timeout and
> >>  then
> >>  > disable auto-adjust after then when someone will want to enable it they
> >>  > should know what value of timeout was before auto-adjust was disabled. I
> >>  > think "negative value" pattern good choice for always usable parameters
> >>  > like timeout of connection (ex. -1 equal to endless waiting) and so on,
> >>  but
> >>  > in our case we want to disable whole functionality rather than change
> >>  > parameter value.
> >>  >
> >>  > --
> >>  > Best regards,
> >>  > Anton Kalashnikov
> >>  >
> >>  >
> >>  > 24.01.2019, 22:03, "Vladimir Ozerov" <voze...@gridgain.com>:
> >>  > > Hi Anton,
> >>  > >
> >>  > > This is great feature, but I am a bit confused about automatic
> >>  disabling
> >>  > of
> >>  > > a feature during manual baseline adjustment. This may lead to
> >>  unpleasant
> >>  > > situations when a user enabled auto-adjustment, then re-adjusted it
> >>  > > manually somehow (e.g. from some previously created script) so that
> >>  > > auto-adjustment disabling went unnoticed, then added more nodes hoping
> >>  > that
> >>  > > auto-baseline is still active, etc.
> >>  > >
> >>  > > Instead, I would rather make manual and auto adjustment mutually
> >>  > exclusive
> >>  > > - baseline cannot be adjusted manually when auto mode is set, and vice
> >>  > > versa. If exception is thrown in that cases, administrators will 
> >> always
> >>  > > know current behavior of the system.
> >>  > >
> >>  > > As far as configuration, wouldn’t it be enough to have a single long
> >>  > value
> >>  > > as opposed to Boolean + long? Say, 0 - immediate auto adjustment,
> >>  > negative
> >>  > > - disabled, positive - auto adjustment after timeout.
> >>  > >
> >>  > > Thoughts?
> >>  > >
> >>  > > чт, 24 янв. 2019 г. в 18:33, Anton Kalashnikov <kaa....@yandex.ru>:
> >>  > >
> >>  > >> Hello, Igniters!
> >>  > >>
> >>  > >> Work on the Phase II of IEP-4 (Baseline topology) [1] has started. I
> >>  > want
> >>  > >> to start to discuss of implementation of "Baseline auto-adjust" [2].
> >>  > >>
> >>  > >> "Baseline auto-adjust" feature implements mechanism of auto-adjust
> >>  > >> baseline corresponding to current topology after event join/left was
> >>  > >> appeared. It is required because when a node left the grid and nobody
> >>  > would
> >>  > >> change baseline manually it can lead to lost data(when some more
> >>  nodes
> >>  > left
> >>  > >> the grid on depends in backup factor) but permanent tracking of grid
> >>  > is not
> >>  > >> always possible/desirible. Looks like in many cases auto-adjust
> >>  > baseline
> >>  > >> after some timeout is very helpfull.
> >>  > >>
> >>  > >> Distributed metastore[3](it is already done):
> >>  > >>
> >>  > >> First of all it is required the ability to store configuration data
> >>  > >> consistently and cluster-wide. Ignite doesn't have any specific API
> >>  for
> >>  > >> such configurations and we don't want to have many similar
> >>  > implementations
> >>  > >> of the same feature in our code. After some thoughts is was proposed
> >>  to
> >>  > >> implement it as some kind of distributed metastorage that gives the
> >>  > ability
> >>  > >> to store any data in it.
> >>  > >> First implementation is based on existing local metastorage API for
> >>  > >> persistent clusters (in-memory clusters will store data in memory).
> >>  > >> Write/remove operation use Discovery SPI to send updates to the
> >>  > cluster, it
> >>  > >> guarantees updates order and the fact that all existing (alive) nodes
> >>  > have
> >>  > >> handled the update message. As a way to find out which node has the
> >>  > latest
> >>  > >> data there is a "version" value of distributed metastorage, which is
> >>  > >> basically <number of all updates, hash of updates>. All updates
> >>  history
> >>  > >> until some point in the past is stored along with the data, so when
> >>  an
> >>  > >> outdated node connects to the cluster it will receive all the missing
> >>  > data
> >>  > >> and apply it locally. If there's not enough history stored or joining
> >>  > node
> >>  > >> is clear then it'll receive shapshot of distributed metastorage so
> >>  > there
> >>  > >> won't be inconsistencies.
> >>  > >>
> >>  > >> Baseline auto-adjust:
> >>  > >>
> >>  > >> Main scenario:
> >>  > >> - There is grid with the baseline is equal to the current
> >>  > topology
> >>  > >> - New node joins to grid or some node left(failed) the grid
> >>  > >> - New mechanism detects this event and it add task for
> >>  changing
> >>  > >> baseline to queue with configured timeout
> >>  > >> - If new event are happened before baseline would be changed
> >>  > task
> >>  > >> would be removed from queue and new task will be added
> >>  > >> - When timeout are expired the task would try to set new
> >>  > baseline
> >>  > >> corresponded to current topology
> >>  > >>
> >>  > >> First of all we need to add two parameters[4]:
> >>  > >> - baselineAutoAdjustEnabled - enable/disable "Baseline
> >>  > >> auto-adjust" feature.
> >>  > >> - baselineAutoAdjustTimeout - timeout after which baseline
> >>  > should
> >>  > >> be changed.
> >>  > >>
> >>  > >> This parameters are cluster wide and can be changed in real time
> >>  > because
> >>  > >> it is based on "Distributed metastore". On first time this parameters
> >>  > would
> >>  > >> be initiated by corresponded
> >>  parameters(initBaselineAutoAdjustEnabled,
> >>  > >> initBaselineAutoAdjustTimeout) from "Ignite Configuration". Init
> >>  value
> >>  > >> valid only before first changing of it after value would be changed
> >>  it
> >>  > is
> >>  > >> stored in "Distributed metastore".
> >>  > >>
> >>  > >> Restrictions:
> >>  > >> - This mechanism handling events only on active grid
> >>  > >> - If baselineNodes != gridNodes on activate this feature
> >>  would
> >>  > be
> >>  > >> disabled
> >>  > >> - If lost partitions was detected this feature would be
> >>  > disabled
> >>  > >> - If baseline was adjusted manually on baselineNodes !=
> >>  > gridNodes
> >>  > >> this feature would be disabled
> >>  > >>
> >>  > >> Draft implementation you can find here[5]. Feel free to ask more
> >>  > details
> >>  > >> and make suggestions.
> >>  > >>
> >>  > >> [1]
> >>  > >>
> >>  >
> >>  
> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-4+Baseline+topology+for+caches
> >>  > >> [2] https://issues.apache.org/jira/browse/IGNITE-8571
> >>  > >> [3] https://issues.apache.org/jira/browse/IGNITE-10640
> >>  > >> [4] https://issues.apache.org/jira/browse/IGNITE-8573
> >>  > >> [5] https://github.com/apache/ignite/pull/5907
> >>  > >>
> >>  > >> --
> >>  > >> Best regards,
> >>  > >> Anton Kalashnikov
> >>  >



-- 
Best regards,
Ivan Pavlukhin

Reply via email to