date:20190125

Re: Baseline auto-adjust`s discuss

2019-01-25 Thread Anton Kalashnikov

Vladimir, thanks  for your notes, both of them looks good enough but I have two 
different thoughts about it. 

I think I agree about enabling only one of manual/auto adjustment. It is easier 
than current solution and in fact as extra feature  we can allow user to force 
task to execute(if they doesn't want to wait until timeout expired). 
But about second one I don't sure that one parameters instead of two would be 
more convenient. For example: in case when user changed timeout and then 
disable auto-adjust after then when someone will want to enable it they should 
know what value of timeout was before auto-adjust was disabled. I think 
"negative value" pattern good choice for always usable parameters like timeout 
of connection (ex. -1 equal to endless waiting) and so on, but in our case we 
want to disable whole functionality rather than change parameter value.

-- 
Best regards,
Anton Kalashnikov


24.01.2019, 22:03, "Vladimir Ozerov" :
> Hi Anton,
>
> This is great feature, but I am a bit confused about automatic disabling of
> a feature during manual baseline adjustment. This may lead to unpleasant
> situations when a user enabled auto-adjustment, then re-adjusted it
> manually somehow (e.g. from some previously created script) so that
> auto-adjustment disabling went unnoticed, then added more nodes hoping that
> auto-baseline is still active, etc.
>
> Instead, I would rather make manual and auto adjustment mutually exclusive
> - baseline cannot be adjusted manually when auto mode is set, and vice
> versa. If exception is thrown in that cases, administrators will always
> know current behavior of the system.
>
> As far as configuration, wouldn’t it be enough to have a single long value
> as opposed to Boolean + long? Say, 0 - immediate auto adjustment, negative
> - disabled, positive - auto adjustment after timeout.
>
> Thoughts?
>
> чт, 24 янв. 2019 г. в 18:33, Anton Kalashnikov :
>
>>  Hello, Igniters!
>>
>>  Work on the Phase II of IEP-4 (Baseline topology) [1] has started. I want
>>  to start to discuss of implementation of "Baseline auto-adjust" [2].
>>
>>  "Baseline auto-adjust" feature implements mechanism of auto-adjust
>>  baseline corresponding to current topology after event join/left was
>>  appeared. It is required because when a node left the grid and nobody would
>>  change baseline manually it can lead to lost data(when some more nodes left
>>  the grid on depends in backup factor) but permanent tracking of grid is not
>>  always possible/desirible. Looks like in many cases auto-adjust baseline
>>  after some timeout is very helpfull.
>>
>>  Distributed metastore[3](it is already done):
>>
>>  First of all it is required the ability to store configuration data
>>  consistently and cluster-wide. Ignite doesn't have any specific API for
>>  such configurations and we don't want to have many similar implementations
>>  of the same feature in our code. After some thoughts is was proposed to
>>  implement it as some kind of distributed metastorage that gives the ability
>>  to store any data in it.
>>  First implementation is based on existing local metastorage API for
>>  persistent clusters (in-memory clusters will store data in memory).
>>  Write/remove operation use Discovery SPI to send updates to the cluster, it
>>  guarantees updates order and the fact that all existing (alive) nodes have
>>  handled the update message. As a way to find out which node has the latest
>>  data there is a "version" value of distributed metastorage, which is
>>  basically . All updates history
>>  until some point in the past is stored along with the data, so when an
>>  outdated node connects to the cluster it will receive all the missing data
>>  and apply it locally. If there's not enough history stored or joining node
>>  is clear then it'll receive shapshot of distributed metastorage so there
>>  won't be inconsistencies.
>>
>>  Baseline auto-adjust:
>>
>>  Main scenario:
>>  - There is grid with the baseline is equal to the current topology
>>  - New node joins to grid or some node left(failed) the grid
>>  - New mechanism detects this event and it add task for changing
>>  baseline to queue with configured timeout
>>  - If new event are happened before baseline would be changed task
>>  would be removed from queue and new task will be added
>>  - When timeout are expired the task would try to set new baseline
>>  corresponded to current topology
>>
>>  First of all we need to add two parameters[4]:
>>  - baselineAutoAdjustEnabled - enable/disable "Baseline
>>  auto-adjust" feature.
>>  - baselineAutoAdjustTimeout - timeout after which baseline should
>>  be changed.
>>
>>  This parameters are cluster wide and can be changed in real time because
>>  it is based on "Distributed metastore". On first time this parameters would
>>  be initiated by corresponded parameters(initBaselineAutoAdjustEnabled,
>>  initBaselineAutoAdjustTimeout) from "

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

2019-01-25 Thread Sergey Antonov

Ivan, thank you. I started wrote same test today)

пт, 25 янв. 2019 г. в 10:16, Павлухин Иван :

> Pavel,
>
> Initially I meant Java thick client. And I see the difference between
> thin and thick Java clients. As was already mentioned thin Java client
> behaves in way like each cache is enforced to not unwrap binary
> objects (withKeepBinary()). You can see a test demonstrating current
> behavior [1].
>
> [1] https://gist.github.com/pavlukhin/2c76d11cde5243a73f01019cdd15d243
>
> чт, 24 янв. 2019 г. в 23:01, Pavel Tupitsyn :
> >
> > Ivan,
> >
> > There is no inconsistency between thick and thin clients.
> > All of them work with caches in binary mode, see ClientCacheRequest
> (thin)
> > and PlatformCache (thick) classes.
> >
> > On Thu, Jan 24, 2019 at 10:26 PM Ivan Pavlukhina 
> > wrote:
> >
> > > Sergey,
> > >
> > > There are couple of things which should be addressed:
> > > 1. Unnecessary deserialization.
> > > 2. Inconsistent behavior.
> > > 3. Unclear documentation.
> > >
> > > Deserialization is not free and in my mind should be avoided where
> > > possible. I think that if some feature (like interceptors) requires
> > > deserialization then it should be enabled explicitly and an impact
> should
> > > be clear to user. I can imagine a toggle “withAllowedDeserialization”.
> > >
> > > If there is an inconsistency between thick and thin clients it should
> be
> > > eliminated. I do not see a reason why behavior should be different.
> > >
> > > If something is a good thing but it is not intuitive it could be
> > > documented. But if there is s really good reason for it. Otherwise
> > > simplicity and consistency are better alias.
> > >
> > > > On 24 Jan 2019, at 17:42, Sergey Antonov 
> > > wrote:
> > > >
> > > > I think it's bad idea. This contract nowhere defined and it's not
> clear
> > > for
> > > > users.
> > > >
> > > > чт, 24 янв. 2019 г. в 17:18, Pavel Tupitsyn :
> > > >
> > > >> Yes
> > > >>
> > > >> On Thu, Jan 24, 2019 at 5:15 PM Sergey Antonov <
> > > antonovserge...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Pavel,
> > > >>>
> > > >>> "Leave it as is, use instanceof."
> > > >>> You meant always use CacheInterceptor and in all
> > > methods
> > > >>> check, that passed arguments is BinaryObject?
> > > >>>
> > > >>> чт, 24 янв. 2019 г. в 17:10, Pavel Tupitsyn  >:
> > > >>>
> > >  I don't think we should complicate things. Leave it as is, use
> > > >>> instanceof.
> > >  The fact is - you can get anything, BinaryObject or any user
> class, so
> > > >> be
> > >  prepared.
> > >  Good example of older API is CacheEvent, which actually has
> oldValue()
> > > >>> and
> > >  newValue() as Object.
> > > 
> > >  Igniters, any other thoughts?
> > > 
> > > 
> > >  On Thu, Jan 24, 2019 at 2:16 PM Sergey Antonov <
> > > >>> antonovserge...@gmail.com>
> > >  wrote:
> > > 
> > > > Pavel, how about marker interface
> DeserializedValueCacheInterceptor?
> > > >> We
> > > > will deserialize data and pass it to cache interceptor, if
> > >  CacheInterceptor
> > > > implements marker interface.
> > > >
> > > > чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn <
> ptupit...@apache.org>:
> > > >
> > > >> You are exactly right, generic parameters don't make much sense
> > > >> here.
> > > >> Ignite caches are not restricted to any type, and there is type
> > > >>> erasure
> > > > in
> > > >> Java so you have no runtime guarantees.
> > > >>
> > > >> Maybe Interceptor design should be improved (e.g. add a flag to
> > > >> force
> > > >> binary or non-binary mode),
> > > >> but Thin or Thick client connector logic is unrelated here.
> > > >> withKeepBinary() call is valid and should not depend on
> Interceptor
> > > >> presence or implementation.
> > > >>
> > > >> On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov <
> > > > antonovserge...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi, Pavel,
> > > >>>
> > > >>> "Interceptor should support both modes, binary or not. Any code
> > > >> can
> > > > call
> > > >>> withKeepBinary(), this should be expected.
> > > >>> Just add if (x instanceof BinaryObject) and go from there. "
> > > >>> I don't agree. The cache interceptor[1] is a parametrized class
> > > >> and
> > >  you
> > > >>> couldn't pass multiple cache interceptors in cache
> configuration.
> > > >>> So
> > > > all
> > > >>> cache interceptors must have Object, Object parameters for
> > > >>> supporting
> > > >> both
> > > >>> modes: binary and deserialized. In this case parametrized class
> > > >> no
> > > > sense.
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
> > > >>>
> > > >>> чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn <
> > > >> ptupit...@apache.or

Distributed MetaStorage discussion

2019-01-25 Thread Ivan Bessonov

Hello, Igniters!

Here's more info "Distributed MetaStorage" feature [1]. It is a part of
Phase II for
IEP-4 (Baseline topology) [2] and was mentioned in recent "Baseline
auto-adjust`s
discuss" topic. I'll partially duplicate that message here.

One of key requirements is the ability to store configuration data (or any
other data)
consistently and cluster-wide. There are also other tickets that require
similar
mechanisms, for example [3]. Ignite doesn't have any specific API for such
configurations and we don't want to have many similar implementations of the
same feature across the code.

There are several API methods required for the feature:

- read(key) / iterate(keyPrefix) - access to the distributed data. Should
be
consistent for all nodes in cluster when it's in active state.
- write / remove - modify data in distributed metastorage. Should
guarantee that
every node in cluster will have this update after the method is finished.
- writeAsync / removeAsync (not yet implemented) - same as above, but
async.
Might be useful if one needs to update several values one after another.
- compareAndWrite / compareAndRemove - helpful to reduce number of data
updates (more on that later).
- listen(keyPredicate) - a way of being notified when some data was
changed.
Normally it is triggered on "write/remove" operation or node activation.
Listener
itself will be notified with .

Now some implementation details:

First implementation is based on existing local metastorage API for
persistent
clusters (in-memory clusters will store data in memory). Write/remove
operation
use Discovery SPI to send updates to the cluster, it guarantees updates
order
and the fact that all existing (alive) nodes have handled the update
message.

As a way to find out which node has the latest data there is a "version"
value of
distributed metastorage, which is basically . Whole updates history until some point in the past is stored
along with
the data, so when an outdated node connects to the cluster it will receive
all the
missing data and apply it locally. Listeners will also be invoked after
such updates.
If there's not enough history stored or joining node is clear then it'll
receive
shapshot of distributed metastorage so there won't be inconsistencies.
"compareAndWrite" / "compareAndRemove" API might help reducing the size of
the history, especially for Boolean or other primitive values.

There are, of course, many more details, feel free to ask about them. First
implementation is in master, but there are already known improvements that
can
be done and I'm working on them right now.

See package "org.apache.ignite.internal.processors.metastorage" for the new
interfaces and comment your opinion or questions. Thank you!

[1] https://issues.apache.org/jira/browse/IGNITE-10640
[2]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-4+Baseline+topology+for+caches
[3] https://issues.apache.org/jira/browse/IGNITE-8717

39 matches

Mail list logo