Denis, In general, service initialization shouldn't prevent a node from joining the cluster or slowing down that process. Thus, I would start the initialization routines only after the node is accepted by the cluster. If the initialization fails then we need to report a respective message to the logs and, probably, generate a system event the user can be subscribed to.
Regardless, of the service initialization time, I think we still need to utilize discovery SPI to avoid problems discussed later. Val, others, what do you think about that? -- Denis On Mon, Apr 16, 2018 at 10:29 AM, Denis Mekhanikov <dmekhani...@gmail.com> wrote: > Basically, my question is: at which moment services should be deployed on > connecting nodes? > Should we reject a node from being included into a topology, if services, > that are assigned to it, fail to deploy? > > It would be good to be sure, that all assigned services are initialised and > working, when node start is finished. > Otherwise it's unclear, how to notify a user about failures in service > initialisation on new nodes. > > If we decide to provide such guarantee, then how are we going to do that? > Is procedure, that I described, viable? > It requires hacking through the discovery protocol, which is a thing, that > should be avoided. > So, maybe there is another way to achieve the same thing? > > Denis > > сб, 14 апр. 2018 г. в 1:48, Denis Magda <dma...@apache.org>: > > > It sounds like it's not a trivial thing to support the automatic services > > redeployment after a restart. Let's postpone it for now, guys > concentrating > > on more urgent things related to the services. > > > > Alex, Vladimir, > > > > Could you have a look at Denis question about the discovery-based > > deployment? Guess it's the only one thing that prevents us from the IEP > > finalization. > > > > -- > > Denis > > > > On Fri, Apr 13, 2018 at 5:30 AM, Denis Mekhanikov <dmekhani...@gmail.com > > > > wrote: > > > > > Vladimir, > > > > > > Currently we don't save binary metadata to disk, when persistence is > > > disabled. > > > But we still persist marshaller mappings for some reason, and I > > personally > > > believe, that we shouldn't. > > > > > > But I agree, that we should separate data and service persistence > > > configuration. > > > Right now persistence of services is configured in a pretty non-obvious > > > manner. > > > It should be a clear way to tell Ignite, whether you want services to > be > > > persisted or not. > > > > > > I'm not sure, that we should make "statefullness" in general > > configurable. > > > Users don't care much, whether metadata is preserved on restarts, or > not. > > > > > > Denis > > > > > > пт, 13 апр. 2018 г. в 14:29, Vladimir Ozerov <voze...@gridgain.com>: > > > > > > > Alex, > > > > > > > > I would say that we've already had this behavior for years - > marshaller > > > > cache. I think it is time to agree that "in-memory" != stateless. > > Instead > > > > "in-memory" means "data is not stored on disk". > > > > May be we can have a flag which will wipe out all metadata on node > > > restart > > > > (e.g. it could make sense for embedded clients)? > > > > > > > > On Fri, Apr 13, 2018 at 12:48 PM, Alexey Goncharuk < > > > > alexey.goncha...@gmail.com> wrote: > > > > > > > > > Denis, > > > > > > > > > > This is a subtle question. It looks like we have now a number of > > > > use-cases > > > > > when persistent storage is required even for a pure in-memory mode. > > One > > > > of > > > > > the use-cases is thin client authentication, the other is service > > grid > > > > > configuration persistence. > > > > > > > > > > Generally, I would agree that this is an expected behavior. > However, > > > this > > > > > means that a user cannot simply start and stop nodes randomly > > anymore. > > > > > Ignite start will require some sort of installation or work folder > > > > > initialization (sort of initdb in postgres) which is ok for > > > > > persistence-enabled modes, but I am not sure if this is expected > for > > > > > in-memory. Of course, we can run this initialization automatically, > > but > > > > it > > > > > is not always a good idea. > > > > > > > > > > If we are ok to have this restrictions for in-memory mode, then > > service > > > > > persistence makes sense. > > > > > > > > > > --AG > > > > > > > > > > 2018-04-11 22:36 GMT+03:00 Denis Magda <dma...@apache.org>: > > > > > > > > > >> Denis, > > > > >> > > > > >> I think that the service deployment state needs be persisted > > > > cluster-wide. > > > > >> I guess that our meta-store is capable of doing so. Alex G., > > Vladimir, > > > > >> could you confirm? > > > > >> > > > > >> As for the split-brain scenarios, I would put them aside for now > > > > because, > > > > >> anyway, they have to be solved at lower levels (meta store, > > discovery, > > > > >> etc.). > > > > >> > > > > >> Also, I heard that presently we store a service configuration in > the > > > > >> system > > > > >> cache that doesn't give us a way to deploy a new version of a > > service > > > > >> without undeployment of the previous one. Will this issue be > > addressed > > > > by > > > > >> the new deployment approach? > > > > >> > > > > >> -- > > > > >> Denis > > > > >> > > > > >> On Wed, Apr 11, 2018 at 1:28 AM, Denis Mekhanikov < > > > > dmekhani...@gmail.com> > > > > >> wrote: > > > > >> > > > > >> > Denis, > > > > >> > > > > > >> > Sounds reasonable. It's not clear, though, what should happen, > if > > a > > > > >> joining > > > > >> > node has some services persisted, that are missing on other > nodes. > > > > >> > Should we deploy them? > > > > >> > If we do so, it could lead to surprising behaviour. For example > > you > > > > >> could > > > > >> > kill a node, undeploy a service, then bring back an old node, > and > > it > > > > >> would > > > > >> > make the service resurrect. > > > > >> > We could store some deployment counter along with the service > > > > >> > configurations on all nodes, that would show how many times the > > > > service > > > > >> > state has changed, i.e. it has been undeployed/redeployed. It > > should > > > > be > > > > >> > kept for undeployed services as well to avoid situations like I > > > > >> described. > > > > >> > > > > > >> > But it still leaves a possibility of incorrect behaviour, if > there > > > > was a > > > > >> > split-brain situation at some point. I don't think we should > > precess > > > > it > > > > >> > somehow, though. If we choose to tackle it, it will > overcomplicate > > > > >> things > > > > >> > for a sake of a minor improvement. > > > > >> > > > > > >> > Denis > > > > >> > > > > > >> > вт, 10 апр. 2018 г. в 0:55, Valentin Kulichenko < > > > > >> > valentin.kuliche...@gmail.com>: > > > > >> > > > > > >> > > I was responding to another Denis :) Agree with you on your > > point > > > > >> though. > > > > >> > > > > > > >> > > -Val > > > > >> > > > > > > >> > > On Mon, Apr 9, 2018 at 2:48 PM, Denis Magda < > dma...@apache.org> > > > > >> wrote: > > > > >> > > > > > > >> > > > Val, > > > > >> > > > > > > > >> > > > Guess we're talking about other situations. I'm bringing up > > the > > > > case > > > > >> > > when a > > > > >> > > > service was deployed dynamically and has to be brought up > > after > > > a > > > > >> full > > > > >> > > > cluster restart w/o user intervention. To achieve this we > need > > > to > > > > >> > persist > > > > >> > > > the service's configuration somewhere. > > > > >> > > > > > > > >> > > > -- > > > > >> > > > Denis > > > > >> > > > > > > > >> > > > On Mon, Apr 9, 2018 at 1:42 PM, Valentin Kulichenko < > > > > >> > > > valentin.kuliche...@gmail.com> wrote: > > > > >> > > > > > > > >> > > > > Denis, > > > > >> > > > > > > > > >> > > > > EVT_CLASS_DEPLOYED should be fired every time a class is > > > > deployed > > > > >> or > > > > >> > > > > redeployed. If this doesn't happen in some cases, I > believe > > > this > > > > >> > would > > > > >> > > > be a > > > > >> > > > > bug. I don't think we need to add any new events. > > > > >> > > > > > > > > >> > > > > -Val > > > > >> > > > > > > > > >> > > > > On Mon, Apr 9, 2018 at 10:50 AM, Denis Magda < > > > dma...@apache.org > > > > > > > > > >> > > wrote: > > > > >> > > > > > > > > >> > > > > > Denis, > > > > >> > > > > > > > > > >> > > > > > I would encourage us to persist a service configuration > in > > > the > > > > >> meta > > > > >> > > > store > > > > >> > > > > > and have this capability enabled by default. That's > > > essential > > > > >> for > > > > >> > > > > services > > > > >> > > > > > started dynamically. Moreover, we support similar > behavior > > > for > > > > >> > > caches, > > > > >> > > > > > indexes, and other DDL changes happened at runtime. > > > > >> > > > > > > > > > >> > > > > > -- > > > > >> > > > > > Denis > > > > >> > > > > > > > > > >> > > > > > On Mon, Apr 9, 2018 at 9:34 AM, Denis Mekhanikov < > > > > >> > > > dmekhani...@gmail.com> > > > > >> > > > > > wrote: > > > > >> > > > > > > > > > >> > > > > > > Another question, that I would like to discuss is > > whether > > > > >> > services > > > > >> > > > > should > > > > >> > > > > > > be preserved on cluster restarts. > > > > >> > > > > > > > > > > >> > > > > > > Currently it depends on persistence configuration. If > > > > >> persistence > > > > >> > > for > > > > >> > > > > any > > > > >> > > > > > > data region is enabled, then services will be > persisted > > as > > > > >> well. > > > > >> > > This > > > > >> > > > > is > > > > >> > > > > > a > > > > >> > > > > > > pretty strange way of configuring this behaviour. > > > > >> > > > > > > I'm not sure, if anybody relies on this functionality > > > right > > > > >> now. > > > > >> > > > Should > > > > >> > > > > > we > > > > >> > > > > > > support it at all? If yes, should we make it > > configurable? > > > > >> > > > > > > > > > > >> > > > > > > Denis > > > > >> > > > > > > > > > > >> > > > > > > пн, 9 апр. 2018 г. в 19:27, Denis Mekhanikov < > > > > >> > > dmekhani...@gmail.com > > > > >> > > > >: > > > > >> > > > > > > > > > > >> > > > > > > > Val, > > > > >> > > > > > > > > > > > >> > > > > > > > Sounds reasonable. I just think, that user should > have > > > > some > > > > >> way > > > > >> > > to > > > > >> > > > > > know, > > > > >> > > > > > > > that new version of a service class was deployed. > > > > >> > > > > > > > One way to do it is to listen to > *EVT_CLASS_DEPLOYED. > > > *I'm > > > > >> not > > > > >> > > > sure, > > > > >> > > > > > > > whether it is triggered on class redeployment, > though. > > > If > > > > >> not, > > > > >> > > then > > > > >> > > > > > > another > > > > >> > > > > > > > event type should be added. > > > > >> > > > > > > > > > > > >> > > > > > > > I don't think, that a lot of people will implement > > their > > > > own > > > > >> > > > > > > > *DeploymentSpi*-s, so we should make work with > > > > >> > *UriDeploymentSpi* > > > > >> > > > as > > > > >> > > > > > > > comfortable as possible. > > > > >> > > > > > > > > > > > >> > > > > > > > Denis > > > > >> > > > > > > > > > > > >> > > > > > > > пт, 6 апр. 2018 г. в 23:40, Valentin Kulichenko < > > > > >> > > > > > > > valentin.kuliche...@gmail.com>: > > > > >> > > > > > > > > > > > >> > > > > > > >> Yes, the class deployment itself has to be > explicit. > > > > I.e., > > > > >> > there > > > > >> > > > has > > > > >> > > > > > to > > > > >> > > > > > > be > > > > >> > > > > > > >> a manual step where user updates the class, and the > > > exact > > > > >> step > > > > >> > > > > > required > > > > >> > > > > > > >> would depend on DeploymentSpi implementation. But > > then > > > > >> Ignite > > > > >> > > > takes > > > > >> > > > > > care > > > > >> > > > > > > >> of > > > > >> > > > > > > >> everything else - service redeployment and restart > is > > > > >> > automatic. > > > > >> > > > > > > >> > > > > >> > > > > > > >> Dmitriy Pavlov, all this is going to be disabled if > > > > >> > > DeploymentSpi > > > > >> > > > is > > > > >> > > > > > not > > > > >> > > > > > > >> configured. In this case service class definitions > > have > > > > to > > > > >> be > > > > >> > > > > deployed > > > > >> > > > > > > on > > > > >> > > > > > > >> local classpath and can't be updated in runtime. > Just > > > > like > > > > >> it > > > > >> > > > works > > > > >> > > > > > > right > > > > >> > > > > > > >> now. > > > > >> > > > > > > >> > > > > >> > > > > > > >> -Val > > > > >> > > > > > > >> > > > > >> > > > > > > >> On Fri, Apr 6, 2018 at 10:20 AM, Dmitriy Setrakyan > < > > > > >> > > > > > > dsetrak...@apache.org > > > > >> > > > > > > >> > > > > > >> > > > > > > >> wrote: > > > > >> > > > > > > >> > > > > >> > > > > > > >> > On Fri, Apr 6, 2018 at 9:13 AM, Dmitry Pavlov < > > > > >> > > > > > dpavlov....@gmail.com> > > > > >> > > > > > > >> > wrote: > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > > Hi Igniters, > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > I like automatic redeploy which can be disabled > > by > > > > >> config > > > > >> > if > > > > >> > > > > user > > > > >> > > > > > > >> wants > > > > >> > > > > > > >> > to > > > > >> > > > > > > >> > > control this process. What do you think? > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > I do not think we should have anything automatic > > when > > > > it > > > > >> > comes > > > > >> > > > to > > > > >> > > > > > > >> > deployment, everything should be explicit. > However, > > > if > > > > we > > > > >> > use > > > > >> > > > the > > > > >> > > > > > > >> > deployment SPI, then a user should be able to do > > > "hot" > > > > >> > > redeploy, > > > > >> > > > > > > where a > > > > >> > > > > > > >> > new service will be deployed if the user drops an > > > > updated > > > > >> > jar. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > We should not create anything new here. Ignite > > > already > > > > >> has a > > > > >> > > > > > > deployment > > > > >> > > > > > > >> SPI > > > > >> > > > > > > >> > and it already works in a certain way. Let's not > > > change > > > > >> it. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > D. > > > > >> > > > > > > >> > > > > > >> > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
