I can tell you as a user that if any library I was using in my application called System.exit without my consent would result in a lot of frustration.
If ignite enters an unrecoverable state then I think that is something that should be observable locally, similar to node segmentation and then the application can decide the best course of action. Of course, if ignite was started as a standalone process do what you think is best, but don't think you can kill the process without backlash from users if it's running in embedded mode. - Nick On Tue, Mar 13, 2018, 5:12 PM Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Ivan, > > If grid hangs, graceful shutdown would most likely hang as well. Almost > never you can recover from a bad state using graceful procedures. > > I agree that we should not create two defaults, especially in this case. > It's not even strictly defined what is embedded node in Ignite. For > example, if I start it using a custom main class and/or custom script > instead of ignite.sh, would it be embedded or standalone node? > > -Val > > On Tue, Mar 13, 2018 at 4:58 PM, Ivan Rakov <ivan.glu...@gmail.com> wrote: > > > One more note: "kill if standalone, stop if embedded" differs from what > > you are suggesting "try graceful, then kill process regardless" only in > > case when graceful shutdown hangs. > > Do we have understanding, how often does graceful shutdown hang? > > Obviously, *grid hang* is often case, but it shouldn't be messed with > > *graceful shutdown hang*. From my experience, if something went wrong, > > users just prefer to do kill -9 because it's much more reliable and > easy. > > Probably, in most of cases when kill -9 worked, graceful stop would have > > worked as well - we just don't have such statistics. > > It may be bad example, but: in our CI tests we intentionally break grid > in > > many harsh ways and perform a graceful stop after the test execution, and > > it doesn't hang - otherwise we'd see many "Execution timeout" test suite > > hangs. > > > > Best Regards, > > Ivan Rakov > > > > > > On 14.03.2018 2:24, Dmitriy Setrakyan wrote: > > > >> On Tue, Mar 13, 2018 at 7:13 PM, Ivan Rakov <ivan.glu...@gmail.com> > >> wrote: > >> > >> I just would like to add my +1 for "kill if standalone, stop if > embedded" > >>> default option. My arguments: > >>> > >>> 1) Regarding "If Ignite hangs - it will likely be impossible to stop": > >>> Unfortunately, it's true that Ignite can hang during stop procedure. > >>> However, most of failures described under IEP-14 (storage IO > exceptions, > >>> death of critical system worker thread, etc) normally shouldn't turn > node > >>> into "impossible to stop" state. Turning into that state is a bug > >>> itself. I > >>> guess that we shouldn't choose system behavior on the basis of known > >>> bugs. > >>> > >> > >> The whole discussion is about protecting against force-major issues, > >> including Ignite bugs. You are assuming that a user application will > >> somehow continue to function if an Ignite node is stopped. In most cases > >> it > >> will just freeze itself and cause the rest of the application to hang. > >> > >> Again, "kill+stop" is the most deterministic and the safest default > >> behavior. Try a graceful shutdown (which will make restart easier), and > >> then kill the process regardless. > >> > >> Note that we are arguing about the default behavior. If a user does not > >> like this default, then this user can change it to another behavior. > >> > >> > >> 2) User might want to handle Ignite node crash before shutting down the > >>> whole JVM - raise alert, close external resources, etc > >>> > >>> Very unlikely, but if a user is this advanced, then this user can > change > >> the default behavior. Most users will not even know how to configure > such > >> custom shutdown behavior and would prefer an automatic kill. > >> > >> 3) IEP-14 document has important notes: "More than one Ignite node could > >> be > >> > >>> started in one JVM process" and "Different nodes in one JVM process > could > >>> belong to different clusters". This is possible only in embedded mode. > I > >>> think, we shouldn't shock user by sudden JVM halt (possibly, along with > >>> another healthy nodes) if there's a chance of successful node stop. > >>> > >>> Has anyone actually seen a real example of that? I have not. This > >> scenario > >> is extremely unlikely and should not define the default behavior. Again, > >> if > >> a user is so advanced to come up with such a sophisticated deployment, > >> then > >> the same user should be able to set different default behaviors for > >> different clusters. > >> > >> > > >
