On Wed, 2014-05-28 at 11:08 -0700, Devananda van der Veen wrote: > While I appreciate the many ideas being discussed here (some of which > we've explored previously and agreed to continue exploring), there is > a fundamental difference vs. what I propose in that spec. I believe > that what I'm proposing will be achievable without any significant > visible changes in the API -- no new API end points or resources, and > the client interaction will be nearly the same. I am just afraid we'll start to get complains like "I ran a request and it disappeared to nowhere, what should I do?" Are we ready to handle this kinds of things (happened because of queue overloaded, user too impatient, some validation failed etc)? I may be misunderstanding, but it looks to me like we'll simply drop requests we can't fulfill. That may be frustrating for a user.
> A few status codes may be different in certain circumstances -- but > it will not require a new major version of the REST API. And it solves > a scalability and stability problem that folks are encountering today. > (It seems my spec didn't describe those problems well enough -- I'm > updating it now.) Would be great! > > > Cheers, > Devananda > > > > > On Wed, May 28, 2014 at 10:14 AM, Maksym Lobur <mlo...@mirantis.com> > wrote: > BTW a very similar discussion is going in Neutron community > right now, please find a thread under the [openstack-dev] > [Neutron] Introducing task oriented workflows label. > > Best regards, > Max Lobur, > Python Developer, Mirantis, Inc. > > > Mobile: +38 (093) 665 14 28 > Skype: max_lobur > > > 38, Lenina ave. Kharkov, Ukraine > www.mirantis.com > www.mirantis.ru > > > > On Wed, May 28, 2014 at 6:56 PM, Maksym Lobur > <mlo...@mirantis.com> wrote: > Hi All, > > > You've raised a good discussion, something similar > already was started back in february. Could someone > please find the long etherpad with discussion between > Deva and Lifeless, as I recall most of the points > mentioned above have a good comments there. > > > Up to this point I have the only idea how to elegantly > address these problems. This is a tasks concept and > probably a scheduler service, which not necessarily > should be separate from the API at the moment (Finally > we already have a hash ring on the api side which is a > kind of scheduler right?) It was already proposed > earlier, but I would like to try to fit all these > issues into this concept. > > 1. "Executability" > We need to make sure that request can be > theoretically executed, > which includes: > a) Validating request body > > We cannot validate everything on the API side, relying > on the fact that DB state is actual is not a good > idea, especially under heavy load. > > > In tasks concept we could assume that all the requests > are executable, and do not perform any validation in > the API thread at all. Instead of this the API will > just create a task and return it's ID to the user. > Task scheduler may perform some minor validations > before the task is queued or started for convenience, > but they should be duplicated inside task body because > there is an arbitrary time between queuing up and > start ((c)lifeless). I assume the scheduler will have > it's own thread or even process. The user will need to > poke received ID to know the current state of his > submission. > > b) For each of entities (e.g. nodes) touched, > check that they are > available > at the moment (at least exist). > This is arguable, as checking for entity > existence requires going to > DB. > > Same here, DB round trip is a potential block, > therefore this will be done inside task (after it's > queued and started) and will not affect the API. The > user will just observe the task state by poking the > API (or using callback as an option). > > > 2. Appropriate state > For each entity in question, ensure that it's > either in a proper state > or > moving to a proper state. > It would help avoid users e.g. setting deploy > twice on the same node > It will still require some kind of > NodeInAWrongStateError, but we won't > necessary need a client retry on this one. > Allowing the entity to be _moving_ to > appropriate state gives us a > problem: > Imagine OP1 was running and OP2 got scheduled, > hoping that OP1 will come > to desired state. What if OP1 fails? What if > conductor, doing OP1 > crashes? > > Let's say OP1 and OP2 are two separate tasks. Each one > have the initial state validation inside it's body. > Once OP2 gets its turn it will perform validation and > fail, which looks reasonable to me. > > Similar problem with checking node state. > Imagine we schedule OP2 while we had OP1 - > regular checking node state. > OP1 discovers that node is actually absent and > puts it to maintenance > state. > What to do with OP2? > > The task will fail once it get it's turn. > > b) Can we make client wait for the results of > periodic check? > That is, wait for OP1 _before scheduling_ > OP2? > > > We will just schedule the task and the user will > observe its progress, once OP1 is finished and OP2 > started - he will see a fail. > > 3. Status feedback > People would like to know, how things are > going with their task. > What they know is that their request was > scheduled. Options: > a) Poll: return some REQUEST_ID and expect > users to poll some endpoint. > Pros: > - Should be easy to implement > Cons: > - Requires persistent storage for tasks. > Does AMQP allow to do this > kinds > of queries? If not, we'll need to > duplicate tasks in DB. > - Increased load on API instances and DB > > Exactly described the tasks concept :) > > b) Callback: take endpoint, call it once task > is done/fails. > Pros: > - Less load on both client and server > - Answer exactly when it's ready > Cons: > - Will not work for cli and similar > - If conductor crashes, there will be no > callback. > > > Add to Cons: > - Callback is not reliable since it may get lost. > We should have an ability to poke anyway, though I see > a great benefit from implementing a callbacks - to > decrease API load. > > 4. Debugging consideration > a) This is an open question: how to debug, if > we have a lot of requests > and something went wrong? > > > We will be able to see the queue state (btw what about > security here, should the user be able to see all the > tasks, or just his ones, or all but others with hidden > details). > > b) One more thing to consider: how to make > command like `node-show` > aware of > scheduled transitioning, so that people > don't try operations that are > doomed to failure. > > > node-show will always show current state of the node, > though we may check if there are any tasks queued or > going, which will change the state. If any - add a > notification to the response. > > 5. Performance considerations > a) With async approach, users will be able to > schedule nearly unlimited > number of tasks, thus essentially blocking > work of Ironic, without > any > signs of the problem (at least for some > time). > I think there are 2 common answers to this > problem: > - Request throttling: disallow user to make > too many requests in some > amount of time. Send them 503 with > Retry-After header set. > > > Can this be achieved by some web-server settings? > Looks like a typical problem. > > - Queue management: watch queue length, > deny new requests if it's too > large. > > > Yes, I really like the limited queue size idea. Please > see my comments in the spec. > Also, if we have a tasks and the queue, we could merge > similar tasks > > > b) State framework from (2), if invented, can > become a bottleneck as > well. > Especially with polling approach. > > > True. > If we have tasks, all the node actions will be done > through them. We can synchronise node state with DB > only during the task, and remove periodic syncs. > Off-course someone may go and turn off the node, in > this case the Ironic will lie about the node state > until some task is executed on this node, which may be > suitable behaviour. Otherwise rare periodic syncs may > work as well. > > 6. Usability considerations > a) People will be unaware, when and whether > their request is going to be > finished. As there will be tempted to > retry, we may get flooded by > duplicates. I would suggest at least make > it possible to request > canceling > any task (which will be possible only if it > is not started yet, > obviously). > > > Since we will have a limited number of kinds of tasks, > we could calculate some estimates basing on previous > similar tasks. Looks like an improvement for a distant > future. In the end I wouldn't want Ironic to perform > estimates like windows's copy-paste dialog :) > > > Tasks may be easily interrupted while they are in a > queue. But if it's already started - there's a > separate > dicsussion > https://blueprints.launchpad.net/ironic/+spec/make-tasks-interruptible (I'm > going to port this bp to the specs repo in some time) > > b) We should try to avoid scheduling > contradictive requests. > > > A task scheduler responsibility: this is basically a > state check before task is scheduled, and it should be > done one more time once the task is started, as > mentioned above. > > c) Can we somehow detect duplicated requests > and ignore them? > E.g. we won't want user to make 2-3-4 > reboots in a row just because > the user > was not patient enough. > > > Queue similar tasks. All the users will be pointed to > the similar task resource, or maybe to a different > resources which tied to the same conductor action. > > Best regards, > Max Lobur, > Python Developer, Mirantis, Inc. > Mobile: +38 (093) 665 14 28 > Skype: max_lobur > 38, Lenina ave. Kharkov, Ukraine > www.mirantis.com > www.mirantis.ru > > > On Wed, May 28, 2014 at 5:10 PM, Lucas Alvares > Gomes <lucasago...@gmail.com> wrote: > On Wed, May 28, 2014 at 2:02 PM, Dmitry > Tantsur <dtant...@redhat.com> wrote: > > Hi Ironic folks, hi Devananda! > > > > I'd like to share with you my thoughts on > asynchronous API, which is > > spec https://review.openstack.org/#/c/94923 > > First I was planned this as comments to the > review, but it proved to be > > much larger, so I post it for discussion on > ML. > > > > Here is list of different consideration, I'd > like to take into account > > when prototyping async support, some are > reflected in spec already, some > > are from my and other's comments: > > > > 1. "Executability" > > We need to make sure that request can be > theoretically executed, > > which includes: > > a) Validating request body > > b) For each of entities (e.g. nodes) > touched, check that they are > > available > > at the moment (at least exist). > > This is arguable, as checking for entity > existence requires going to > > DB. > > > > > > 2. Appropriate state > > For each entity in question, ensure that > it's either in a proper state > > or > > moving to a proper state. > > It would help avoid users e.g. setting > deploy twice on the same node > > It will still require some kind of > NodeInAWrongStateError, but we won't > > necessary need a client retry on this one. > > > > Allowing the entity to be _moving_ to > appropriate state gives us a > > problem: > > Imagine OP1 was running and OP2 got > scheduled, hoping that OP1 will come > > to desired state. What if OP1 fails? What if > conductor, doing OP1 > > crashes? > > That's why we may want to approve only > operations on entities that do > > not > > undergo state changes. What do you think? > > > > Similar problem with checking node state. > > Imagine we schedule OP2 while we had OP1 - > regular checking node state. > > OP1 discovers that node is actually absent > and puts it to maintenance > > state. > > What to do with OP2? > > a) Obvious answer is to fail it > > b) Can we make client wait for the results > of periodic check? > > That is, wait for OP1 _before scheduling_ > OP2? > > > > Anyway, this point requires some state > framework, that knows about > > states, > > transitions, actions and their compatibility > with each other. > For {power, provision} state changes should we > queue the requests? We > may want to only accept 1 request to change > the state per time, if a > second request comes when there's another > state change mid-operation > we may just return 409 (Conflict) to indicate > that a state change is > already in progress. This is similar of what > we have today but instead > of checking the node lock and states on the > conductor side the API > service could do it, since it's on the DB. > > > > 3. Status feedback > > People would like to know, how things are > going with their task. > > What they know is that their request was > scheduled. Options: > > a) Poll: return some REQUEST_ID and expect > users to poll some endpoint. > > Pros: > > - Should be easy to implement > > Cons: > > - Requires persistent storage for tasks. > Does AMQP allow to do this > > kinds > > of queries? If not, we'll need to > duplicate tasks in DB. > > - Increased load on API instances and DB > > b) Callback: take endpoint, call it once > task is done/fails. > > Pros: > > - Less load on both client and server > > - Answer exactly when it's ready > > Cons: > > - Will not work for cli and similar > > - If conductor crashes, there will be no > callback. > > > > Seems like we'd want both (a) and (b) to > comply with current needs. > +1, we could allow pooling by default (like > checking > nodes/<uuid>/states to know the current and > target state of the node) > but we may also want to include a callback > parameter that users could > use to input a URL that the conductor will > call out as soon as the > operation is finished. So if the callback URl > exists, the conductor > will submit a POST request to that URL with > some data structure > identifying the operation and the current > state. > > > > If we have a state framework from (2), we > can also add notifications to > > it. > > > > 4. Debugging consideration > > a) This is an open question: how to debug, > if we have a lot of requests > > and something went wrong? > > b) One more thing to consider: how to make > command like `node-show` > > aware of > > scheduled transitioning, so that people > don't try operations that are > > doomed to failure. > > > > 5. Performance considerations > > a) With async approach, users will be able > to schedule nearly unlimited > > number of tasks, thus essentially > blocking work of Ironic, without > > any > > signs of the problem (at least for some > time). > > I think there are 2 common answers to > this problem: > > - Request throttling: disallow user to > make too many requests in some > > amount of time. Send them 503 with > Retry-After header set. > > - Queue management: watch queue length, > deny new requests if it's too > > large. > > This means actually getting back error > 503 and will require retrying > > again! > > At least it will be exceptional case, and > won't affect Tempest run... > > b) State framework from (2), if invented, > can become a bottleneck as > > well. > > Especially with polling approach. > > > > 6. Usability considerations > > a) People will be unaware, when and whether > their request is going to be > > finished. As there will be tempted to > retry, we may get flooded by > > duplicates. I would suggest at least make > it possible to request > > canceling > > any task (which will be possible only if > it is not started yet, > > obviously). > > b) We should try to avoid scheduling > contradictive requests. > > c) Can we somehow detect duplicated requests > and ignore them? > > E.g. we won't want user to make 2-3-4 > reboots in a row just because > > the user > > was not patient enough. > > > > ------ > > > > Possible takeaways from this letter: > > - We'll need at least throttling to avoid > DoS > > - We'll still need handling of 503 error, > though it should not happen > > under > > normal conditions > > - Think about state framework that unifies > all this complex logic with > > features: > > * Track entities, their states and actions > on entities > > * Check whether new action is compatible > with states of entities it > > touches > > and with other ongoing and scheduled > actions on these entities. > > * Handle notifications for finished and > failed actions by providing > > both > > pull and push approaches. > > * Track whether started action is still > executed, perform error > > notification, > > if not. > > * HA and high performance > > - Think about policies for corner cases > > - Think, how we can make a user aware of > what is going on with both > > request > > and entity that some requests may touch. > Also consider canceling > > requests. > > > > Please let me know, what you think. > > > > Dmitry. > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > + 1 > > > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
