> -----Original Message----- > From: openstack-bounces+armando.migliaccio=eu.citrix....@lists.launchpad.net > [mailto:openstack- > bounces+armando.migliaccio=eu.citrix....@lists.launchpad.net] On Behalf Of Jay > Pipes > Sent: 02 March 2012 15:17 > To: openstack@lists.launchpad.net > Subject: Re: [Openstack] eventlet weirdness > > On 03/02/2012 05:34 AM, Day, Phil wrote: > > In our experience (running clusters of several hundred nodes) the DB > performance is not generally the significant factor, so making its calls non- > blocking gives only a very small increase in processing capacity and creates > other side effects in terms of slowing all eventlets down as they wait for > their turn to run. > > Yes, I believe I said that this was the case at the last design summit > -- or rather, I believe I said "is there any evidence that the database is a > performance or scalability problem at all"? > > > That shouldn't really be surprising given that the Nova DB is pretty small > and MySQL is a pretty good DB - throw reasonable hardware at the DB server and > give it a bit of TLC from a DBA (remove deleted entries from the DB, add > indexes where the slow query log tells you to, etc) and it shouldn't be the > bottleneck in the system for performance or scalability. > > ++ > > > We use the python driver and have experimented with allowing the eventlet > code to make the db calls non-blocking (its not the default setting), and it > works, but didn't give us any significant advantage. > > Yep, identical results to the work that Mark Washenberger did on the same > subject. > > > For example in the API server (before we made it properly > > multi-threaded) > > By "properly multi-threaded" are you instead referring to making the nova-api > server multi-*processed* with eventlet greenthread pools in each process? i.e. > The way Swift (and now Glance) works? Or are you referring to a different > approach entirely? > > > with blocking db calls the server was essentially a serial processing queue > - each request was fully processed before the next. With non-blocking db > calls we got a lot more apparent concurrencybut only at the expense of making > all of the requests equally bad. > > Yep, not surprising. > > > Consider a request takes 10 seconds, where after 5 seconds there is a call > to the DB which takes 1 second, and three are started at the same time: > > > > Blocking: > > 0 - Request 1 starts > > 10 - Request 1 completes, request 2 starts > > 20 - Request 2 completes, request 3 starts > > 30 - Request 3 competes > > Request 1 completes in 10 seconds > > Request 2 completes in 20 seconds > > Request 3 completes in 30 seconds > > Ave time: 20 sec > > > > Non-blocking > > 0 - Request 1 Starts > > 5 - Request 1 gets to db call, request 2 starts > > 10 - Request 2 gets to db call, request 3 starts > > 15 - Request 3 gets to db call, request 1 resumes > > 19 - Request 1 completes, request 2 resumes > > 23 - Request 2 completes, request 3 resumes > > 27 - Request 3 completes > > > > Request 1 completes in 19 seconds (+ 9 seconds) Request 2 completes > > in 24 seconds (+ 4 seconds) Request 3 completes in 27 seconds (- 3 > > seconds) Ave time: 20 sec > > > > So instead of worrying about making db calls non-blocking we've been working > to make certain eventlets non-blocking - i.e. add sleep(0) calls to long > running iteration loops - which IMO has a much bigger impact on the > performance of the apparent latency of the system. > > Yep, and I think adding a few sleep(0) calls in various places in the Nova > codebase (as was recently added in the _sync_power_states() periodic task) is > an easy and simple win with pretty much no ill side-effects. :)
I'd be cautious to say that no ill side-effects were introduced. I found a race condition right in the middle of sync_power_states, which I assume was exposed by "breaking" the task deliberately. > > Curious... do you have a list of all the places where sleep(0) calls were > inserted in the HP Nova code? I can turn that into a bug report and get to > work on adding them... > > All the best, > -jay > > > Phil > > > > > > > > -----Original Message----- > > From: openstack-bounces+philip.day=hp....@lists.launchpad.net > > [mailto:openstack-bounces+philip.day=hp....@lists.launchpad.net] On > > Behalf Of Brian Lamar > > Sent: 01 March 2012 21:31 > > To: openstack@lists.launchpad.net > > Subject: Re: [Openstack] eventlet weirdness > > > >>> How is MySQL access handled in eventlet? Presumably it's external C > >>> library so it's not going to be monkey patched. Does that make every > >>> db access call a blocking call? Thanks, > > > >> Nope, it goes through a thread pool. > > > > I feel like this might be an over-simplification. If the question is: > > > > "How is MySQL access handled in nova?" > > > > The answer would be that we use SQLAlchemy which can load any number of SQL- > drivers. These drivers can be either pure Python or C-based drivers. In the > case of pure Python drivers, monkey patching can occur and db calls are non- > blocking. In the case of drivers which contain C code (or perhaps other > blocking calls), db calls will most likely be blocking. > > > > If the question is "How is MySQL access handled in eventlet?" the answer > would be to use the eventlet.db_pool module to allow db access using thread > pools. > > > > B > > > > -----Original Message----- > > From: "Adam Young"<ayo...@redhat.com> > > Sent: Thursday, March 1, 2012 3:27pm > > To: openstack@lists.launchpad.net > > Subject: Re: [Openstack] eventlet weirdness > > > > On 03/01/2012 02:45 PM, Yun Mao wrote: > >> There are plenty eventlet discussion recently but I'll stick my > >> question to this thread, although it's pretty much a separate > >> question. :) > >> > >> How is MySQL access handled in eventlet? Presumably it's external C > >> library so it's not going to be monkey patched. Does that make every > >> db access call a blocking call? Thanks, > > > > Nope, it goes through a thread pool. > >> > >> Yun > >> > >> On Wed, Feb 29, 2012 at 9:18 PM, Johannes Erdfelt<johan...@erdfelt.com> > wrote: > >>> On Wed, Feb 29, 2012, Yun Mao<yun...@gmail.com> wrote: > >>>> Thanks for the explanation. Let me see if I understand this. > >>>> > >>>> 1. Eventlet will never have this problem if there is only 1 OS > >>>> thread > >>>> -- let's call it main thread. > >>> In fact, that's exactly what Python calls it :) > >>> > >>>> 2. In Nova, there is only 1 OS thread unless you use xenapi and/or > >>>> the virt/firewall driver. > >>>> 3. The python logging module uses locks. Because of the monkey > >>>> patch, those locks are actually eventlet or "green" locks and may > >>>> trigger a green thread context switch. > >>>> > >>>> Based on 1-3, does it make sense to say that in the other OS > >>>> threads (i.e. not main thread), if logging (plus other pure python > >>>> library code involving locking) is never used, and we do not run a > >>>> eventlet hub at all, we should never see this problem? > >>> That should be correct. I'd have to double check all of the monkey > >>> patching that eventlet does to make sure there aren't other cases > >>> where you may inadvertently use eventlet primitives across real threads. > >>> > >>> JE > >>> > >>> > >>> _______________________________________________ > >>> Mailing list: https://launchpad.net/~openstack > >>> Post to : openstack@lists.launchpad.net > >>> Unsubscribe : https://launchpad.net/~openstack > >>> More help : https://help.launchpad.net/ListHelp > >> _______________________________________________ > >> Mailing list: https://launchpad.net/~openstack > >> Post to : openstack@lists.launchpad.net > >> Unsubscribe : https://launchpad.net/~openstack > >> More help : https://help.launchpad.net/ListHelp > > > > > > _______________________________________________ > > Mailing list: https://launchpad.net/~openstack > > Post to : openstack@lists.launchpad.net > > Unsubscribe : https://launchpad.net/~openstack > > More help : https://help.launchpad.net/ListHelp > > > > > > > > _______________________________________________ > > Mailing list: https://launchpad.net/~openstack > > Post to : openstack@lists.launchpad.net > > Unsubscribe : https://launchpad.net/~openstack > > More help : https://help.launchpad.net/ListHelp > > > > _______________________________________________ > > Mailing list: https://launchpad.net/~openstack > > Post to : openstack@lists.launchpad.net > > Unsubscribe : https://launchpad.net/~openstack > > More help : https://help.launchpad.net/ListHelp > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
