On 2 February 2017 at 12:19, Sean Dague <s...@dague.net> wrote: > On 02/02/2017 02:28 PM, Armando M. wrote: > > > > > > On 2 February 2017 at 10:08, Sean Dague <s...@dague.net > > <mailto:s...@dague.net>> wrote: > > > > On 02/02/2017 12:49 PM, Armando M. wrote: > > > > > > > > > On 2 February 2017 at 08:40, Sean Dague <s...@dague.net <mailto: > s...@dague.net> > > > <mailto:s...@dague.net <mailto:s...@dague.net>>> wrote: > > > > > > On 02/02/2017 11:16 AM, Matthew Treinish wrote: > > > <snip> > > > > <oops, forgot to finish my though> > > > > > > > > We definitely aren't saying running a single worker is how > > we recommend people > > > > run OpenStack by doing this. But it just adds on to the > > differences between the > > > > gate and what we expect things actually look like. > > > > > > I'm all for actually getting to the bottom of this, but > > honestly real > > > memory profiling is needed here. The growth across projects > > probably > > > means that some common libraries are some part of this. The > > ever growing > > > requirements list is demonstrative of that. Code reuse is > > good, but if > > > we are importing much of a library to get access to a couple of > > > functions, we're going to take a bunch of memory weight on that > > > (especially if that library has friendly auto imports in top > level > > > __init__.py so we can't get only the parts we want). > > > > > > Changing the worker count is just shuffling around deck chairs. > > > > > > I'm not familiar enough with memory profiling tools in python > > to know > > > the right approach we should take there to get this down to > > individual > > > libraries / objects that are containing all our memory. Anyone > > more > > > skilled here able to help lead the way? > > > > > > > > > From what I hear, the overall consensus on this matter is to > determine > > > what actually caused the memory consumption bump and how to > > address it, > > > but that's more of a medium to long term action. In fact, to me > > this is > > > one of the top priority matters we should talk about at the > > imminent PTG. > > > > > > For the time being, and to provide relief to the gate, should we > > want to > > > lock the API_WORKERS to 1? I'll post something for review and see > how > > > many people shoot it down :) > > > > I don't think we want to do that. It's going to force down the > eventlet > > API workers to being a single process, and it's not super clear that > > eventlet handles backups on the inbound socket well. I honestly would > > expect that creates different hard to debug issues, especially with > high > > chatter rates between services. > > > > > > I must admit I share your fear, but out of the tests that I have > > executed so far in [1,2,3], the house didn't burn in a fire. I am > > looking for other ways to have a substantial memory saving with a > > relatively quick and dirty fix, but coming up empty handed thus far. > > > > [1] https://review.openstack.org/#/c/428303/ > > [2] https://review.openstack.org/#/c/427919/ > > [3] https://review.openstack.org/#/c/427921/ > > This failure in the first patch - > http://logs.openstack.org/03/428303/1/check/gate-tempest- > dsvm-neutron-full-ubuntu-xenial/71f42ea/logs/screen-n- > api.txt.gz?level=TRACE#_2017-02-02_19_14_11_751 > looks exactly like I would expect by API Worker starvation. >
Not sure I agree on this one, this has been observed multiple times in the gate already [1] (though I am not sure there's a bug for it), and I don't believe it has anything to do with the number of API workers, unless not even two workers are enough. [1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22('Connection%20aborted.'%2C%20BadStatusLine(%5C%22''%5C%22%2C)%5C%22 > -Sean > > -- > Sean Dague > http://dague.net > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
