On Fri, Mar 17, 2017 at 3:11 PM, Sean Dague <[email protected]> wrote: > On 03/17/2017 09:24 AM, Jordan Pittier wrote: > > > > > > On Fri, Mar 17, 2017 at 1:58 PM, Sean Dague <[email protected] > > <mailto:[email protected]>> wrote: > > > > On 03/17/2017 08:27 AM, Jordan Pittier wrote: > > > The patch that reduced the number of Tempest Scenarios we run in > every > > > job and also reduce the test run concurrency [0] was merged 13 > days ago. > > > Since, the situation (i.e the high number of false negative job > results) > > > has not improved significantly. We need to keep looking > collectively at > > > this. > > > > While the situation hasn't completely cleared out - > > http://tinyurl.com/mdmdxlk - since we've merged this we've not seen > that > > job go over 25% failure rate in the gate, which it was regularly > > crossing in the prior 2 week period. That does feel like progress. In > > spot checking I we are also rarely failing in scenario tests now, but > > the fails tend to end up inside heavy API tests running in parallel. > > > > > > > There seems to be an agreement that we are hitting some memory > limit. > > > Several of our most frequent failures are memory related [1]. So we > > > should either reduce our memory usage or ask for bigger VMs, with > more > > > than 8GB of RAM. > > > > > > There was/is several attempts to reduce our memory usage, by > reducing > > > the Mysql memory consumption ([2] but quickly reverted [3]), > reducing > > > the number of Apache workers ([4], [5]), more apache2 tuning [6]. > If you > > > have any crazy idea to help in this regard, please help. This is > high > > > priority for the whole openstack project, because it's plaguing > many > > > projects. > > > > Interesting, I hadn't seen the revert. It is also curious that it was > > largely limitted to the neutron-api test job. It's also notable that > the > > sort buffers seem to have been set to the minimum allowed limit of > mysql > > - > > https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters. > html#sysvar_innodb_sort_buffer_size > > <https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters. > html#sysvar_innodb_sort_buffer_size> > > - and is over an order of magnitude decrease from the existing > default. > > > > I wonder about redoing the change with everything except it and > seeing > > how that impacts the neutron-api job. > > > > Yes, that would be great because mysql is by far our biggest memory > > consumer so we should target this first. > > While it is the single biggest process, weighing in at 500 MB, the > python services are really our biggest memory consumers. They are > collectively far outweighing either mysql or rabbit, and are the reason > that even with 64MB guests we're running out of memory. So we want to > keep that under perspective. > Absolutely. I have https://review.openstack.org/#/c/446986/ in that vain. And if someone wants to start the work of not running the several Swift *auditor*, *updater*, *reaper*, *replicator* services, in case the Swift Replication factor is set to 1, that's also a good memory saving.
> > -Sean > > -- > Sean Dague > http://dague.net > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
