On 03/17/2017 08:27 AM, Jordan Pittier wrote: > The patch that reduced the number of Tempest Scenarios we run in every > job and also reduce the test run concurrency [0] was merged 13 days ago. > Since, the situation (i.e the high number of false negative job results) > has not improved significantly. We need to keep looking collectively at > this.
While the situation hasn't completely cleared out - http://tinyurl.com/mdmdxlk - since we've merged this we've not seen that job go over 25% failure rate in the gate, which it was regularly crossing in the prior 2 week period. That does feel like progress. In spot checking I we are also rarely failing in scenario tests now, but the fails tend to end up inside heavy API tests running in parallel. > There seems to be an agreement that we are hitting some memory limit. > Several of our most frequent failures are memory related [1]. So we > should either reduce our memory usage or ask for bigger VMs, with more > than 8GB of RAM. > > There was/is several attempts to reduce our memory usage, by reducing > the Mysql memory consumption ([2] but quickly reverted [3]), reducing > the number of Apache workers ([4], [5]), more apache2 tuning [6]. If you > have any crazy idea to help in this regard, please help. This is high > priority for the whole openstack project, because it's plaguing many > projects. Interesting, I hadn't seen the revert. It is also curious that it was largely limitted to the neutron-api test job. It's also notable that the sort buffers seem to have been set to the minimum allowed limit of mysql - https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_sort_buffer_size - and is over an order of magnitude decrease from the existing default. I wonder about redoing the change with everything except it and seeing how that impacts the neutron-api job. -Sean -- Sean Dague http://dague.net __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
