Hi,
By the way OSprofiler trace shows how this regression impacts on amount of DB queries done by Keystone (during the boot of VM): http://boris-42.github.io/b2.html Best regards, Boris Pavlovic On Wed, Jan 20, 2016 at 3:30 PM, Morgan Fainberg <morgan.fainb...@gmail.com> wrote: > As promised here are the fixes: > > > https://review.openstack.org/#/q/Ifc17c27744dac5ad55e84752ca6f68169c2f5a86,n,z > > Proposed to both master and liberty. > > On Wed, Jan 20, 2016 at 12:15 PM, Sean Dague <s...@dague.net> wrote: > >> On 01/20/2016 02:59 PM, Morgan Fainberg wrote: >> > So this was due to a change in keystonemiddleware. We stopped doing >> > in-memory caching of tokens per process, per worker by default [1]. >> > There are a couple of reasons: >> > >> > 1) in-memory caching produced unreliable validation because some >> > processed may have a cache, some may not >> > 2) in-memory caching was unbounded memory wise per worker. >> > >> > I'll spin up a devstack change to enable memcache and use the memcache >> > caching for keystonemiddleware today. This will benefit things in a >> > couple ways >> > >> > * All services and all service's workers will share the offload of the >> > validation, likely producing a real speedup even over the old in-memory >> > caching >> > * There will no longer be inconsistent validation offload/responses >> > based upon which worker you happen to hit for a given service. >> > >> > I'll post to the ML here with the proposed change later today. >> > >> > [1] >> > >> https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52 >> >> This seems like a pretty substantial performance impact. Was there a >> reno associated with this? >> >> I think that we should still probably: >> >> * != the keystone middleware version, it's impacting the ability to land >> fixes in the gate >> * add devstack memcache code >> * find some way to WARN if we are running without memcache config, so >> people realize they are in a regressed state >> * add back keystone middleware at that version >> >> -Sean >> >> > >> > Cheers, >> > --Morgan >> > >> > On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <arma...@gmail.com >> > <mailto:arma...@gmail.com>> wrote: >> > >> > >> > >> > On 19 January 2016 at 22:46, Kevin Benton <blak...@gmail.com >> > <mailto:blak...@gmail.com>> wrote: >> > >> > Hi all, >> > >> > We noticed a major jump in the neutron tempest and API test run >> > times recently in Neutron. After digging through logstash I >> > found out that it first occurred on the requirements bump here: >> > https://review.openstack.org/#/c/265697/ >> > >> > After locally testing each requirements change individually, I >> > found that the keystonemiddleware change seems to be the >> > culprit. It almost doubles the time it takes to fulfill simple >> > port-list requests in Neutron. >> > >> > Armando pushed up a patch here to >> > confirm: https://review.openstack.org/#/c/270024/ >> > Once that's verified, we should probably put a cap on the >> > middleware because it's causing the tests to run up close to >> > their time limits. >> > >> > >> > Kevin, >> > >> > As usual your analytical skills are to be praised. >> > >> > I wonder if anyone else is aware of the issue/s, because during the >> > usual hunting I could see other projects being affected and showing >> > abnormally high run times of the dsvm jobs. >> > >> > I am not sure that [1] is the right approach, but it should give us >> > some data points if executed successfully. >> > >> > Cheers, >> > Armando >> > >> > [1] https://review.openstack.org/#/c/270024/ >> > >> > >> > -- >> > Kevin Benton >> > >> > >> __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > < >> http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > >> __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > < >> http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > >> > >> __________________________________________________________________________ >> > OpenStack Development Mailing List (not for usage questions) >> > Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> >> -- >> Sean Dague >> http://dague.net >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
