So this was due to a change in keystonemiddleware. We stopped doing in-memory caching of tokens per process, per worker by default [1]. There are a couple of reasons:
1) in-memory caching produced unreliable validation because some processed may have a cache, some may not 2) in-memory caching was unbounded memory wise per worker. I'll spin up a devstack change to enable memcache and use the memcache caching for keystonemiddleware today. This will benefit things in a couple ways * All services and all service's workers will share the offload of the validation, likely producing a real speedup even over the old in-memory caching * There will no longer be inconsistent validation offload/responses based upon which worker you happen to hit for a given service. I'll post to the ML here with the proposed change later today. [1] https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52 Cheers, --Morgan On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <arma...@gmail.com> wrote: > > > On 19 January 2016 at 22:46, Kevin Benton <blak...@gmail.com> wrote: > >> Hi all, >> >> We noticed a major jump in the neutron tempest and API test run times >> recently in Neutron. After digging through logstash I found out that it >> first occurred on the requirements bump here: >> https://review.openstack.org/#/c/265697/ >> >> After locally testing each requirements change individually, I found that >> the keystonemiddleware change seems to be the culprit. It almost doubles >> the time it takes to fulfill simple port-list requests in Neutron. >> >> Armando pushed up a patch here to confirm: >> https://review.openstack.org/#/c/270024/ >> Once that's verified, we should probably put a cap on the middleware >> because it's causing the tests to run up close to their time limits. >> > > Kevin, > > As usual your analytical skills are to be praised. > > I wonder if anyone else is aware of the issue/s, because during the usual > hunting I could see other projects being affected and showing abnormally > high run times of the dsvm jobs. > > I am not sure that [1] is the right approach, but it should give us some > data points if executed successfully. > > Cheers, > Armando > > [1] https://review.openstack.org/#/c/270024/ > > >> -- >> Kevin Benton >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
