On 1/20/2016 2:15 PM, Sean Dague wrote:
On 01/20/2016 02:59 PM, Morgan Fainberg wrote:
So this was due to a change in keystonemiddleware. We stopped doing
in-memory caching of tokens per process, per worker by default [1].
There are a couple of reasons:
1) in-memory caching produced unreliable validation because some
processed may have a cache, some may not
2) in-memory caching was unbounded memory wise per worker.
I'll spin up a devstack change to enable memcache and use the memcache
caching for keystonemiddleware today. This will benefit things in a
couple ways
* All services and all service's workers will share the offload of the
validation, likely producing a real speedup even over the old in-memory
caching
* There will no longer be inconsistent validation offload/responses
based upon which worker you happen to hit for a given service.
I'll post to the ML here with the proposed change later today.
[1]
https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52
This seems like a pretty substantial performance impact. Was there a
reno associated with this?
I think that we should still probably:
* != the keystone middleware version, it's impacting the ability to land
fixes in the gate
* add devstack memcache code
* find some way to WARN if we are running without memcache config, so
people realize they are in a regressed state
* add back keystone middleware at that version
-Sean
Cheers,
--Morgan
On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <arma...@gmail.com
<mailto:arma...@gmail.com>> wrote:
On 19 January 2016 at 22:46, Kevin Benton <blak...@gmail.com
<mailto:blak...@gmail.com>> wrote:
Hi all,
We noticed a major jump in the neutron tempest and API test run
times recently in Neutron. After digging through logstash I
found out that it first occurred on the requirements bump here:
https://review.openstack.org/#/c/265697/
After locally testing each requirements change individually, I
found that the keystonemiddleware change seems to be the
culprit. It almost doubles the time it takes to fulfill simple
port-list requests in Neutron.
Armando pushed up a patch here to
confirm: https://review.openstack.org/#/c/270024/
Once that's verified, we should probably put a cap on the
middleware because it's causing the tests to run up close to
their time limits.
Kevin,
As usual your analytical skills are to be praised.
I wonder if anyone else is aware of the issue/s, because during the
usual hunting I could see other projects being affected and showing
abnormally high run times of the dsvm jobs.
I am not sure that [1] is the right approach, but it should give us
some data points if executed successfully.
Cheers,
Armando
[1] https://review.openstack.org/#/c/270024/
--
Kevin Benton
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
The blacklist on 4.1.0 went into g-r:
https://review.openstack.org/#/c/270417/
But that will just be back once a new keystonemiddleware is released.
IMO this should have been reverted, blacklist 4.1.0, and then deprecated
at least until newton with warnings and release notes saying the
automatic caching is going away so people have time to prepare.
I'm cross-posting to the ops list as an FYI.
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
