Re: [bisected] pre-3.16 regression on open() scalability

2014-06-20 Thread Paul E. McKenney
On Fri, Jun 20, 2014 at 10:39:50AM -0700, Dave Hansen wrote: > On 06/20/2014 09:30 AM, Paul E. McKenney wrote: > > On Fri, Jun 20, 2014 at 11:07:13AM -0500, Christoph Lameter wrote: > >> On Fri, 20 Jun 2014, Paul E. McKenney wrote: > >> > >>> And the fraction of the small systems that care that muc

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-20 Thread Dave Hansen
On 06/20/2014 09:30 AM, Paul E. McKenney wrote: > On Fri, Jun 20, 2014 at 11:07:13AM -0500, Christoph Lameter wrote: >> On Fri, 20 Jun 2014, Paul E. McKenney wrote: >> >>> And the fraction of the small systems that care that much about latency >>> is also quite small. >> >> This also impacts genera

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-20 Thread Paul E. McKenney
On Fri, Jun 20, 2014 at 11:07:13AM -0500, Christoph Lameter wrote: > On Fri, 20 Jun 2014, Paul E. McKenney wrote: > > > And the fraction of the small systems that care that much about latency > > is also quite small. > > This also impacts general performance of course not only latency. > > > Of

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-20 Thread Christoph Lameter
On Fri, 20 Jun 2014, Paul E. McKenney wrote: > And the fraction of the small systems that care that much about latency > is also quite small. This also impacts general performance of course not only latency. > Of course, if you have performance results that show otherwise, please > share them.

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-20 Thread Paul E. McKenney
On Fri, Jun 20, 2014 at 10:20:36AM -0500, Christoph Lameter wrote: > On Thu, 19 Jun 2014, Paul E. McKenney wrote: > > > I am putting together patches based on Eric Dumazet's suggestion and on a > > variant of the above approach. However, I do expect the distros to starve > > to death between thos

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-20 Thread Christoph Lameter
On Thu, 19 Jun 2014, Paul E. McKenney wrote: > I am putting together patches based on Eric Dumazet's suggestion and on a > variant of the above approach. However, I do expect the distros to starve > to death between those two bales of hay. But perhaps the actual patches > will inspire a bit of l

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Paul E. McKenney
On Thu, Jun 19, 2014 at 02:32:03PM -0700, j...@joshtriplett.org wrote: > On Thu, Jun 19, 2014 at 04:16:34PM -0500, Christoph Lameter wrote: > > This looks very much like the CONFIG_PREEMPT problem in not so > > extreme form. Maybe we need to add another config option: > > > > CONFIG_REALLY_REALLY_

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread josh
On Thu, Jun 19, 2014 at 04:16:34PM -0500, Christoph Lameter wrote: > This looks very much like the CONFIG_PREEMPT problem in not so > extreme form. Maybe we need to add another config option: > > CONFIG_REALLY_REALLY_NO_PREEMPT > > to get the fastest code possible and those cond_rescheds removed

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Christoph Lameter
This looks very much like the CONFIG_PREEMPT problem in not so extreme form. Maybe we need to add another config option: CONFIG_REALLY_REALLY_NO_PREEMPT to get the fastest code possible and those cond_rescheds removed from the critical paths? Or better CONFIG_PREEMPT_HALF_WAY to enable those c

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Christoph Lameter
On Thu, 19 Jun 2014, Paul E. McKenney wrote: > On Thu, Jun 19, 2014 at 03:31:56PM -0500, Christoph Lameter wrote: > > On Thu, 19 Jun 2014, Paul E. McKenney wrote: > > > > > That is a separate issue, but unnecessary calls to cond_resched() > > > should of course be removed -- no argument there. > >

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Paul E. McKenney
On Thu, Jun 19, 2014 at 01:50:44PM -0700, Andi Kleen wrote: > On Thu, Jun 19, 2014 at 01:42:20PM -0700, Paul E. McKenney wrote: > > On Thu, Jun 19, 2014 at 03:31:56PM -0500, Christoph Lameter wrote: > > > On Thu, 19 Jun 2014, Paul E. McKenney wrote: > > > > > > > That is a separate issue, but unne

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Andi Kleen
On Thu, Jun 19, 2014 at 01:42:20PM -0700, Paul E. McKenney wrote: > On Thu, Jun 19, 2014 at 03:31:56PM -0500, Christoph Lameter wrote: > > On Thu, 19 Jun 2014, Paul E. McKenney wrote: > > > > > That is a separate issue, but unnecessary calls to cond_resched() > > > should of course be removed -- n

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Paul E. McKenney
On Thu, Jun 19, 2014 at 03:31:56PM -0500, Christoph Lameter wrote: > On Thu, 19 Jun 2014, Paul E. McKenney wrote: > > > That is a separate issue, but unnecessary calls to cond_resched() > > should of course be removed -- no argument there. > > It looks like we are fighting higher latencies by add

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Christoph Lameter
On Thu, 19 Jun 2014, Paul E. McKenney wrote: > That is a separate issue, but unnecessary calls to cond_resched() > should of course be removed -- no argument there. It looks like we are fighting higher latencies by adding calls in most of the critical sections which will in turn increase the late

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Paul E. McKenney
On Thu, Jun 19, 2014 at 07:24:18AM +0200, Mike Galbraith wrote: > On Wed, 2014-06-18 at 21:19 -0700, Paul E. McKenney wrote: > > On Wed, Jun 18, 2014 at 08:38:16PM -0700, Andi Kleen wrote: > > > On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > > > > On Wed, Jun 18, 2014 at 06:42

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Paul E. McKenney
On Thu, Jun 19, 2014 at 09:42:18AM -0500, Christoph Lameter wrote: > On Wed, 18 Jun 2014, Andi Kleen wrote: > > > I still think it's totally the wrong direction to pollute so > > many fast paths with this obscure debugging check workaround > > unconditionally. > > > > cond_resched() is in EVERY sl

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-19 Thread Christoph Lameter
On Wed, 18 Jun 2014, Andi Kleen wrote: > I still think it's totally the wrong direction to pollute so > many fast paths with this obscure debugging check workaround > unconditionally. > > cond_resched() is in EVERY sleeping lock and in EVERY memory allocation! > And these are really critical paths

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Mike Galbraith
On Wed, 2014-06-18 at 21:19 -0700, Paul E. McKenney wrote: > On Wed, Jun 18, 2014 at 08:38:16PM -0700, Andi Kleen wrote: > > On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > > > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > > > > > I still think it's total

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 09:52:25PM -0700, Eric Dumazet wrote: > On Wed, 2014-06-18 at 20:38 -0700, Andi Kleen wrote: > > On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > > > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > > > > > I still think it's totally th

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Eric Dumazet
On Wed, 2014-06-18 at 20:38 -0700, Andi Kleen wrote: > On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > > > I still think it's totally the wrong direction to pollute so > > > many fast paths with this obscure

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Thu, Jun 19, 2014 at 04:50:19AM +0200, Mike Galbraith wrote: > On Wed, 2014-06-18 at 19:13 -0700, Paul E. McKenney wrote: > > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > > > I still think it's totally the wrong direction to pollute so > > > many fast paths with this ob

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 08:38:16PM -0700, Andi Kleen wrote: > On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > > > I still think it's totally the wrong direction to pollute so > > > many fast paths with this

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Andi Kleen
On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > I still think it's totally the wrong direction to pollute so > > many fast paths with this obscure debugging check workaround > > unconditionally. > > OOM preve

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Mike Galbraith
On Wed, 2014-06-18 at 19:13 -0700, Paul E. McKenney wrote: > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > I still think it's totally the wrong direction to pollute so > > many fast paths with this obscure debugging check workaround > > unconditionally. > > OOM prevention

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 07:13:37PM -0700, Paul E. McKenney wrote: > On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > > > I still think it's totally the wrong direction to pollute so > > many fast paths with this obscure debugging check workaround > > unconditionally. > > OOM preve

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 06:42:00PM -0700, Andi Kleen wrote: > > I still think it's totally the wrong direction to pollute so > many fast paths with this obscure debugging check workaround > unconditionally. OOM prevention should count for something, I would hope. > cond_resched() is in EVERY sl

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Andi Kleen
I still think it's totally the wrong direction to pollute so many fast paths with this obscure debugging check workaround unconditionally. cond_resched() is in EVERY sleeping lock and in EVERY memory allocation! And these are really critical paths for many workloads. If you really wanted to do

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 01:30:52PM -0700, Paul E. McKenney wrote: > On Wed, Jun 18, 2014 at 10:36:25AM -0700, Dave Hansen wrote: > > On 06/18/2014 05:58 AM, Paul E. McKenney wrote: > > >> > This is the previous kernel, plus RCU tracing, so it's not 100% > > >> > apples-to-apples (and it peaks a bit

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 03:03:38PM -0700, Dave Hansen wrote: > On 06/18/2014 02:48 PM, Paul E. McKenney wrote: > > On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dave Hansen wrote: > >> Hi Paul, > >> > >> I'm seeing a regression when comparing 3.15 to Linus's current tree. > >> I'm using Anton Blanchard

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Dave Hansen
On 06/18/2014 02:48 PM, Paul E. McKenney wrote: > On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dave Hansen wrote: >> Hi Paul, >> >> I'm seeing a regression when comparing 3.15 to Linus's current tree. >> I'm using Anton Blanchard's will-it-scale "open1" test which creates a >> bunch of processes and d

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dave Hansen wrote: > Hi Paul, > > I'm seeing a regression when comparing 3.15 to Linus's current tree. > I'm using Anton Blanchard's will-it-scale "open1" test which creates a > bunch of processes and does open()/close() in a tight loop: > > > https://git

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 10:36:25AM -0700, Dave Hansen wrote: > On 06/18/2014 05:58 AM, Paul E. McKenney wrote: > >> > This is the previous kernel, plus RCU tracing, so it's not 100% > >> > apples-to-apples (and it peaks a bit lower than the other kernel). But > >> > here's the will-it-scale open1

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Dave Hansen
On 06/18/2014 05:58 AM, Paul E. McKenney wrote: >> > This is the previous kernel, plus RCU tracing, so it's not 100% >> > apples-to-apples (and it peaks a bit lower than the other kernel). But >> > here's the will-it-scale open1 throughput on the y axis vs >> > RCU_COND_RESCHED_EVERY_THIS_JIFFIES

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Christoph Lameter
On Tue, 17 Jun 2014, Andi Kleen wrote: > I still think it's totally the wrong place. cond_resched() is in so > many fast paths (every lock, every allocation). It just doesn't > make sense to add non essential things like this to it. > > I would be rather to just revert the original patch. I fully

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Tue, Jun 17, 2014 at 11:33:56PM -0700, Dave Hansen wrote: > On 06/17/2014 05:18 PM, Paul E. McKenney wrote: > > So if I understand correctly, a goodly part of the regression is due not > > to the overhead added to cond_resched(), but rather because grace periods > > are now happening faster, thu

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Paul E. McKenney
On Wed, Jun 18, 2014 at 05:40:28AM -0700, Andi Kleen wrote: > On Tue, Jun 17, 2014 at 09:47:45PM -0700, Paul E. McKenney wrote: > > On Tue, Jun 17, 2014 at 07:27:31PM -0700, Andi Kleen wrote: > > > > OK. What would you suggest instead? If all we do is to revert the > > > > > > Hang checker shoul

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-18 Thread Andi Kleen
On Tue, Jun 17, 2014 at 09:47:45PM -0700, Paul E. McKenney wrote: > On Tue, Jun 17, 2014 at 07:27:31PM -0700, Andi Kleen wrote: > > > OK. What would you suggest instead? If all we do is to revert the > > > > Hang checker should have two timer phases: > > > > Timer fires first time: > > - Save c

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Dave Hansen
On 06/17/2014 05:18 PM, Paul E. McKenney wrote: > So if I understand correctly, a goodly part of the regression is due not > to the overhead added to cond_resched(), but rather because grace periods > are now happening faster, thus incurring more overhead. Is that correct? Yes, that's the theory

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Paul E. McKenney
On Tue, Jun 17, 2014 at 07:27:31PM -0700, Andi Kleen wrote: > > OK. What would you suggest instead? If all we do is to revert the > > Hang checker should have two timer phases: > > Timer fires first time: > - Save context switch counter on that. Force a reschedule to some > work queue. Rearm ti

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Andi Kleen
> OK. What would you suggest instead? If all we do is to revert the Hang checker should have two timer phases: Timer fires first time: - Save context switch counter on that. Force a reschedule to some work queue. Rearm timer Timer fires again: - Check reschedule count. If the reschedule count

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Paul E. McKenney
On Tue, Jun 17, 2014 at 05:15:17PM -0700, Andi Kleen wrote: > > It also ends up eating a new cacheline in a bunch of pretty hot paths. > > It would be nice to be able to keep the fast path part of this as at > > least read-only. > > > > Could we do something (functionally) like the attached patch?

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Paul E. McKenney
On Tue, Jun 17, 2014 at 04:10:29PM -0700, Dave Hansen wrote: > On 06/13/2014 03:45 PM, Paul E. McKenney wrote: > >> > Could the additional RCU quiescent states be causing us to be doing more > >> > RCU frees that we were before, and getting less benefit from the lock > >> > batching that RCU normal

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Andi Kleen
> It also ends up eating a new cacheline in a bunch of pretty hot paths. > It would be nice to be able to keep the fast path part of this as at > least read-only. > > Could we do something (functionally) like the attached patch? Instead > of counting cond_resched() calls, we could just specify so

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Josh Triplett
On Tue, Jun 17, 2014 at 04:10:29PM -0700, Dave Hansen wrote: > On 06/13/2014 03:45 PM, Paul E. McKenney wrote: > >> > Could the additional RCU quiescent states be causing us to be doing more > >> > RCU frees that we were before, and getting less benefit from the lock > >> > batching that RCU normal

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-17 Thread Dave Hansen
On 06/13/2014 03:45 PM, Paul E. McKenney wrote: >> > Could the additional RCU quiescent states be causing us to be doing more >> > RCU frees that we were before, and getting less benefit from the lock >> > batching that RCU normally provides? > Quite possibly. One way to check would be to use the

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-13 Thread Paul E. McKenney
On Fri, Jun 13, 2014 at 04:35:01PM -0700, Dave Hansen wrote: > On 06/13/2014 03:45 PM, Paul E. McKenney wrote: > > On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dav > >> So, I bisected it down to this: > >> > >>> commit ac1bea85781e9004da9b3e8a4b097c18492d857c > >>> Author: Paul E. McKenney > >>> Date

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-13 Thread Dave Hansen
On 06/13/2014 03:45 PM, Paul E. McKenney wrote: > On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dav >> So, I bisected it down to this: >> >>> commit ac1bea85781e9004da9b3e8a4b097c18492d857c >>> Author: Paul E. McKenney >>> Date: Sun Mar 16 21:36:25 2014 -0700 >>> >>> sched,rcu: Make cond_resched

Re: [bisected] pre-3.16 regression on open() scalability

2014-06-13 Thread Paul E. McKenney
On Fri, Jun 13, 2014 at 01:04:28PM -0700, Dave Hansen wrote: > Hi Paul, > > I'm seeing a regression when comparing 3.15 to Linus's current tree. > I'm using Anton Blanchard's will-it-scale "open1" test which creates a > bunch of processes and does open()/close() in a tight loop: > > > https://git

[bisected] pre-3.16 regression on open() scalability

2014-06-13 Thread Dave Hansen
Hi Paul, I'm seeing a regression when comparing 3.15 to Linus's current tree. I'm using Anton Blanchard's will-it-scale "open1" test which creates a bunch of processes and does open()/close() in a tight loop: > https://github.com/antonblanchard/will-it-scale/blob/master/tests/open1.c At about 50