Re: sched_yield() makes OpenLDAP slow

2005-08-23 Thread Lincoln Dale
Denis Vlasenko wrote: This is what I would expect if run on an otherwise idle machine. sched_yield just puts you at the back of the line for runnable processes, it doesn't magically cause you to go to sleep somehow. When a kernel build is occurring??? Plus `top` itself It damn well s

Re: sched_yield() makes OpenLDAP slow

2005-08-23 Thread Denis Vlasenko
On Tuesday 23 August 2005 14:17, linux-os \(Dick Johnson\) wrote: > > On Mon, 22 Aug 2005, Robert Hancock wrote: > > > linux-os (Dick Johnson) wrote: > >> I reported thet sched_yield() wasn't working (at least as expected) > >> back in March of 2004. > >> > >>for(;;) > >>

Re: sched_yield() makes OpenLDAP slow

2005-08-23 Thread linux-os \(Dick Johnson\)
On Mon, 22 Aug 2005, Robert Hancock wrote: > linux-os (Dick Johnson) wrote: >> I reported thet sched_yield() wasn't working (at least as expected) >> back in March of 2004. >> >> for(;;) >> sched_yield(); >> >> ... takes 100% CPU time as reported by `top`. It sho

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread Howard Chu
Florian Weimer wrote: * Howard Chu: > That's not the complete story. BerkeleyDB provides a > db_env_set_func_yield() hook to tell it what yield function it > should use when its internal locking routines need such a function. > If you don't set a specific hook, it just uses sleep(). The > OpenLD

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread Howard Chu
Florian Weimer wrote: * Andi Kleen: Has anybody contacted the Sleepycat people with a description of the problem yet? Berkeley DB does not call sched_yield, but OpenLDAP does in some wrapper code around the Berkeley DB backend. That's not the complete story. BerkeleyDB provides a db_

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread Howard Chu
Nikita Danilov wrote: Howard Chu writes: > That's beside the point. Folks are making an assertion that > sched_yield() is meaningless; this example demonstrates that there are > cases where sched_yield() is essential. It is not essential, it is non-portable. Code you described is based on n

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread linux-os \(Dick Johnson\)
On Sat, 20 Aug 2005, Robert Hancock wrote: > Howard Chu wrote: >> I'll note that we removed a number of the yield calls (that were in >> OpenLDAP 2.2) for the 2.3 release, because I found that they were >> redundant and causing unnecessary delays. My own test system is running >> on a Linux 2.6.1

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread Florian Weimer
* Howard Chu: >>> Has anybody contacted the Sleepycat people with a description of >>> the problem yet? >> Berkeley DB does not call sched_yield, but OpenLDAP does in some >> wrapper code around the Berkeley DB backend. > That's not the complete story. BerkeleyDB provides a > db_env_set_func_yi

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread Andi Kleen
> processes (PTHREAD_SCOPE_SYSTEM). The previous comment about slapd only > needing to yield within a single process is inaccurate; since we allow > slapcat to run concurrently with slapd (to allow hot backups) we need > BerkeleyDB's locking/yield functions to work in System scope. That's broke

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread Robert Hancock
linux-os (Dick Johnson) wrote: I reported thet sched_yield() wasn't working (at least as expected) back in March of 2004. for(;;) sched_yield(); ... takes 100% CPU time as reported by `top`. It should take practically 0. Somebody said that this was because `

Re: sched_yield() makes OpenLDAP slow

2005-08-22 Thread Howard Chu
Andi Kleen wrote: > processes (PTHREAD_SCOPE_SYSTEM). The previous comment about slapd > only needing to yield within a single process is inaccurate; since > we allow slapcat to run concurrently with slapd (to allow hot > backups) we need BerkeleyDB's locking/yield functions to work in > System s

Re: sched_yield() makes OpenLDAP slow

2005-08-21 Thread Florian Weimer
* Andi Kleen: > Has anybody contacted the Sleepycat people with a description of the > problem yet? Berkeley DB does not call sched_yield, but OpenLDAP does in some wrapper code around the Berkeley DB backend. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body

Re: sched_yield() makes OpenLDAP slow

2005-08-21 Thread Nikita Danilov
Howard Chu writes: > Lee Revell wrote: > > On Sat, 2005-08-20 at 11:38 -0700, Howard Chu wrote: > > > But I also found that I needed to add a new yield(), to work around > > > yet another unexpected issue on this system - we have a number of > > > threads waiting on a condition variable, and

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Robert Hancock
Howard Chu wrote: I'll note that we removed a number of the yield calls (that were in OpenLDAP 2.2) for the 2.3 release, because I found that they were redundant and causing unnecessary delays. My own test system is running on a Linux 2.6.12.3 kernel (installed over a SuSE 9.2 x86_64 distro),

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Nick Piggin
Howard Chu wrote: Lee Revell wrote: On Sat, 2005-08-20 at 11:38 -0700, Howard Chu wrote: > But I also found that I needed to add a new yield(), to work around > yet another unexpected issue on this system - we have a number of > threads waiting on a condition variable, and the thread holding t

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Nikita Danilov
Howard Chu writes: > Nikita Danilov wrote: > > That returns us to the core of the problem: sched_yield() is used to > > implement a synchronization primitive and non-portable assumptions are > > made about its behavior: SUS defines that after sched_yield() thread > > ceases to run on the CPU "

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Lee Revell
On Sat, 2005-08-20 at 11:38 -0700, Howard Chu wrote: > Nick Piggin wrote: > > Robert Hancock wrote: > > > I fail to see how sched_yield is going to be very helpful in this > > > situation. Since that call can sleep from a range of time ranging > > > from zero to a long time, it's going to give unp

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Howard Chu
Lee Revell wrote: On Sat, 2005-08-20 at 11:38 -0700, Howard Chu wrote: > But I also found that I needed to add a new yield(), to work around > yet another unexpected issue on this system - we have a number of > threads waiting on a condition variable, and the thread holding the > mutex signals t

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Lee Revell
On Sat, 2005-08-20 at 11:38 -0700, Howard Chu wrote: > But I also found that I needed to add a new > yield(), to work around yet another unexpected issue on this system - > we have a number of threads waiting on a condition variable, and the > thread holding the mutex signals the var, unlocks the

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Howard Chu
Nikita Danilov wrote: That returns us to the core of the problem: sched_yield() is used to implement a synchronization primitive and non-portable assumptions are made about its behavior: SUS defines that after sched_yield() thread ceases to run on the CPU "until it again becomes the head of its t

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Howard Chu
Nick Piggin wrote: Robert Hancock wrote: > I fail to see how sched_yield is going to be very helpful in this > situation. Since that call can sleep from a range of time ranging > from zero to a long time, it's going to give unpredictable results. Well, not sleep technically, but yield the CP

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Andi Kleen
Howard Chu <[EMAIL PROTECTED]> writes: > In this specific example, we use whatever > BerkeleyDB provides and we're certainly not about to write our own > transactional embedded database engine just for this. BerkeleyDB is free software after all that comes with source code. Surely it can be fixe

Re: sched_yield() makes OpenLDAP slow

2005-08-20 Thread Nikita Danilov
Howard Chu writes: > Nikita Danilov wrote: [...] > > > What prevents transaction monitor from using, say, condition > > variables to "yield cpu"? That would have an additional advantage of > > blocking thread precisely until specific event occurs, instead of > > blocking for some vague

Re: sched_yield() makes OpenLDAP slow

2005-08-19 Thread Nick Piggin
Robert Hancock wrote: I fail to see how sched_yield is going to be very helpful in this situation. Since that call can sleep from a range of time ranging from zero to a long time, it's going to give unpredictable results. Well, not sleep technically, but yield the CPU for some undefined a

Re: sched_yield() makes OpenLDAP slow

2005-08-19 Thread Robert Hancock
Howard Chu wrote: You assume that spinlocks are the only reason a developer may want to yield the processor. This assumption is unfounded. Case in point - the primary backend in OpenLDAP uses a transactional database with page-level locking of its data structures to provide high levels of conc

Re: sched_yield() makes OpenLDAP slow

2005-08-19 Thread Howard Chu
Nikita Danilov wrote: Howard Chu <[EMAIL PROTECTED]> writes: > concurrency. It is the nature of such a system to encounter > deadlocks over the normal course of operations. When a deadlock is > detected, some thread must be chosen (by one of a variety of > algorithms) to abort its transaction, i

Re: sched_yield() makes OpenLDAP slow

2005-08-19 Thread Howard Chu
Chris Wedgwood wrote: On Thu, Aug 18, 2005 at 11:03:45PM -0700, Howard Chu wrote: > If the 2.6 kernel makes this programming model unreasonably slow, > then quite simply this kernel is not viable as a database platform. Pretty much everyone else manages to make it work. And this contribute

Re: sched_yield() makes OpenLDAP slow

2005-08-19 Thread Nikita Danilov
Howard Chu <[EMAIL PROTECTED]> writes: [...] > concurrency. It is the nature of such a system to encounter deadlocks > over the normal course of operations. When a deadlock is detected, some > thread must be chosen (by one of a variety of algorithms) to abort its > transaction, in order to allow

Re: sched_yield() makes OpenLDAP slow

2005-08-19 Thread Chris Wedgwood
On Thu, Aug 18, 2005 at 11:03:45PM -0700, Howard Chu wrote: > If the 2.6 kernel makes this programming model unreasonably slow, > then quite simply this kernel is not viable as a database platform. Pretty much everyone else manages to make it work. - To unsubscribe from this list: send the line "

Re: sched_yield() makes OpenLDAP slow

2005-08-18 Thread Nick Piggin
Hi Howard, Thanks for joining the discussion. One request, if I may, can you retain the CC list on posts please? Howard Chu wrote: > AFAIKS, sched_yield should only really be used by realtime applications that know exactly what they're doing. pthread_yield() was deleted from the POSIX thre

Re: sched_yield() makes OpenLDAP slow

2005-08-18 Thread Bernardo Innocenti
Andi Kleen wrote: > Bernardo Innocenti <[EMAIL PROTECTED]> writes: > > It's really more a feature than a bug that it breaks so easily > because they should be really using futexes instead, which > have much better behaviour than any sched_yield ever could > (they will directly wake up another proc

Re: sched_yield() makes OpenLDAP slow

2005-08-18 Thread Andi Kleen
Bernardo Innocenti <[EMAIL PROTECTED]> writes: It's really more a feature than a bug that it breaks so easily because they should be really using futexes instead, which have much better behaviour than any sched_yield ever could (they will directly wake up another process waiting for the lock and a

Re: sched_yield() makes OpenLDAP slow

2005-08-18 Thread Bernardo Innocenti
Nick Piggin wrote: > We class the SCHED_OTHER policy as having a single priority, which > I believe is allowed (and even makes good sense, because dynamic > and even nice priorities aren't really well defined). > > That also makes our sched_yield() behaviour correct. > > AFAIKS, sched_yield shou

Re: sched_yield() makes OpenLDAP slow

2005-08-18 Thread Maciej Soltysiak
Hello Con, Thursday, August 18, 2005, 2:47:25 AM, you wrote: > sched_yield behaviour changed in 2.5 series more than 3 years ago and > applications that use this as a locking primitive should be updated. I remember open office had a problem with excessive use of sched_yield() during 2.5. I guess t

Re: sched_yield() makes OpenLDAP slow

2005-08-17 Thread Nick Piggin
Joseph Fannin wrote: On Thu, Aug 18, 2005 at 02:50:16AM +0200, Bernardo Innocenti wrote: The relative timestamp reveals that slapd is spending 50ms after yielding. Meanwhile, GCC is probably being scheduled for a whole quantum. Reading the man-page of sched_yield() it seems this isn't the c

Re: sched_yield() makes OpenLDAP slow

2005-08-17 Thread Bernardo Innocenti
Joseph Fannin wrote: >The behavior of sched_yield changed for 2.6. I suppose the man > page didn't get updated. Now I remember reading about that on LWN or maybe KernelTraffic. Thanks! >>I also think OpenLDAP is wrong. First, it should be calling >>pthread_yield() because slapd is a multit

Re: sched_yield() makes OpenLDAP slow

2005-08-17 Thread Joseph Fannin
On Thu, Aug 18, 2005 at 02:50:16AM +0200, Bernardo Innocenti wrote: > The relative timestamp reveals that slapd is spending 50ms > after yielding. Meanwhile, GCC is probably being scheduled > for a whole quantum. > > Reading the man-page of sched_yield() it seems this isn't > the correct behavior

Re: sched_yield() makes OpenLDAP slow

2005-08-17 Thread Con Kolivas
On Thu, 18 Aug 2005 10:50 am, Bernardo Innocenti wrote: > Hello, > > I've been investigating a performance problem on a > server using OpenLDAP 2.2.26 for nss resolution and > running kernel 2.6.12. > > When a CPU bound process such as GCC is running in the > background (even at nice 10), many triv

sched_yield() makes OpenLDAP slow

2005-08-17 Thread Bernardo Innocenti
Hello, I've been investigating a performance problem on a server using OpenLDAP 2.2.26 for nss resolution and running kernel 2.6.12. When a CPU bound process such as GCC is running in the background (even at nice 10), many trivial commands such as "su" or "groups" become extremely slow and take a