Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Tobias Oberstein
> Just a final note.. a single no-fds call to select with a 0 timeout seems to > take > around 280ns on my Core 2. Presumably the better interfaces (e.g. epoll, but Are you sure there is actually a context switch happening with this syscall using no FDs and timeout 0? 280ns means your machine c

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Tobias Oberstein
>>I want to trade less precision (timers fire at less exact times) for higher >>efficiency (less context switches). >It's easy enough to write one yourself. This might work: >from twisted.internet.task import Clock, LoopingCall > >clock = Clock() >LoopingCall(lambda: clock.advance(0.001)).start(

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Itamar Turner-Trauring
On 08/10/2014 06:23 PM, Tobias Oberstein wrote: I want to trade less precision (timers fire at less exact times) for higher efficiency (less context switches). It's easy enough to write one yourself. This might work: from twisted.internet.task import Clock, LoopingCall clock = Clock()

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread dw+twisted-python
On Sun, Aug 10, 2014 at 03:31:11PM -0700, Tobias Oberstein wrote: > Thanks a lot for those hints! I will read into this material. Just a final note.. a single no-fds call to select with a 0 timeout seems to take around 280ns on my Core 2. Presumably the better interfaces (e.g. epoll, but not poll

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Tobias Oberstein
> > But the coalescing you describe would also apply to those implicit > > timers (created from select(timeout = ..) within the Linux kernel)? > > It applies to all kernel timers not created by realtime processes. > Alright. I see. > > > This is all fine. But how do I _explicitly_ limit the ra

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Tobias Oberstein
> >What I am after is to explicitly _control_ the maximum syscall rate to > >select() - not simply max. out the box on syscall rate. > > > >Like: limit syscall rate to select() at 1000Hz - regardless how many > >timers I issue per second. > > In other words: > > If you ask Twisted to wake up N ti

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread exarkun
On 09:38 pm, tobias.oberst...@tavendo.de wrote: What I am after is to explicitly _control_ the maximum syscall rate to select() - not simply max. out the box on syscall rate. Like: limit syscall rate to select() at 1000Hz - regardless how many timers I issue per second. In other words: If

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread dw+twisted-python
On Sun, Aug 10, 2014 at 02:51:02PM -0700, Tobias Oberstein wrote: > But the coalescing you describe would also apply to those implicit > timers (created from select(timeout = ..) within the Linux kernel)? It applies to all kernel timers not created by realtime processes. > This is all fine. But

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Tobias Oberstein
> Individual OS have their own mechanisms for avoiding the kind of waste you're > describing. For example, Linux quite aggressively rounds up the expiry of > certain classes of timer at progressively less granular intervals the further > in > the future they're scheduled ("timer coalescing"). As

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Tobias Oberstein
>There is only one select() call (or whatever) at any given time, regardless of >how many timers. Yes, I do understand this. >Syscalls are thus O(1). Timers are stored in sorted order. When event loop >wakes up it removes timers that have been reached, which is fast because >they're sorted so

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread dw+twisted-python
Hey Tobias, Individual OS have their own mechanisms for avoiding the kind of waste you're describing. For example, Linux quite aggressively rounds up the expiry of certain classes of timer at progressively less granular intervals the further in the future they're scheduled ("timer coalescing"). W

Re: [Twisted-Python] Scalability of timers

2014-08-10 Thread Itamar Turner-Trauring
There is only one select() call (or whatever) at any given time, regardless of how many timers. Syscalls are thus O(1). Timers are stored in sorted order. When event loop wakes up it removes timers that have been reached, which is fast because they're sorted so when you hit one that is still in

[Twisted-Python] Scalability of timers

2014-08-10 Thread Tobias Oberstein
Hi, I have a question regarding scalability of timers in Twisted. Say I have a massive number of periodic timers (lets say each with period 1s, but all slightly time shifted to each other). As far as I understand, timers are implemented ultimately by setting the timeout parameter when calling