Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-15 Thread Tobias Oberstein
> > and the fact that Python uses fopen() from libc, not open() from Posix > > I know you've solved it know, but just to check, you are of course aware of > os.open (which maps directly to Posix open) Yes, thanks, I am aware. However, i.e. Twisted Web uses open(), not os.open(): twisted.python.

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-15 Thread Phil Mayers
On 11/14/2011 05:57 PM, Tobias Oberstein wrote: > and the fact that Python uses fopen() from libc, not open() from Posix I know you've solved it know, but just to check, you are of course aware of os.open (which maps directly to Posix open) ___ Twiste

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Tobias Oberstein
> > Well, this is all absolutely sad. Now we (nearly) have new kqueue, it > > does fly, but I can't break above 32k anyway .. > > Epoll on Linux? what is linux? ;) with the monkey patch to use new IO, I can use the Little Daemon flying on kqueue .. so problem solved. __

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Itamar Turner-Trauring
> Well, this is all absolutely sad. Now we (nearly) have new kqueue, > it does fly, but I can't break above 32k anyway .. Epoll on Linux? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Tobias Oberstein
> > I was told that the new Python 3 IO system does not use fopen(), > > however Twisted is not yet there on Python 3, right? > > If you are wild, you can try https://bitbucket.org/pitrou/t3k/ (at least as an > experiment :-)) I am .. but not thaaat much;) > > > There is a backport of that new

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Antoine Pitrou
On Mon, 14 Nov 2011 09:57:46 -0800 Tobias Oberstein wrote: > Now I'm running out of options. > > I was told that the new Python 3 IO system does not use fopen(), > however Twisted is not yet there on Python 3, right? If you are wild, you can try https://bitbucket.org/pitrou/t3k/ (at least as an

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Tobias Oberstein
> That's definitely not a limit on Python or Twisted. That's an OS limit. > Are you opening lots of files in addition to your sockets? As it turns out, you are right, and the full answer is most unpleasant: This http://www.freebsd.org/cgi/query-pr.cgi?pr=148581&cat= bug in FreeBSD libc, which i

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Tobias Oberstein
> What does resource.getrlimit(resource.RLIMIT_OFILE) return? > > The Python docs say: > > resource.RLIMIT_OFILE > The BSD name for RLIMIT_NOFILE. It's not there .. neither 2.6 nor 2.7 [autobahn@autobahnhub ~/AutobahnHub/service]$ python Python 2.7.1 (r271:86832, Dec 13 2010, 15:52:15) [GCC 4.

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Donal McMullan
What does resource.getrlimit(resource.RLIMIT_OFILE) return? The Python docs say: resource.RLIMIT_OFILE The BSD name for RLIMIT_NOFILE. D On 15 November 2011 03:57, Tobias Oberstein wrote: >> > 2) Too many files. >> > Is there another limit specifically for files, and/or something tunable in

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Tobias Oberstein
> > 2) Too many files. > > Is there another limit specifically for files, and/or something tunable in > Python/Twisted? > > That's definitely not a limit on Python or Twisted. That's an OS limit. > Are you opening lots of files in addition to your sockets? Python open() will bail out at 32768 ope

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Tobias Oberstein
> Besides callLater, a higher level construct to do this is > twisted.internet.task.cooperate(). > https://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2010-cooperative- > multitasking-with-twisted-getting-things-done-concurrently-11-3352182 is a > presentation dreid did, which should really be tur

Re: [Twisted-Python] data dispatch on massive connection counts

2011-11-14 Thread Itamar Turner-Trauring
On Mon, 2011-11-14 at 04:07 -0800, Tobias Oberstein wrote: > I am testing a WebSocket based pubsub system .. I have 2 questions .. any > hints welcome! > > Environment: > + FreeBSD 8.2 i386 > + Python 2.7.2 32-bit > + Twisted Trunk > + new kqueue() reactor > > The FreeBSD is tuned for ma