Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-11 Thread Tobias Oberstein
> Now, as I understand it, sendfile() will perform zero-copy IO; since the > contents > of the file will undoubtedly be in the page cache, it should in theory DMA the > data straight from the (single copy of the) data in RAM to the NIC buffers. > > It should also handle refcounting for you - you

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-11 Thread Tobias Oberstein
> For what it's worth, real IP multicast is quite commonly used for distributing > short messages to many clients in realtime in some closed networks, such as > financial trading systems. With good network equipment that can handle low- > or zero-loss timely delivery, it does work very well. Empha

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-11 Thread Tobias Oberstein
> Not to mention the fact that inevitably, you probably are going to want some > security on those connections, which means TLS, which means individual crypto > connections. If there is a need for encryption, then yes, .. but it's not always needed > > I believe the best model for this kind of h

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-11 Thread Phil Mayers
On 02/10/2012 08:20 PM, Tobias Oberstein wrote: > >> store the socket buffer as a (fairly complex) linked list of >> reference-counted >> blocks, and use scatter-gather IO to the network card. > > Doesn't a (modern) kernel do something like that for virtual memory pages ie.? Possibly. My knowledg

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-11 Thread Phil Mayers
On 02/10/2012 09:54 PM, Glyph Lefkowitz wrote: > I believe the best model for this kind of high-volume > reliable-multicast-over-unicast is a spanning tree, like what IRC For what it's worth, real IP multicast is quite commonly used for distributing short messages to many clients in realtime in

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-10 Thread Glyph Lefkowitz
On Feb 10, 2012, at 12:49 PM, Phil Mayers wrote: > On 10/02/12 16:56, Tobias Oberstein wrote: >> Hi there, >> >> what is the most efficient/performant way of doing the following? >> >> I have a short message prepared .. say a string of 100 octets. >> I want to push out that _same_ string on 100

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-10 Thread Tobias Oberstein
> > what is the most efficient/performant way of doing the following? > > > > I have a short message prepared .. say a string of 100 octets. > > I want to push out that _same_ string on 100k connected TCPs (on a server). > > > > == > > > > My thinking was: ideally, the 100 bytes would be transferre

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-10 Thread Phil Mayers
On 10/02/12 16:56, Tobias Oberstein wrote: > Hi there, > > what is the most efficient/performant way of doing the following? > > I have a short message prepared .. say a string of 100 octets. > I want to push out that _same_ string on 100k connected TCPs (on a server). > > == > > My thinking was: i

Re: [Twisted-Python] pushing out same message on 100k TCPs

2012-02-10 Thread Corbin Simpson
AFAIK that's not possible in TCP, only in multicast, and the kernel will make copies of that string no matter what, but I am highly unknowledgeable in this area. On 2/10/12, Tobias Oberstein wrote: > Hi there, > > what is the most efficient/performant way of doing the following? > > I have a shor

[Twisted-Python] pushing out same message on 100k TCPs

2012-02-10 Thread Tobias Oberstein
Hi there, what is the most efficient/performant way of doing the following? I have a short message prepared .. say a string of 100 octets. I want to push out that _same_ string on 100k connected TCPs (on a server). == My thinking was: ideally, the 100 bytes would be transferred to kernel/NIC sp