[Twisted-Python] i want to update 1000 devices running as servers
I have like 1000 devices that need update that takes like 1 minute each. I want them to update as quick as possible, sending lines from sx file, using UDP and for each line i got reponse "OK". Here is code that works for up to 500 servers, but not so well when i go more (it just does not finish) my current simplified client and server_emulator code is here: https://gist.github.com/3144857 currently i have only one server for simulation, but in real world it will be 1000 different IP-s. I was trying to stop and run reactor but did not work (reactor is not rerunnable exception or somthing like that). I want to run lets say 100 updates at time and once some update is finished i want to remove this protocol out of job and schedule new one. Can somebody give some pointers? I am running this on windows if that is an issue. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] i want to update 1000 devices running as servers
What version of twisted are you using and which reactor is it using? (You can tell from the logs) Note that your process for getting data from the datasource is blocking. What sort of data source is it, and why are you stripping stuff off of it? You might want to consider something more cooperative, such as for example a file producer: http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.FileSender.html I'm assuming you are aware of all of the issues involving UDP (specifically, that in the real world your file is probably not going to make it to the other side) The basic pattern will probably be something like: 1) create 100 deferreds that will fire when you're done transferring 2) gatherResults/DeferredList to wait on all of them 3) do (1) again in a callback to that deferred list until you run out of clients Twisted might have some standard tool somewhere to implement this, but I don't know about it. cheers lvh On 20 Jul 2012, at 09:01, Luka Rahne wrote: > I have like 1000 devices that need update that takes like 1 minute each. I > want them to update as quick as possible, sending lines from sx file, using > UDP and for each line i got reponse "OK". > > Here is code that works for up to 500 servers, but not so well when i go more > (it just does not finish) > > my current simplified client and server_emulator code is here: > https://gist.github.com/3144857 > > > currently i have only one server for simulation, but in real world it will be > 1000 different IP-s. > > I was trying to stop and run reactor but did not work (reactor is not > rerunnable exception or somthing like that). > I want to run lets say 100 updates at time and once some update is finished > i want to remove this protocol out of job and schedule new one. > > Can somebody give some pointers? > > I am running this on windows if that is an issue. > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] AMPBox as an Argument?
Hi,
Apparently AMPBoxes aren't Arguments. However, I kind of want an AMPBox (like
an AMPList, but only one).
Use case: my responses have a "location", but a location is composed of several
sub-things: place name, country and postal code. {"location": {"placeName":
"Krakow", "countryCode": "PL", postalCode: "30-015"}} would be a lot nicer
than having those keys in the top level namespace :)
cheers
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] i want to update 1000 devices running as servers
Due to historical reasons there is UDP only protocol whith response
ack for each packet.
In reality there are also some delays because received device can not
handle data after it sends ack (some ancient and buggy eth chip).
Also there are not just lines from sx file, but are enveloped in other
packet that is transfered.
In general protocol goes like:
for each line in sx:
encode line whit extra data in packet (packet is like len + encrypted_)
Send packet over UDP (tcp not supported)
wait for ACK ("OK" packet in my example)
wait for 0.05 s (not in included example)
I done some code that should support this and i have tried to stop
protocol but calling self.doStop() function
(twisted.internet.protocol.AbstractDatagramProtocol)raises assert
assert self.numPorts > 0
I was hoping that this function will remove protocol instance from reactor.
2012/7/20 Laurens Van Houtven <_...@lvh.cc>
>
> What version of twisted are you using and which reactor is it using? (You can
> tell from the logs)
>
> Note that your process for getting data from the datasource is blocking. What
> sort of data source is it, and why are you stripping stuff off of it? You
> might want to consider something more cooperative, such as for example a file
> producer:
>
> http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.FileSender.html
>
> I'm assuming you are aware of all of the issues involving UDP (specifically,
> that in the real world your file is probably not going to make it to the
> other side)
>
> The basic pattern will probably be something like:
> 1) create 100 deferreds that will fire when you're done transferring
> 2) gatherResults/DeferredList to wait on all of them
> 3) do (1) again in a callback to that deferred list until you run out of
> clients
>
> Twisted might have some standard tool somewhere to implement this, but I
> don't know about it.
>
> cheers
> lvh
>
>
> On 20 Jul 2012, at 09:01, Luka Rahne wrote:
>
> > I have like 1000 devices that need update that takes like 1 minute each. I
> > want them to update as quick as possible, sending lines from sx file, using
> > UDP and for each line i got reponse "OK".
> >
> > Here is code that works for up to 500 servers, but not so well when i go
> > more (it just does not finish)
> >
> > my current simplified client and server_emulator code is here:
> > https://gist.github.com/3144857
> >
> >
> > currently i have only one server for simulation, but in real world it will
> > be 1000 different IP-s.
> >
> > I was trying to stop and run reactor but did not work (reactor is not
> > rerunnable exception or somthing like that).
> > I want to run lets say 100 updates at time and once some update is
> > finished i want to remove this protocol out of job and schedule new one.
> >
> > Can somebody give some pointers?
> >
> > I am running this on windows if that is an issue.
> >
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] AMPBox as an Argument?
On Jul 20, 2012, at 1:11 AM, Laurens Van Houtven <_...@lvh.cc> wrote:
> Hi,
>
>
> Apparently AMPBoxes aren't Arguments. However, I kind of want an AMPBox (like
> an AMPList, but only one).
>
> Use case: my responses have a "location", but a location is composed of
> several sub-things: place name, country and postal code. {"location":
> {"placeName": "Krakow", "countryCode": "PL", postalCode: "30-015"}} would be
> a lot nicer than having those keys in the top level namespace :)
>
> cheers
> lvh
Seems like an easy enough thing to write. Given that AMPList doesn't use a
length prefix (it uses null-key box-termination, just like the rest of the
protocol) the representation would be exactly the same. Just add a trivial
wrapper that uses AMPList, unpacks its argument, and assert that there's only
one of them?
-glyph
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] i want to update 1000 devices running as servers
On 07:01 am, luka.ra...@gmail.com wrote: >I have like 1000 devices that need update that takes like 1 minute >each. I >want them to update as quick as possible, sending lines from sx file, >using >UDP and for each line i got reponse "OK". > >Here is code that works for up to 500 servers, but not so well when i >go >more (it just does not finish) One problem you need to consider is that all platforms limit the number of simultaneous file descriptors (sockets) you may use at once. On Windows, this limit is around 500 by default. You can raise it to a much larger number (but I forget what) by using Twisted's IOCP reactor instead of the default select reactor. See the reactor selection howto on the website for details. >my current simplified client and server_emulator code is here: >https://gist.github.com/3144857 Another problem with this code is the while loop inside sendDatagram. Twisted is a cooperative multitasking system. You cannot execute large loops like that and have everything work properly or well. Programs based on Twisted do a little bit of work, give up execution control, then do a little bit more work when they regain execution control. The work is event-driven, so (for example) perhaps you want to replace the while loop with code that sends the next datagram when the previous datagram is acknowledged. > >currently i have only one server for simulation, but in real world it >will >be 1000 different IP-s. > >I was trying to stop and run reactor but did not work (reactor is not >rerunnable exception or somthing like that). Yes. You can only call reactor.run() once. That shouldn't be a very important limitation. You can do as much networking as you want while only running the reactor once. >I want to run lets say 100 updates at time and once some update >is finished i want to remove this protocol out of job and schedule new >one. Take a look at twisted.internet.task.cooperate. Here's an example demonstrating its use to do a limited amount of work concurrently: http://as.ynchrono.us/2006/05/limiting-parallelism_22.html >Can somebody give some pointers? > >I am running this on windows if that is an issue. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] i want to update 1000 devices running as servers
On 07/20/2012 04:26 AM, Luka Rahne wrote:
> for each line in sx:
> encode line whit extra data in packet (packet is like len + encrypted_)
> Send packet over UDP (tcp not supported)
> wait for ACK ("OK" packet in my example)
> wait for 0.05 s (not in included example)
>
>
> I done some code that should support this and i have tried to stop
> protocol but calling self.doStop() function
> (twisted.internet.protocol.AbstractDatagramProtocol)raises assert
> assert self.numPorts > 0
>
> I was hoping that this function will remove protocol instance from reactor.
>
You can do:
port = reactor.listenUDP()
port.stopListening() # disconnect port
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] i want to update 1000 devices running as servers
My current problem is, that i do not know how to stop protocol and quit listening. What should i call to stop protocol and told reactor to remove from schedule at StopIteration exception? About while 1: loop- that loop is there just to find first non empty line in sx file, in most case it break just in fist condition, since sx file, does not have empty lines. 2012/7/20 : > On 07:01 am, luka.ra...@gmail.com wrote: >>I have like 1000 devices that need update that takes like 1 minute >>each. I >>want them to update as quick as possible, sending lines from sx file, >>using >>UDP and for each line i got reponse "OK". >> >>Here is code that works for up to 500 servers, but not so well when i >>go >>more (it just does not finish) > > One problem you need to consider is that all platforms limit the number > of simultaneous file descriptors (sockets) you may use at once. On > Windows, this limit is around 500 by default. You can raise it to a > much larger number (but I forget what) by using Twisted's IOCP reactor > instead of the default select reactor. > > See the reactor selection howto on the website for details. >>my current simplified client and server_emulator code is here: >>https://gist.github.com/3144857 > > Another problem with this code is the while loop inside sendDatagram. > Twisted is a cooperative multitasking system. You cannot execute large > loops like that and have everything work properly or well. > > Programs based on Twisted do a little bit of work, give up execution > control, then do a little bit more work when they regain execution > control. The work is event-driven, so (for example) perhaps you want to > replace the while loop with code that sends the next datagram when the > previous datagram is acknowledged. >> >>currently i have only one server for simulation, but in real world it >>will >>be 1000 different IP-s. >> >>I was trying to stop and run reactor but did not work (reactor is not >>rerunnable exception or somthing like that). > > Yes. You can only call reactor.run() once. That shouldn't be a very > important limitation. You can do as much networking as you want while > only running the reactor once. >>I want to run lets say 100 updates at time and once some update >>is finished i want to remove this protocol out of job and schedule new >>one. > > Take a look at twisted.internet.task.cooperate. Here's an example > demonstrating its use to do a limited amount of work concurrently: > > http://as.ynchrono.us/2006/05/limiting-parallelism_22.html >>Can somebody give some pointers? >> >>I am running this on windows if that is an issue. > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] i want to update 1000 devices running as servers
Thank you
It was needed to call self.transport.stopListening() and i updated new
client whit working code and now it works much better.
https://gist.github.com/3144857/d0446f8654326c93a5e9def0b6c9172351494219
and old code i was referencing to is here (just for reference)
https://gist.github.com/3144857/5a38e287dec7c480ad34e5073ae1773b95a86662
2012/7/20 Itamar Turner-Trauring :
>
> On 07/20/2012 04:26 AM, Luka Rahne wrote:
>> for each line in sx:
>> encode line whit extra data in packet (packet is like len + encrypted_)
>> Send packet over UDP (tcp not supported)
>> wait for ACK ("OK" packet in my example)
>> wait for 0.05 s (not in included example)
>>
>>
>> I done some code that should support this and i have tried to stop
>> protocol but calling self.doStop() function
>> (twisted.internet.protocol.AbstractDatagramProtocol)raises assert
>> assert self.numPorts > 0
>>
>> I was hoping that this function will remove protocol instance from reactor.
>>
>
> You can do:
>
> port = reactor.listenUDP()
> port.stopListening() # disconnect port
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
