On 15 Gen, 10:22, Laszlo Nagy <gand...@shopzeus.com> wrote:
> > then the speed goes up to 64 messages/sec on windows and 500
> > messages/sec on Linux.
>
> Finally I could reach 1500 messages/sec without using the queue. If I
> comment out one line (use the queue instead of direct write into socket)
> then speed decreases to 40-60 messages/sec. I don't understand why the
> slow version is slower by a factor of 40?
>
> Fast version:
>
> def send_message(self,sender,recipient,msgtype,body,timeout=3600):
>
> self.write_str(self.serializer.serialize([sender,recipient,msgtype,body]))
>
> Slow version:
>
> def send_message(self,sender,recipient,msgtype,body,timeout=3600):
> self.outgoing.put(self.serializer.serialize([
> sender,recipient,msgtype,body
> ]),1,timeout)
>
> plus this method, executed in a different thread:
>
> def _process_outgoing(self):
> try:
> while not self.stop_requested.isSet():
> data_ok = False
> while not self.stop_requested.isSet():
> try:
> data = self.outgoing.get(1,1)
> data_ok = True
> break
> except Queue.Empty:
> pass
> if data_ok:
> self.write_str(data)
> except Exception, e:
> if self.router:
> if not isinstance(e,TransportClosedError):
> self.router.logger.error(dumpexc(e))
> self.router.unregister_endpoint(self)
> self.shutdown()
> raise SystemExit(0)
I would try something like this inside _process_outgoing:
while not self.stop_requested.isSet():
data_ok = False
while not self.stop_requested.isSet():
if not self.outgoing.empty():
try:
data = self.outgoing.get(True,0.1)
data_ok = True
break
except Queue.Empty:
pass
else:
time.sleep(0.1) # maybe, if the thread usess
too much CPU
The hypotesis I made for this suggestion are:
- if the queue is found empty, the queue.get could keep the global
interpreter lock until a message arrive
(blocking also the thread that put the message in the queue)
- if the queue is found empty, the handling of the exception can slows
down the execution
Not sure they are good guesses, because at the rate of your messages
the queue should be almost always full,
and most probably the implementation of Queue.get is smarted than
myself :-). ANyway, it is worth a try ...
Also, is you are using fixed-length queues (the ones that make the
sender wait if the queue is full), try to increase
the queue size, or to use an infinite-size queue.
HTH
Ciao
----
FB
--
http://mail.python.org/mailman/listinfo/python-list
