In article <da2362e0-ec68-467b-b50b-6067057d7...@y36g2000yqh.googlegroups.com>, Miguel P <prosper.spur...@gmail.com> wrote: > I've been working on parsing (tailing) a named pipe which is the > syslog output of the traffic for a rather busy haproxy instance. It's > a fair bit of traffic (upto 3k hits/s per server), but I am finding > that simply tailing the file in python, without any processing, is > taking up 15% of a CPU core. In contrast HAProxy takes 25% and syslogd > takes 5% with the same load. `cat < /named.pipe` takes 0-2% > > Am I just doing things horribly wrong or is this normal? > > Here is my code: > > from collections import deque > import io, sys > > WATCHED_PIPE = '/var/log/haproxy.pipe' > > if __name__ == '__main__': > try: > log_pool = deque([],10000) > fd = io.open(WATCHED_PIPE) > for line in fd: > log_pool.append(line) > except KeyboardInterrupt: > sys.exit() > > Deque appends are O(1) so that's not it. And I am using 2.6's io > module because it's supposed to handle named pipes better. I have > commented the deque appending line and it still takes about the same > CPU.
Be aware that the io module in Python 2.6 is written in Python and was viewed as a prototype. In the current svn trunk, what will be Python 2.7 has a much faster C implementation of the io module backported from Python 3.1. -- Ned Deily, n...@acm.org -- http://mail.python.org/mailman/listinfo/python-list
