Re: [Twisted-Python] UDP Logging Server
On Sat, Mar 12, 2011 at 3:32 AM, Tim Allen wrote:
> On Fri, Mar 11, 2011 at 01:15:47PM -0600, SIC FS LIST wrote:
> > So far I have a "working" implementation ... but I'm noticing that if I
> do
> > the following:
> > -- log when a message is received
> > -- that for that message it "might" show up in the file a pretty lengthy
> > period of time later
>
> Assuming the objects stored in DISKINFO[1] etc. are file objects, you
> seem to be writing to the files but never calling flush(). If you don't
> call flush(), Python (well, the C standard library) won't send the data
> on to the OS until its buffer is full, or the file handle is closed. If
> you're not getting that many log lines, it can take a while for that to
> happen.
>
> Of course, if you flush after every disk read, your program will run
> a bit more slowly and with more I/O... for an application where
> reliability is more important than performance (like logging) that's
> probably acceptable.
>
You may also setup a timer that flushes files every, say, 5 minutes.
One other thing I've read in the Python.org site is that flush() is not 100%
sure to work immediately and should be used in combination with os.fsync().
Is there someone that can explain if that is correct?
>
> > The actual UDP protocol:
> >
> > class VocsLogger(DatagramProtocol):
> > def datagramReceived(self, data, (host, port)):
> > _proc_msg(self.transport, data, (host,
> > port))._new_msg().addCallback(handler)
>
> _proc_msg doesn't seem to be complicated enough to need its own class,
> why not just do what _proc_msg does in VocsLogger?
>
> > The _proc_msg class:
> >
> > class _proc_msg:
> > def __init__(self, sck, data, (host, port)):
> > self._sck = sck
> > self._data = data
> > self._host = host
> > self._port = port
> >
> > def _new_msg(self):
> > d, _ = LogMsg().ParseSocketMsg(self._data)
> > if d.type.upper() == DISKINFO[0]:
> > DISKINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> > elif d.type.upper() == LOADAVG[0]:
> > LOADAVG[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> > elif d.type.upper() == MEMINFO[0]:
> > MEMINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> > elif d.type.upper() == NETDEV[0]:
> > NETDEV[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> > elif d.type.upper() == PSAUX[0]:
> > PSAUX[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> > elif d.type.upper() == WHOINFO[0]:
> > WHOINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> > else:
> > DEFAULT[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
>
> It depends on what DISKINFO[0] and DISKINFO[1] actually are, but
> assuming they're a string and a file-handle, this code would look more
> Pythonic as something like this:
>
> LOGSINKS = {
>"DISKINFO": open("/var/log/diskinfo", "w"),
>"LOADAVG": open("/var/log/loadavg", "w"),
>"MEMINFO": open("/var/log/meminfo", "w"),
>"NETDEV": open("/var/log/netdev", "w"),
>"PSAUX": open("/var/log/psaux", "w"),
>"WHOINFO": open("/var/log/whoinfo", "w"),
>"DEFAULT": open("/var/log/default", "w"),
>}
>
> def _new_msg(self, data):
>d, _ = LogMsg().ParseSocketMsg(data)
>type = d.type.upper()
>sink = LOGSINKS.get(type, LOGSINKS['DEFAULT'])
>sink.write("%s\n%s\n" % (d.ToString(), LOG_DELIM))
>
> Hope that helps!
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] UDP Logging Server
Hello, Thanks for the help. Adding a timerservice that calls .flush() on the files every minute (and before rotating the logs) seems to have helped. Appreciate the quick response and feedback. Thanks again! SDR ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] UDP Logging Server
Tim,
Just a quick note to answer your questions:
I have _proc_msg separated out as I'll be adding some additional
functionality on later (adding ACK messages for certain types of log
messages). I do like the way you handled the _new_msg func. I'll probably
do that instead ... but with a couple of if's the handle the cases where I
need to ack the message back.
The [1] index of the log types is the file that is opened by
open('/var/log/...etc...')
Thanks again for your help.
SDR
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] UDP Logging Server
On Sat, Mar 12, 2011 at 02:33:45PM +0200, Pandelis Theodosiou wrote: > On Sat, Mar 12, 2011 at 3:32 AM, Tim Allen wrote: > > Of course, if you flush after every disk read, your program will run > > a bit more slowly and with more I/O... for an application where > > reliability is more important than performance (like logging) that's > > probably acceptable. > > You may also setup a timer that flushes files every, say, 5 minutes. > > One other thing I've read in the Python.org site is that flush() is not 100% > sure to work immediately and should be used in combination with os.fsync(). > Is there someone that can explain if that is correct? Depends what you mean by 'work'. The standard library (Python's or C's) buffers reads and writes because calling into the kernel is expensive. The kernel buffers reads and writes because disk I/O is even more expensive. flush() tells the standard library "send buffered data to the kernel right now" which means your data should survive if your process crashes. fsync() tells the kernel "send buffered data to the disk right now", which means your data should survive if the entire machine crashes. Whether you call nothing, just flush(), or both flush() and fsync() depends on how your software balances performance versus reliability. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Weekly Bug Summary
Bug summary __ Summary for 2011-03-06 through 2011-03-13 Opened Closed Total Change Enhancements: 6 16716-10 Defects: 9 14532 -5 Tasks: 1 1 64 +0 Regressions: 0 0 1 +0 Total:16 31 1313-15 |== Type Changes |== Priority Changes |== Component Changes |Defect: -5 |Highest: +1 |Conch: +0 |Enhancement: -10 |High: -7 |Core: +1 |Task: +0 |Normal: -7 |Ftp: +1 |Low: -1 |Pb:+1 |Lowest: -1 |Vfs: +0 |Web: +0 |Web2: -17 |Web2.Dav: -2 |Words: +1 Total Tickets Open Tickets New / Reopened Bugs __ = Highest = [#4938] SEO Australia - How To Get Benefits In (opened by seoaustralia) taskcore http://twistedmatrix.com/trac/ticket/4938 = High = [#4934] delete vfs (opened by glyph) (CLOSED, fixed) enhancement vfshttp://twistedmatrix.com/trac/ticket/4934 = Normal = [#4925] Conch, SFTP on OSX (opened by rymurr) (CLOSED, invalid) defect conch http://twistedmatrix.com/trac/ticket/4925 [#4926] iocpreactor.c linker errors on Python 3.2 msvc9 (opened by Tourman) defect core http://twistedmatrix.com/trac/ticket/4926 [#4927] twisted.tap.portforward should use string endpoints for listen and destination addresses (opened by exarkun) enhancement core http://twistedmatrix.com/trac/ticket/4927 [#4928] Replace uses of twisted.python.reflect.allYourBase with inspect.getmro (opened by exarkun) enhancement core http://twistedmatrix.com/trac/ticket/4928 [#4929] Add LIST to IRCClient (opened by taos) enhancement words http://twistedmatrix.com/trac/ticket/4929 [#4931] twisted responsible for segmentation fault in embedded systems (opened by alberto) defect core http://twistedmatrix.com/trac/ticket/4931 [#4932] gtk2reactor I/O significantly slower under Windows (opened by detly) defect core http://twistedmatrix.com/trac/ticket/4932 [#4933] doc/core/development/policy/coding-standard.html still talks about dates in copyright headers (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/4933 [#4935] spread.jelly doesn't properly resolve some circular references (opened by speth) defect pb http://twistedmatrix.com/trac/ticket/4935 [#4937] Fix for locale-dependant date formatting in imap and conch (opened by facundobatista) enhancement core http://twistedmatrix.com/trac/ticket/4937 [#3903] Resource routing can't handle empty paths (opened by jpsimons) defect webhttp://twistedmatrix.com/trac/ticket/3903 [#4456] twisted.web.error.Error triggers DeprecationWarning in Python2.6 (opened by olix0r) defect webhttp://twistedmatrix.com/trac/ticket/4456 [#3621] twisted.web should allow getChild to return a Deferred. (opened by radix) enhancement webhttp://twistedmatrix.com/trac/ticket/3621 = Low = [#4930] RNTO without prior calling RNFR should send a bad command sequence response (opened by adiroiban) defect ftphttp://twistedmatrix.com/trac/ticket/4930 Closed Bugs __ = High = [#4934] delete vfs (opened by glyph, closed by exarkun, fixed) enhancement vfshttp://twistedmatrix.com/trac/ticket/4934 [#2129] COPY and MOVE do not copy and move properties (opened by wsanchez, closed by washort, wontfix) enhancement web2.dav http://twistedmatrix.com/trac/ticket/2129 [#2196] Need a generic API for reading and writing resource contents (opened by wsanchez, closed by washort, wontfix) enhancement web2.dav http://twistedmatrix.com/trac/ticket/2196 [#2843] twisted.web2.log.FileAccessLoggingObserver should take a file like object instead of a logPath. (opened by dreid, closed by washort, wontfix) defect web2 http://twistedmatrix.com/trac/ticket/2843 [#2497] FastCGI: infinite loop for packets bigger than 65k (opened by j2a, closed by washort, wontfix) defect web2 http://twistedmatrix.com/trac/ticket/2497 [#2478] Erroneously written resumeProducing and pauseProducing methods in WSGIHandler (opened by dialtone, closed by washort, wontfix) defect
