> On Dec 5, 2016, at 10:18 PM, Amber Hawkie Brown <hawk...@atleastfornow.net> > wrote: > > >> On 6 Dec. 2016, at 17:09, Glyph Lefkowitz <gl...@twistedmatrix.com >> <mailto:gl...@twistedmatrix.com>> wrote: >> >> >> twisted.logger._initialBuffer can consume a surprisingly large amount of >> memory if logging is not initialized >> >> >> The way that `twisted.logger` is supposed to work is that at process startup >> time, the global log observer has a ring buffer for any messages emitted >> before logging is initialized, and emit those messages to the initial set of >> log observers passed to `globalLogBeginner.beginLoggingTo`. >> >> The size of this buffer (in `twisted.logger._buffer._DEFAULT_BUFFER_MAXIMUM` >> is 65535. This value was selected arbitrarily, probably because somebody >> (me or wsanchez) thought "huh, yeah, 64k, that's probably a fine number); >> but of course, that's 64k ''bytes''. >> >> If this were a buffer of actual formatted log messages, of say 200 bytes >> each, that would be about 13 megabytes, which is maybe an acceptable amount >> of RAM to spend on a log buffer. >> >> However, it isn't that. It's a buffer of 64k log ''events'', each of which >> probably has a `log_logger` and `log_source` set, each of which is an object >> attached to potentially arbitrary data. For example, every `Factory` that >> starts up logs something, which means you're holding on to an instance, and >> an instance dictionary, and the protocol instance, and the protocol instance >> dictionary. Worse yet, any logged ''failures'' might hold on to all the >> stuff on their stack. >> >> Add it all up and you end up with a log buffer totaling in the hundreds of >> megabytes, or even gigabytes, once it's full. In an application that >> naively uses Twisted without ever initializing logging, this hangs around >> forever. >> >> This buffer should probably be a ''lot'' smaller, and we might want to emit >> a warning when it fills up, reminding people that it is ''only polite'' to >> start up the logging subsystem, even just to explicitly throw logs away. > > > I fixed trac: https://twistedmatrix.com/trac/ticket/8936#ticket > <https://twistedmatrix.com/trac/ticket/8936#ticket> Thanks!
By "Fixed" I take it you mean rolled back, via: trac@dornkirk:~$ ./virtualenv/bin/pip freeze | grep Trac== Trac==1.0.13 and https://github.com/twisted-infra/braid/commit/c2d393fd501c6464b1b475eff214cab64f13ee2a Don't get me wrong, I'm not complaining :). This is certainly the right thing to do for now. But I'm wondering if you know what's wrong with 1.2 so we can upgrade soonish? -glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
