> On 6 Dec. 2016, at 17:09, Glyph Lefkowitz <gl...@twistedmatrix.com> wrote:
>
>
> twisted.logger._initialBuffer can consume a surprisingly large amount of
> memory if logging is not initialized
>
>
> The way that `twisted.logger` is supposed to work is that at process startup
> time, the global log observer has a ring buffer for any messages emitted
> before logging is initialized, and emit those messages to the initial set of
> log observers passed to `globalLogBeginner.beginLoggingTo`.
>
> The size of this buffer (in `twisted.logger._buffer._DEFAULT_BUFFER_MAXIMUM`
> is 65535. This value was selected arbitrarily, probably because somebody (me
> or wsanchez) thought "huh, yeah, 64k, that's probably a fine number); but of
> course, that's 64k ''bytes''.
>
> If this were a buffer of actual formatted log messages, of say 200 bytes
> each, that would be about 13 megabytes, which is maybe an acceptable amount
> of RAM to spend on a log buffer.
>
> However, it isn't that. It's a buffer of 64k log ''events'', each of which
> probably has a `log_logger` and `log_source` set, each of which is an object
> attached to potentially arbitrary data. For example, every `Factory` that
> starts up logs something, which means you're holding on to an instance, and
> an instance dictionary, and the protocol instance, and the protocol instance
> dictionary. Worse yet, any logged ''failures'' might hold on to all the
> stuff on their stack.
>
> Add it all up and you end up with a log buffer totaling in the hundreds of
> megabytes, or even gigabytes, once it's full. In an application that naively
> uses Twisted without ever initializing logging, this hangs around forever.
>
> This buffer should probably be a ''lot'' smaller, and we might want to emit a
> warning when it fills up, reminding people that it is ''only polite'' to
> start up the logging subsystem, even just to explicitly throw logs away.
I fixed trac: https://twistedmatrix.com/trac/ticket/8936#ticket
<https://twistedmatrix.com/trac/ticket/8936#ticket>
- Amber
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
