Hi Sehrope, On 2019-Jul-10, Sehrope Sarkuni wrote:
> While working on adding a new log_destination I noticed that the > syslogger piping would need to be updated. At the moment both ends > only handle stderr/csvlog as the pipe message header has a char > "is_last" that is either t/f (stderr last, stderr partial) or T/F > (csvlog last, csvlog partial). I spent some time a couple of weeks ago looking at profiles of the syslogger code, and my impression is that the current design of using a pipe to move data from one process to another may benefit from a complete rewrite; it seems that the kernel overhead of going over the pipe is significant. (The test case was a couple dozen processes all generating a thousand of couple-hundred-KB log lines. In perf, the pipe read/write takes up 99% of the CPU time). Maybe we can use something like a shared memory queue, working in a similar way to wal_buffers -- where backends send over the shm queue to syslogger, and syslogger writes in order to the actual log file. Or maybe something completely different; I didn't actually prototype anything, just observed the disaster. I'm not opposed to your patches, just trying to whet your appetite for something bigger in the vicinity. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
