I've pared down the debug file to 18MB and bzip2 - too large for pastebin. How an I get it to you?
On Thu, Dec 15, 2022 at 3:38 PM helices <[email protected]> wrote: > It happened again this afternoon: > > 2022-12-15T14:01:13.006027-06:00 hermes rsyslogd[10975]: > rsyslogd[internal_messages]: 793 messages lost due to rate-limiting (500 > allowed within 5 seconds) > 2022-12-15T14:01:19.005580-06:00 hermes rsyslogd[10975]: > rsyslogd[internal_messages]: 1272 messages lost due to rate-limiting (500 > allowed within 5 seconds) > 2022-12-15T14:01:25.000544-06:00 hermes rsyslogd[10975]: > rsyslogd[internal_messages]: 870 messages lost due to rate-limiting (500 > allowed within 5 seconds) > 2022-12-15T14:01:31.002353-06:00 hermes rsyslogd[10975]: > rsyslogd[internal_messages]: 1041 messages lost due to rate-limiting (500 > allowed within 5 seconds) > > On Wed, Dec 14, 2022 at 11:31 AM Rainer Gerhards <[email protected]> > wrote: > >> I ignore the database logging issue. When you have rate-limiting >> issues again, please report, together with the description of what >> happens. >> >> If you think this is related to mysql, please address that issue first. >> >> Rainer >> >> El mié, 14 dic 2022 a las 17:48, helices >> (<[email protected]>) escribió: >> > >> > REF: Rsyslogd/ommysql.so: Not writing to DB intermittently >> > >> > Rainer asked us to start a new post for the rate-limit issue. >> > >> > >> > A few of many hundreds of rate-limit errors and lost messages: >> > >> > 2022-12-13T02:23:44.003241-06:00 hermes rsyslogd[2539]: >> rsyslogd[internal_messages]: 1792 messages lost due to rate-limiting (500 >> allowed within 5 seconds) >> > 2022-12-13T02:23:50.001278-06:00 hermes rsyslogd[2539]: >> rsyslogd[internal_messages]: 1779 messages lost due to rate-limiting (500 >> allowed within 5 seconds) >> > 2022-12-13T02:23:56.001273-06:00 hermes rsyslogd[2539]: >> rsyslogd[internal_messages]: 1835 messages lost due to rate-limiting (500 >> allowed within 5 seconds) >> > 2022-12-13T02:24:02.005300-06:00 hermes rsyslogd[2539]: >> rsyslogd[internal_messages]: 1768 messages lost due to rate-limiting (500 >> allowed within 5 seconds) >> > s >> > >> > >> > # date; grep -v "^\(#\|\s*$\)" /etc/rsyslog.conf ;date >> > Wed Dec 14 10:35:41 CST 2022 >> > $DebugFile /var/log/rsyslog.debug >> > $DebugLevel 2 >> > module(load="imjournal" Ratelimit.Burst="30000" >> Ratelimit.Interval="1000" StateFile="imjournal.state") >> > module(load="imklog") >> > module(load="immark") >> > module(load="impstats" interval="600" severity="7") >> > syslog.=debug /var/log/rsyslog-stats >> > module(load="imtcp") >> > input(type="imtcp" port="514") >> > module(load="imudp") >> > input(type="imudp" port="514") >> > module(load="ommysql.so") >> > global(workDirectory="/var/lib/rsyslog") >> > authpriv.none;cron.none;*.info;mail.none /var/log/messages >> > authpriv.* /var/log/secure >> > cron.* /var/log/cron >> > *.emerg :omusrmsg:* >> > ftp.* /var/log/vsftpd.log >> > local7.* /var/log/boot.log >> > mail.* /var/log/maillog >> > uucp,news.crit /var/log/spooler >> > $ActionName Ftp >> > $ActionQueueFileName dbFtpQueue # Set file name, also enables disk >> mode >> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown >> > $ActionQueueType LinkedList # Use asynchronous processing >> > $ActionResumeRetryCount -1 # Infinite retries on insert failure >> > ftp.* >> :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____ >> > $ActionName Sftp >> > $ActionQueueFileName dbSftpQueue # Set file name, also enables disk >> mode >> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown >> > $ActionQueueType LinkedList # Use asynchronous processing >> > $ActionResumeRetryCount -1 # Infinite retries on insert failure >> > authpriv.* >> :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____ >> > $ActionName Admin >> > $ActionQueueFileName ZenossQueue # Set file name, also enables disk >> mode >> > $ActionQueueSaveOnShutdown on # Save messages to disk on shutdown >> > $ActionQueueType LinkedList # Use asynchronous processing >> > $ActionResumeRetryCount -1 # Infinite retries on insert failure >> > *.* @@10.199.1.160 >> > Wed Dec 14 10:35:41 CST 2022 >> > >> > >> > Rainer asked us to setup a debug log, according to: >> > https://www.rsyslog.com/doc/master/troubleshooting/howtodebug.html >> > >> > Initial startup here: >> > https://pastebin.com/DUgwmPC >> > >> > >> > No rate-limiting occurred since early yesterday (12/13) morning. This >> appears to be associated with the errors and multi-line syslog entries >> mentioned in the other post. >> > >> > The sole intent of the database logging is tracking all incoming remote >> file transfer (SFTP) activities. There is a firewall between this host and >> the internet. Only "whitelisted" IP addresses can get through, and are to >> be inserted into the database. >> > >> > Apparently, at least one client connects in the early morning hours, >> and this unusual SFTP unusual activity results in multi-line syslog entries >> that come in very large numbers. One problem is, the multiple line entries >> are not written to /var/log/messages, are not inserted into the database, >> and rate-limiting obscures all content. Hence, this support request is our >> attempt to understand what is happening, after which we can act to correct >> these problems. >> > >> > Interestingly, we are not aware of any missing files from this or any >> other file transfer clients. >> > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
