We're still missing something: module(load="imjournal" Ratelimit.Burst="30000" Ratelimit.Interval="1000" StateFile="imjournal.state")
2022-12-12T00:53:14.001626-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1728 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:53:20.004006-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1818 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:53:26.003870-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1794 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:53:32.005388-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1797 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:53:38.001367-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1812 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:53:44.006085-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1791 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:53:50.005487-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1797 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:53:56.001546-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1808 messages lost due to rate-limiting (500 allowed within 5 seconds) 2022-12-12T00:54:02.007743-06:00 hermes rsyslogd[1536]: rsyslogd[internal_messages]: 1759 messages lost due to rate-limiting (500 allowed within 5 seconds) What are we missing? Please, advise. Thank you. On Fri, Dec 9, 2022 at 8:49 AM Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > you set the interval, but not ratelimit.burst > > doc: > https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html > > Rainer > > El mar, 6 dic 2022 a las 15:16, helices via rsyslog > (<rsyslog@lists.adiscon.com>) escribió: > > > > David, > > > > What am I doing wrong? > > > > module(load="imjournal" Ratelimit.Interval="10000" > > StateFile="imjournal.state") > > > > 2022-12-06T07:19:26.004772-06:00 hermes rsyslogd[29735]: > > rsyslogd[internal_messages]: 1755 messages lost due to rate-limiting (500 > > allowed within 5 seconds) > > > > Please, advise. Thank you. > > > > ~ Mike > > > > > > > > On Thu, Dec 1, 2022 at 3:12 PM David Lang <da...@lang.hm> wrote: > > > > > On Thu, 1 Dec 2022, helices wrote: > > > > > > > [1] What is "action() syntax?" Which lines ought to be converted? > How? > > > > > > > > > > https://www.rsyslog.com/doc/master/configuration/basic_structure.html#statement-types > > > > > > instead of > > > > > > @@10.0.0.1 > > > > > > you would do > > > > > > action(type="omfwd" target="10.0.0.1" port="514" protocol="tcp") > > > > > > for this trivial example, the earlier syntax makes more sense, but when > > > you have > > > more complex things (like the queues that you have), adding them all > into > > > the > > > action makes it clearer exactly what is happening > > > > > > > > > so you currently have > > > > > > >>> $ActionName Admin > > > >>> $ActionQueueDequeueSlowdown 1000 # How long (in microseconds) > > > dequeueing > > > >>> should be delayed > > > >>> $ActionQueueFileName ZenossQueue # Set file name, also enables > disk > > > mode > > > >>> $ActionQueueSaveOnShutdown on # Save messages to disk on > shutdown > > > >>> $ActionQueueType LinkedList # Use asynchronous processing > > > >>> $ActionResumeRetryCount -1 # Infinite retries on insert > failure > > > >>> *.* @@10.199.1.160 > > > > > > > > > This would be > > > > > > action(name="Admin" type="omfwd" target="10.199.1.160" protocol="tcp" > > > queue.filename="ZenossQueue" queue.saveonshutdown="on" > > > queue.type="linkedlist" > > > resumeretrycount="-1" queue.dequeueslowdown="1000") > > > > > > this makes it very clear that all these parameters apply only to this > > > action > > > (which is what the old syntax does, but it's less obvious to people > that > > > it only > > > applies to the next action) > > > > > > > [2] Where is the "pause" you mention? I don't recognize that. > > > > > > $ActionQueueDequeueSlowdown 1000 # How long (in microseconds) > dequeueing > > > > > > This tells rsyslog to pause after each batch of messages before > processing > > > the > > > next batch. > > > > > > > [3] impstats? Permanently? Only for this debugging? > > > > > > I like to have it on permanently, but especially for debugging it > provides > > > a lot > > > of useful info > > > > > > > [4] How to modify imjournal rate limits? > > > > > > see > > > > https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html > > > > > > > [5] RSYSLOG_DebugFormat? I found this: > > > > https://www.rsyslog.com/doc/v8-stable/configuration/templates.html > - Is > > > > that example proper by itself? Where does this template go? How can I > > > > specify the file and location for debugging? > > > > > > as I said below > > > > > > >> ftp.* /var/log/ftp;RSYSLOG_DebugFormat (legacy format, add > > > template="RSYSLOG_DebugFormat" to that action() format) > > > > > > > If there are URLs to inform me, I appreciate your direction. > > > > > > > https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html > > > > https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommysql.html > > > https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html > > > https://www.rsyslog.com/doc/master/configuration/actions.html > > > https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html > > > > https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html > > > > > > feel free to keep asking questions. > > > > > > David Lang > > > > > > > > > > ~ Mike > > > > > > > > > > > > > > > > On Thu, Dec 1, 2022 at 1:33 PM David Lang <da...@lang.hm> wrote: > > > > > > > >> it would be useful to convert to the action() syntax as it makes it > > > >> clearer > > > >> what's happening. > > > >> > > > >> Why are you pausing between writing logs? (this could be why you are > > > >> dropping > > > >> logs) > > > >> > > > >> given the number of queues and actions, look at configuring > impstats so > > > >> that you > > > >> can see the number of messages in the queues, number processed, etc. > > > >> > > > >> imjournal defaults to some fairly aggressive rate limiting, I find > that > > > I > > > >> always > > > >> need to drastically increase the limits. > > > >> > > > >> writing logs using the RSYSLOG_DebugFormat is adding the template > to the > > > >> file > > > >> > > > >> ftp.* /var/log/ftp;RSYSLOG_DebugFormat (legacy format, add > > > >> template="RSYSLOG_DebugFormat" to that action() format) > > > >> > > > >> the debug format is large, but you really need to see the message > that's > > > >> failing > > > >> to figure out why it's failing. The MySQL logs may give you better > info > > > on > > > >> that. > > > >> > > > >> David Lang > > > >> > > > >> On Thu, 1 Dec 2022, helices wrote: > > > >> > > > >>> Date: Thu, 1 Dec 2022 13:26:47 -0600 > > > >>> From: helices <mike+rsys...@mdsresource.net> > > > >>> To: David Lang <da...@lang.hm> > > > >>> Cc: helices via rsyslog <rsyslog@lists.adiscon.com> > > > >>> Subject: Re: [rsyslog] Rsyslogd/ommysql.so: Not writing to DB > > > >> intermittently > > > >>> > > > >>> Thank you. > > > >>> > > > >>> [1] rsyslog.conf > > > >>> > > > >>> # date; grep -v "^\(#\|\s*$\)" /etc/rsyslog.conf ;date > > > >>> Thu Dec 1 13:19:34 CST 2022 > > > >>> module(load="imjournal" StateFile="imjournal.state") > > > >>> module(load="imklog") > > > >>> module(load="immark") > > > >>> module(load="impstats" interval="600" severity="7") > > > >>> syslog.=debug /var/log/rsyslog-stats > > > >>> module(load="imtcp") > > > >>> input(type="imtcp" port="514") > > > >>> module(load="imudp") > > > >>> input(type="imudp" port="514") > > > >>> module(load="ommysql.so") > > > >>> global(workDirectory="/var/lib/rsyslog") > > > >>> authpriv.none;cron.none;*.info;mail.none /var/log/messages > > > >>> authpriv.* /var/log/secure > > > >>> cron.* /var/log/cron > > > >>> *.emerg :omusrmsg:* > > > >>> ftp.* /var/log/vsftpd.log > > > >>> local7.* /var/log/boot.log > > > >>> mail.* /var/log/maillog > > > >>> uucp,news.crit /var/log/spooler > > > >>> $ActionName Ftp > > > >>> $ActionQueueDequeueSlowdown 1000 # How long (in microseconds) > > > dequeueing > > > >>> should be delayed > > > >>> $ActionQueueFileName dbFtpQueue # Set file name, also enables > disk > > > mode > > > >>> $ActionQueueSaveOnShutdown on # Save messages to disk on > shutdown > > > >>> $ActionQueueType LinkedList # Use asynchronous processing > > > >>> $ActionResumeRetryCount -1 # Infinite retries on insert > failure > > > >>> ftp.* > > > >>> :ommysql:10.199.5.177,vsftplog,hermesvsftplog,_____ > > > >>> $ActionName Sftp > > > >>> $ActionQueueDequeueSlowdown 1000 # How long (in microseconds) > > > >> dequeueing > > > >>> should be delayed > > > >>> $ActionQueueFileName dbSftpQueue # Set file name, also enables > disk > > > >> mode > > > >>> $ActionQueueSaveOnShutdown on # Save messages to disk on > shutdown > > > >>> $ActionQueueType LinkedList # Use asynchronous processing > > > >>> $ActionResumeRetryCount -1 # Infinite retries on insert > failure > > > >>> authpriv.* > > > >>> :ommysql:10.199.5.177,sftplogDB,hermesvsftplog,_____ > > > >>> $ActionName Admin > > > >>> $ActionQueueDequeueSlowdown 1000 # How long (in microseconds) > > > dequeueing > > > >>> should be delayed > > > >>> $ActionQueueFileName ZenossQueue # Set file name, also enables > disk > > > mode > > > >>> $ActionQueueSaveOnShutdown on # Save messages to disk on > shutdown > > > >>> $ActionQueueType LinkedList # Use asynchronous processing > > > >>> $ActionResumeRetryCount -1 # Infinite retries on insert > failure > > > >>> *.* @@10.199.1.160 > > > >>> Thu Dec 1 13:19:34 CST 2022 > > > >>> > > > >>> > > > >>> [2] How do we "log the message with the template > RSYSLOG_DebugFormat > > > to a > > > >>> file?" How much disk space is needed? This problem appears to have > > > >> started > > > >>> recently, and appears to happen once or twice per day, without a > common > > > >>> time. > > > >>> > > > >>> [3] I didn't notice the rate-limiting until now. It is not > uncommon. > > > How > > > >>> can we avoid losing so many messages? > > > >>> > > > >>> ~ Mike > > > >>> > > > >>> > > > >>> On Thu, Dec 1, 2022 at 1:05 PM David Lang <da...@lang.hm> wrote: > > > >>> > > > >>>> please post your full config. > > > >>>> > > > >>>> It would also help to log the message with the template > > > >>>> RSYSLOG_DebugFormat to a > > > >>>> file and find the log entry that is failing to insert. > > > >>>> > > > >>>> my guess is that the quotes in the message are confusing mysql > > > >>>> > > > >>>> note that rate limiting is throwing away messages because you are > > > trying > > > >>>> to > > > >>>> process them too fast. > > > >>>> > > > >>>> David Lang > > > >>>> > > > >>>> On Thu, 1 Dec 2022, helices via rsyslog wrote: > > > >>>> > > > >>>>> Date: Thu, 1 Dec 2022 10:08:01 -0600 > > > >>>>> From: helices via rsyslog <rsyslog@lists.adiscon.com> > > > >>>>> To: rsyslog-users <rsyslog@lists.adiscon.com> > > > >>>>> Cc: helices <mike+rsys...@mdsresource.net> > > > >>>>> Subject: [rsyslog] Rsyslogd/ommysql.so: Not writing to DB > > > >> intermittently > > > >>>>> > > > >>>>> # date; /bin/yum list rsyslog rsyslog-mysql ;date > > > >>>>> Thu Dec 1 09:47:18 CST 2022 > > > >>>>> Loaded plugins: fastestmirror > > > >>>>> Loading mirror speeds from cached hostfile > > > >>>>> * base: download.cf.centos.org > > > >>>>> * epel: mirror.genesisadaptive.com > > > >>>>> * extras: download.cf.centos.org > > > >>>>> * remi-php56: mirror.pit.teraswitch.com > > > >>>>> * remi-safe: mirror.pit.teraswitch.com > > > >>>>> * updates: download.cf.centos.org > > > >>>>> Installed Packages > > > >>>>> rsyslog.x86_64 > > > 8.2210.0-1.el7 > > > >>>>> @rsyslog_v8 > > > >>>>> rsyslog-mysql.x86_64 > > > 8.2210.0-1.el7 > > > >>>>> @rsyslog_v8 > > > >>>>> Thu Dec 1 09:47:19 CST 2022 > > > >>>>> > > > >>>>> > > > >>>>> Sample of numerous error messages (/var/log/messages): > > > >>>>> rsyslogd[17344]: ommysql: db error (1172): Result consisted of > more > > > >> than > > > >>>>> one row [v8.2210.0] > > > >>>>> rsyslogd[17344]: The error statement was: insert into > SystemEvents > > > >>>>> (Message, Facility, FromHost, Priority, DeviceReportedTime, > > > ReceivedAt, > > > >>>>> InfoUnitID, SysLogTag) values ('close > > > >>>>> "/incoming/wood.pgez.scen.11302022.sa.pgp" bytes read 0 written > 2603 > > > >>>>> [postauth]', 10, 'hermes', 6, '20221201081257', > '20221201081257', 1, > > > >>>>> 'sshd[19654]:') [v8.2210.0 try https://www.rsyslog.com/e/2218 ] > > > >>>>> rsyslogd[17344]: rsyslogd[internal_messages]: 215 messages lost > due > > > to > > > >>>>> rate-limiting (500 allowed within 5 seconds) > > > >>>>> rsyslogd[17344]: action 'Sftp' (module 'ommysql.so') message > lost, > > > >> could > > > >>>>> not be processed. Check for additional error messages before this > > > one. > > > >>>>> [v8.2210.0 try https://www.rsyslog.com/e/2218 ] > > > >>>>> > > > >>>>> > > > >>>>> We have been writing all data from Internet file transfers to a > Mysql > > > >>>> table > > > >>>>> for years. Recently, we began seeing intermittent errors like > those > > > >>>> above. > > > >>>>> > > > >>>>> What is happening here? > > > >>>>> > > > >>>>> What can we do to fix this problem? > > > >>>>> > > > >>>>> Please, advise. Thank you. > > > >>>>> > > > >>>>> ~ Mike > > > >>>>> _______________________________________________ > > > >>>>> rsyslog mailing list > > > >>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > > > >>>>> http://www.rsyslog.com/professional-services/ > > > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > >> myriad > > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you > > > >>>> DON'T LIKE THAT. > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
