It was related to size of dynaFileCache. Strange is that the omfile thread was stopped.
Might make sense to update documentation in a way that the consequences of invalidating cache entry to omfile threading will be more clear. It is not mentioned at all. Peter On Tue, May 5, 2020 at 10:33 AM Peter Viskup <[email protected]> wrote: > Reported bug for Debian package > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959774 > > Following is the evidence of the rotated thread PIDs: > root@fwd01:~# date; pstree -t -sap 9276 > Tue 05 May 2020 08:05:27 AM UTC > systemd,1 > └─rsyslogd-local,9276 -n -f /etc/rsyslog.d/rsyslog-local.conf > ├─{in:impstats},9279 > ├─{in:imptcp},9281 > ├─{in:imptcp},9282 > ├─{in:imptcp},9283 > ├─{in:imudp},9280 > ├─{rs:ESP01-SURI q},6817 > ├─{rs:ESP03-WEB qu},9438 > ├─{rs:SIEMEP2 queu},6818 > ├─{rs:apache-site-},1001 > ├─{rs:apache-site-},1002 > ├─{rs:apache-site-},1003 > ├─{rs:haproxy.log},729 > ├─{rs:haproxy.log},992 > ├─{rs:local-all.lo},999 > ├─{rs:main Q:Reg},9437 > ├─{rs:nginx-site-a},914 > ├─{rs:nginx-site-a},987 > ├─{rs:nginx-site-e},986 > ├─{rs:nginx-site-e},989 > ├─{rsyslogd-local},9277 > └─{rsyslogd-local},9278 > root@fwd01:~# date; pstree -t -sap 9276 > Tue 05 May 2020 08:05:31 AM UTC > systemd,1 > └─rsyslogd-local,9276 -n -f /etc/rsyslog.d/rsyslog-local.conf > ├─{in:impstats},9279 > ├─{in:imptcp},9281 > ├─{in:imptcp},9282 > ├─{in:imptcp},9283 > ├─{in:imudp},9280 > ├─{rs:ESP01-SURI q},6817 > ├─{rs:ESP03-WEB qu},9438 > ├─{rs:SIEMEP2 queu},6818 > ├─{rs:apache-site-},1281 > ├─{rs:haproxy.log},1034 > ├─{rs:haproxy.log},1266 > ├─{rs:local-all.lo},1277 > ├─{rs:local-all.lo},1278 > ├─{rs:main Q:Reg},9437 > ├─{rs:nginx-site-a},1261 > ├─{rs:nginx-site-a},1273 > ├─{rs:nginx-site-e},1223 > ├─{rs:nginx-site-e},1260 > ├─{rsyslogd-local},9277 > ├─{rsyslogd-local},9278 > └─{rsyslogd-local},1276 > > > And stable list of thread PIDs after rsyslog restart > > root@fwd01:~# date; pstree -t -sap 12417 > Tue 05 May 2020 08:15:06 AM UTC > systemd,1 > └─rsyslogd-local,12417 -n -f /etc/rsyslog.d/rsyslog-local.conf > ├─{in:impstats},12420 > ├─{in:imptcp},12422 > ├─{in:imptcp},12423 > ├─{in:imptcp},12424 > ├─{in:imudp},12421 > ├─{rs:ESP03-GWS qu},14096 > ├─{rs:ESP03-WAF qu},14094 > ├─{rs:ESP03-WEB qu},14092 > ├─{rs:local-all.lo},14089 > ├─{rs:local-all.lo},14090 > ├─{rs:local-all.lo},14091 > ├─{rs:main Q:Reg},14088 > ├─{rs:nginx-server},14097 > ├─{rs:nginx-site-a},14093 > ├─{rs:nginx-site-e},14095 > ├─{rsyslogd-local},12418 > └─{rsyslogd-local},12419 > root@fwd01:~# date; pstree -t -sap 12417 > Tue 05 May 2020 08:15:36 AM UTC > systemd,1 > └─rsyslogd-local,12417 -n -f /etc/rsyslog.d/rsyslog-local.conf > ├─{in:impstats},12420 > ├─{in:imptcp},12422 > ├─{in:imptcp},12423 > ├─{in:imptcp},12424 > ├─{in:imudp},12421 > ├─{rs:ESP03-GWS qu},14096 > ├─{rs:ESP03-WAF qu},14094 > ├─{rs:ESP03-WEB qu},14092 > ├─{rs:local-all.lo},14089 > ├─{rs:local-all.lo},14090 > ├─{rs:local-all.lo},14091 > ├─{rs:main Q:Reg},14088 > ├─{rs:nginx-server},14097 > ├─{rs:nginx-site-a},14093 > ├─{rs:nginx-site-e},14095 > ├─{rsyslogd-local},12418 > └─{rsyslogd-local},12419 > > Peter > > On Mon, May 4, 2020 at 5:28 PM Peter Viskup <[email protected]> wrote: > >> For some weeks there are a lot of closing logfile notification via >> inotify seen on one syslog relay running rsyslog 8.1901 version. >> >> The messages like these >> >> May 4 15:10:04 fwd01 iWatch[31831]: * >> /chroot/local/var/log/h1/local-all.log is closed >> May 4 15:10:04 fwd01 iWatch[31831]: * >> /chroot/local/var/log/h3/haproxy.log is closed >> May 4 15:10:04 fwd01 iWatch[31831]: * >> /chroot/local/var/log/h1/haproxy.log is closed >> May 4 15:10:04 fwd01 iWatch[31831]: * >> /chroot/local/var/log/h5/nginx-site-access.log is closed >> May 4 15:10:04 fwd01 iWatch[31831]: * >> /chroot/local/var/log/h1/apache-site-error.log is closed >> May 4 15:10:04 fwd01 iWatch[31831]: * >> /chroot/local/var/log/h1/apache-site-access.log is closed >> >> are seen. With simple check of top -p PID I see that the thread PIDs of >> omfile are changing more times in second. Seems the logfiles are just >> updated. After restarting the instance, the situation is solved. The issue >> is maybe caused by logrotation job. >> >> Seems this bug is hit >> https://github.com/rsyslog/rsyslog/blob/master/ChangeLog#L533 >> >> Is there any way to prove it somehow? >> In that case would like to open bug report at Debian to make an patch >> backport to 8.1901 version which is Debian10 base. >> >> Peter >> > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
