Indeed this was a logrotate issue. Due to hourly logrotate cronjob, a
particular file needed more than an hour to compress with result another
logrotate process to kick in and mess things up.
I got misled because the particular large file was the last one before the
templated log files. For example, in my logrotate configuration:
file1
file2
large_file <--- logrotate will error here
syslog_template1
syslog_template2
syslog_templateN
adding debug and redirecting the output to a file on logrotate cronjob provided
more insight
I guess I reach a point where I have to implement rotation solutions like the
ones you guys mentioned.
Thanks again for the help
D.
________________________________
From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of David Lang via
rsyslog <rsyslog@lists.adiscon.com>
Sent: Friday, December 1, 2023 2:06 AM
To: Dan Pritts via rsyslog <rsyslog@lists.adiscon.com>
Cc: David Lang <da...@lang.hm>
Subject: Re: [rsyslog] logrotate wont compress-rotate syslog templated files
On my central rsyslog servers, I have rsyslog write the logs out to a directory
tree, then have a cron job that does a mv to a parallel directory structure on
the same filesystem (this is very fast and atomic, even when there are a lot of
files), then kick rsyslog to have it start writing to new files. Then I have a
separate process do any compression on the files later.
David Lang
On Thu, 30 Nov 2023, Dan Pritts via rsyslog wrote:
> [ catching up on lists ]
>
> One thing I’d strongly suggest is decoupling rotation from compression. If
> you compress from logrotate, it compresses files as it rotates them. This
> can add a significant amount of time to the logrotate job. if it happens
> significantly differently night to night, it might mess up your “daily”
> rotations.
>
> This also makes it easy to rotate everything at midnight so that the files
> get the right names for the days containing the data, modulo a couple
> minutes’ worth. IIRC I do the rotation at 23:59; if you have a truly large
> number of files o rotate you might need another minute.
>
> Compression and/or deletion later on is pretty simple with a find piped to
> xargs.
>
> --
> Dan Pritts
> System & Security Architect
> ICPSR Computing and Network Services
>
> On 14 Nov 2023, at 6:03, Dimi Onobodies via rsyslog wrote:
>
>> So I added "-v" option on the cronjob and redirected output to a file. I
>> observed the following:
>>
>> considering log /data/servers/rsyslog/ldap-access.log
>> log needs rotating
>>
>> rotating log /data/logs/apache.log, log->rotateCount is 4
>> dateext suffix '-20231114'
>> glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
>> glob finding old rotated logs failed
>>
>> rotating log /data/logs/maillog, log->rotateCount is 4
>> dateext suffix '-20231114'
>> glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
>> destination /data/logs/maillog-20231114.gz already exists, skipping rotation
>>
>>
>> All templated syslog files get "glob finding old rotated logs failed". Thing
>> is, I can see the file present in the filesystem:
>>
>> # ls /data/logs/apache.log-20231114
>> /data/logs/apache.log-20231114
>>
>> However, when I tried to view the logrotate file I noticed that it has
>> different output:
>>
>> "considering log /data/logs/apache.log
>> log does not need rotating (log has been rotated at 2023-11-14 0:1, that
>> is not day ago yet)"
>>
>> Its like logrotate was executed again. Also, there are two logrotate
>> messages in system log
>>
>> # grep ALERT /var/log/messages
>> 2023-11-14T01:01:06.664171-08:00 ALERT exited abnormally with [1]
>> 2023-11-14T01:24:56.322123-08:00 ALERT exited abnormally with [1]
>>
>>
>> One more thing I noticed when i sshed on the system was that gzip was
>> running:
>>
>>
>> # stat /data/logs/logrotate_debug.txt
>> File: ‘/data/servers/logrotate_debug.txt’
>> Size: 14410 Blocks: 32 IO Block: 4096 regular file
>> Device: 810h/2064d Inode: 36438030 Links: 1
>> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
>> Access: 2023-11-13 07:26:25.868304144 -0800
>> Modify: 2023-11-14 01:01:06.616954323 -0800
>> Change: 2023-11-14 01:01:06.616954323 -0800
>> Birth: -
>>
>>
>> # ps -ef|grep gz
>> root 1313 17221 97 00:45 ? 00:19:29 /bin/gzip
>> root 3849 1392 0 01:05 pts/1 00:00:00 grep --color=auto gz
>>
>> # ps -ef|grep 17221
>> root 1313 17221 97 00:45 ? 00:19:46 /bin/gzip
>> root 4108 1392 0 01:05 pts/1 00:00:00 grep --color=auto 17221
>> root 17221 17219 0 00:01 ? 00:00:05 /usr/sbin/logrotate -v -s
>> /var/lib/logrotate.status /etc/logrotate.conf
>>
>> # ps -ef|grep logrotate
>> root 4750 1392 0 01:06 pts/1 00:00:00 grep --color=auto logrotate
>> root 17219 17197 0 00:01 ? 00:00:00 /bin/sh
>> /etc/cron.hourly/logrotate
>> root 17220 17197 0 00:01 ? 00:00:00 awk -v
>> progname=/etc/cron.hourly/logrotate progname { ???? print progname ":\n"
>> ???? progname=""; ??? } ??? { print; }
>> root 17221 17219 0 00:01 ? 00:00:05 /usr/sbin/logrotate -v -s
>> /var/lib/logrotate.status /etc/logrotate.conf
>>
>>
>> Still cannot pinpoint if this is a rsyslog, logrotate, gzip issue. I dont
>> get it why this is happening only on logs generated with template
>> configuration directives.
>>
>> D.
>>
>> ________________________________
>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Mariusz Kruk
>> via rsyslog <rsyslog@lists.adiscon.com>
>> Sent: Monday, November 13, 2023 1:32 PM
>> To: rsyslog@lists.adiscon.com <rsyslog@lists.adiscon.com>
>> Cc: Mariusz Kruk <k...@epsilon.eu.org>
>> Subject: Re: [rsyslog] logrotate wont compress-rotate syslog templated files
>>
>> If the logrotate daily invocation already happened today, it won't
>> consider the files "rotate-worthy" before next day.
>>
>> So you might either add "-v" to the cron job so that it will tell you
>> verbosely what/why it does on the next "proper" run. Or do "-f -d".
>>
>> MK
>>
>> On 13.11.2023 14:13, Dimi Onobodies via rsyslog wrote:
>>> Hi Maiusz
>>>
>>> Forgot to add that in my description. I already tried that and i didnt see
>>> any error reporte from logrotate:
>>>
>>>
>>> * logrotate -d -v -s /var/lib/logrotate.status /etc/logrotate.conf
>>>
>>> considering log /data/logs/apache.log
>>> log does not need rotating (log has been already rotated)
>>>
>>>
>>> I am thinking to force (-f) rotation and see if this will show anything
>>> ________________________________
>>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Mariusz Kruk
>>> via rsyslog <rsyslog@lists.adiscon.com>
>>> Sent: Monday, November 13, 2023 9:13 AM
>>> To: rsyslog@lists.adiscon.com <rsyslog@lists.adiscon.com>
>>> Cc: Mariusz Kruk <k...@epsilon.eu.org>
>>> Subject: Re: [rsyslog] logrotate wont compress-rotate syslog templated files
>>>
>>> First and foremost, you should check what and why logrotate does when
>>> rotating files (the -v and - d options for logrotate). That will tell
>>> you what is actually happening when the rotation should occur.
>>>
>>>
>>> On 13.11.2023 09:57, Dimi Onobodies via rsyslog wrote:
>>>> Hi, I am facing a weird logrotate issue in my central syslog system
>>>> (receiving logs from various host via imtcp). Logrotate does not compress
>>>> syslog templated log files. It does compress and rotates other log files
>>>> generated by rsyslog, but not the ones generated using a template.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
