I would suggest that your cleanup process also delete old state files, then if
you restart rsyslog weekly, any state files that have been deleted, but that
rsyslog had open, will be purged by the OS (they are relatively small, so
infrequent restarts should work, you don't want full restarts frequently,
because there is a window during the restart where rsyslog cannot proccess logs)
David Lang
On Fri, 25 Mar 2022, John Chivian wrote:
Date: Fri, 25 Mar 2022 13:42:33 -0500
From: John Chivian <jchiv...@chivian.com>
To: David Lang <da...@lang.hm>
Cc: "Cosmas, Cossy via rsyslog" <rsyslog@lists.adiscon.com>,
"Cosmas, Cossy" <cossy.cos...@dieboldnixdorf.com>
Subject: Re: [rsyslog] Imifile-state File Housekeeping Query...
Excellent, thank you David! Our rotation methodology is confirmed as required
until rsyslog supports deleting orphan state files at either startup or
shutdown, which to my way of thinking is a high priority add. We have some
cleanup tasks that spin every six hours because clients drop dated files as
often as every minute.
Regards,
On Mar 25, 2022, at 13:30, David Lang <da...@lang.hm> wrote:
rsyslog doesn't delete the old state files because it doesn't know if the file
is going to come back in a few seconds or not (such things happen), so the
decision was made to keep the files around as a lesser evil than re-ingesting
an old file that reappears.
for exactly this duplicate ingestion issue, I would suggest you either rotate
the file to a directory that rsyslog is not watching, or rotate it to a
filename that rsyslog is not watching.
If you app is creating files that have a date in their name, and creating new
files over time, this approach doesn't work, but that's a perfect example of
where rsyslog may think it's done with a file, but that it may reappear (either
because it's a new file created because the app is confused with the date, or
because someone restored it from elsewhere to look at it)
There is discussion of this problem elsewhere and talk of adding an option to
have rsyslog remove state files where no file exists to avoid the 'leak' of
state files, but opening up the risk of duplicate ingestion.
David Lang
On Fri, 25 Mar 2022, Cosmas, Cossy via rsyslog wrote:
Date: Fri, 25 Mar 2022 15:31:12 +0000
From: "Cosmas, Cossy via rsyslog" <rsyslog@lists.adiscon.com>
To: John Chivian <jchiv...@chivian.com>
Cc: "Cosmas, Cossy" <cossy.cos...@dieboldnixdorf.com>,
rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Imifile-state File Housekeeping Query...
Hi John,
I stopped (systemctl stop rsyslog.service) and then started(systemctl start
rsyslog.service) the rsyslog service but that hasn't reduced the number of
state files.
When you say rotate the files are you referring to the cron tab job that is
deleting all of the monitored audit log files?
That activity takes place overnight.
Regards,
Cossy
-----Original Message-----
From: Cosmas, Cossy
Sent: 25 March 2022 15:22
To: John Chivian <jchiv...@chivian.com>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: RE: [rsyslog] Imifile-state File Housekeeping Query...
Hi John,
Thanks for the advice, appreciate it.
Ill give it a go now.
Regards,
Cossy
-----Original Message-----
From: John Chivian <jchiv...@chivian.com>
Sent: 25 March 2022 15:21
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Cosmas, Cossy <cossy.cos...@dieboldnixdorf.com>
Subject: Re: [rsyslog] Imifile-state File Housekeeping Query...
! EXTERNAL MESSAGE - Think Before You Click or Download
My best advice is to stop and restart rsyslog after rotating files (a HUP won’t
do it). This makes rsyslog close and verify state files at shutdown, and the
ones for non-existent files will then get removed at startup.
Regards,
On Mar 25, 2022, at 10:10, Cosmas, Cossy via rsyslog
<rsyslog@lists.adiscon.com> wrote:
PS. We are running rsyslog 8.2006.
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Cosmas,
Cossy via rsyslog
Sent: 25 March 2022 10:02
To: rsyslog@lists.adiscon.com
Cc: Cosmas, Cossy <cossy.cos...@dieboldnixdorf.com>
Subject: [rsyslog] Imifile-state File Housekeeping Query...
! EXTERNAL MESSAGE - Think Before You Click or Download
Dear Rsyslog Forum Users,
A quick question from a relative newbie...
I have configured rsyslog to monitor my applications audit log files. I have
also implemented a cron based housekeeping script to delete the application
audit log files when they are over a week old.
I would have expected the rsyslog imifile-state files to automatically reduce
in number as the number of audit files decreases due to the above housekeeping
task but this is not the case.
The number of imifile-state files just keeps on increasing and this is
problematic as we have limits around the number of open files that rsyslog can
maintain.
I would just like to know what should be happening here and what is normal.
Does rsyslog ever automatically remove redundant state files or are they left
in place with the user expected to implement a cron based routine to delete
them manually?
Any advice appreciated.
Thank you.
Regards,
Cossy Cosmas
Payments and Transaction Management Services Diebold Nixdorf
Advanced notice of annual leave:
20th June - 24th June
22nd August - 4th September
Mobile: +44 7717 863755
One The Boulevard, Cain Road,
Bracknell, Berkshire, RG12 1WP
cossy.cos...@dieboldnixdorf.com<mailto:cossy.cos...@dieboldnixdorf.com
DieboldNixdorf.com
[1_twitter_logo_24px]<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=LUATZyF1IN8aMKCSuxYkfho4Vg6eU041XiNrdpyD3so&e=
> [2_facebook_logo_24px]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=X2WIo3MSLhpeBcW6VFX4Sy2SM0WbkIVr7xwShNubav4&e=
> [3_youtube_logo_24px]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=6wy_XQvegYq7h1tm5hOX8BXR_r-hKk9D3osb2Lz0Nro&e=
> [4_linkedin_logo_24px] <https://urldefense.proofpoint.com/v2/url?u=https-3A_
_w
ww.
linkedin.com_company_diebold&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=j7-qHkmmXOVmynEwlaLnBhLfDyOBUuON7TbZ8mexxOw&e=
> [5_blog_logo_24px] <http://blog.dieboldnixdorf.com/>
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon.net
_mailman_listinfo_rsyslog&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZ
bAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHT
VWykmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=FY_T0UNmcLGdLKOQg3GcZNwP-6fDEVWHKy0
aplakdEE&e=
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_pr
ofessional-2Dservices_&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAd
baH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWy
kmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=UF7Hr7h98tGBxyIDcwVhf3axMFQhzWkQSAh6T5
45tFs&e= What's up with rsyslog? Follow
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerh
ards&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_
Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWykmkZNx0bHELIbg9Zzf
t9Q6GbmLh4t&s=3pzUn1DEzQh35h5Uyo_0LY3g6k70GXY3BXy3KSLL55k&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
