Hi David,
I just wanted to confirm that upgrading rsyslog to version 2202 from 2006 has resolved our issue and state files are now being deleted. We are very happy. 😊 Thank you. Regards, Cossy -----Original Message----- From: David Lang <da...@lang.hm> Sent: 25 March 2022 23:19 To: David Lang <da...@lang.hm> Cc: John Chivian <jchiv...@chivian.com>; Cosmas, Cossy via rsyslog <rsyslog@lists.adiscon.com>; Cosmas, Cossy <cossy.cos...@dieboldnixdorf.com> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... ! EXTERNAL MESSAGE - Think Before You Click or Download new parameter that was added deleteStateOnFileDelete you will need to be running a pretty current version to have that. David Lang On Fri, 25 Mar 2022, David Lang wrote: > Date: Fri, 25 Mar 2022 11:48:02 -0700 (PDT) > From: David Lang <da...@lang.hm<mailto:da...@lang.hm>> > To: John Chivian <jchiv...@chivian.com<mailto:jchiv...@chivian.com>> > Cc: David Lang <da...@lang.hm<mailto:da...@lang.hm>>, > "Cosmas, Cossy via rsyslog" > <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>, > "Cosmas, Cossy" > <cossy.cos...@dieboldnixdorf.com<mailto:cossy.cos...@dieboldnixdorf.com>> > Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... > > I would suggest that your cleanup process also delete old state files, > then if you restart rsyslog weekly, any state files that have been > deleted, but that rsyslog had open, will be purged by the OS (they are > relatively small, so infrequent restarts should work, you don't want > full restarts frequently, because there is a window during the restart > where rsyslog cannot proccess > logs) > > David Lang > > On Fri, 25 Mar 2022, John Chivian wrote: > >> Date: Fri, 25 Mar 2022 13:42:33 -0500 >> From: John Chivian <jchiv...@chivian.com<mailto:jchiv...@chivian.com>> >> To: David Lang <da...@lang.hm<mailto:da...@lang.hm>> >> Cc: "Cosmas, Cossy via rsyslog" >> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>, >> "Cosmas, Cossy" >> <cossy.cos...@dieboldnixdorf.com<mailto:cossy.cos...@dieboldnixdorf.com>> >> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... >> >> Excellent, thank you David! Our rotation methodology is confirmed as >> required until rsyslog supports deleting orphan state files at either >> startup or shutdown, which to my way of thinking is a high priority add. >> We have some cleanup tasks that spin every six hours because clients >> drop dated files as often as every minute. >> >> Regards, >> >> >>> On Mar 25, 2022, at 13:30, David Lang <da...@lang.hm<mailto:da...@lang.hm>> >>> wrote: >>> >>> rsyslog doesn't delete the old state files because it doesn't know >>> if the file is going to come back in a few seconds or not (such >>> things happen), so the decision was made to keep the files around as >>> a lesser evil than re-ingesting an old file that reappears. >>> >>> for exactly this duplicate ingestion issue, I would suggest you >>> either rotate the file to a directory that rsyslog is not watching, >>> or rotate it to a filename that rsyslog is not watching. >>> >>> If you app is creating files that have a date in their name, and >>> creating new files over time, this approach doesn't work, but that's >>> a perfect example of where rsyslog may think it's done with a file, >>> but that it may reappear (either because it's a new file created >>> because the app is confused with the date, or because someone >>> restored it from elsewhere to look at it) >>> >>> There is discussion of this problem elsewhere and talk of adding an >>> option to have rsyslog remove state files where no file exists to >>> avoid the 'leak' of state files, but opening up the risk of duplicate >>> ingestion. >>> >>> David Lang >>> >>> On Fri, 25 Mar 2022, Cosmas, Cossy via rsyslog wrote: >>> >>>> Date: Fri, 25 Mar 2022 15:31:12 +0000 >>>> From: "Cosmas, Cossy via rsyslog" >>>> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>> >>>> To: John Chivian <jchiv...@chivian.com<mailto:jchiv...@chivian.com>> >>>> Cc: "Cosmas, Cossy" >>>> <cossy.cos...@dieboldnixdorf.com<mailto:cossy.cos...@dieboldnixdorf.com>>, >>>> rsyslog-users >>>> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>> >>>> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... >>>> >>>> Hi John, >>>> >>>> I stopped (systemctl stop rsyslog.service) and then >>>> started(systemctl start rsyslog.service) the rsyslog service but >>>> that hasn't reduced the number of state files. >>>> >>>> When you say rotate the files are you referring to the cron tab job >>>> that is deleting all of the monitored audit log files? >>>> >>>> That activity takes place overnight. >>>> >>>> Regards, >>>> Cossy >>>> >>>> -----Original Message----- >>>> From: Cosmas, Cossy >>>> Sent: 25 March 2022 15:22 >>>> To: John Chivian <jchiv...@chivian.com<mailto:jchiv...@chivian.com>> >>>> Cc: rsyslog-users >>>> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>> >>>> Subject: RE: [rsyslog] Imifile-state File Housekeeping Query... >>>> >>>> >>>> Hi John, >>>> >>>> Thanks for the advice, appreciate it. >>>> >>>> Ill give it a go now. >>>> >>>> Regards, >>>> Cossy >>>> >>>> -----Original Message----- >>>> From: John Chivian <jchiv...@chivian.com<mailto:jchiv...@chivian.com>> >>>> Sent: 25 March 2022 15:21 >>>> To: rsyslog-users >>>> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>> >>>> Cc: Cosmas, Cossy >>>> <cossy.cos...@dieboldnixdorf.com<mailto:cossy.cos...@dieboldnixdorf.com>> >>>> Subject: Re: [rsyslog] Imifile-state File Housekeeping Query... >>>> >>>> ! EXTERNAL MESSAGE - Think Before You Click or Download >>>> >>>> My best advice is to stop and restart rsyslog after rotating files >>>> (a HUP won’t do it). This makes rsyslog close and verify state >>>> files at shutdown, and the ones for non-existent files will then >>>> get removed at startup. >>>> >>>> Regards, >>>> >>>>> On Mar 25, 2022, at 10:10, Cosmas, Cossy via rsyslog >>>>> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>> wrote: >>>>> >>>>> >>>>> PS. We are running rsyslog 8.2006. >>>>> >>>>> -----Original Message----- >>>>> From: rsyslog >>>>> <rsyslog-boun...@lists.adiscon.com<mailto:rsyslog-boun...@lists.adiscon.com>> >>>>> On Behalf Of >>>>> Cosmas, Cossy via rsyslog >>>>> Sent: 25 March 2022 10:02 >>>>> To: rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com> >>>>> Cc: Cosmas, Cossy >>>>> <cossy.cos...@dieboldnixdorf.com<mailto:cossy.cos...@dieboldnixdorf.com>> >>>>> Subject: [rsyslog] Imifile-state File Housekeeping Query... >>>>> >>>>> ! EXTERNAL MESSAGE - Think Before You Click or Download >>>>> >>>>> >>>>> Dear Rsyslog Forum Users, >>>>> >>>>> A quick question from a relative newbie... >>>>> >>>>> I have configured rsyslog to monitor my applications audit log >>>>> files. I have also implemented a cron based housekeeping script to >>>>> delete the application audit log files when they are over a week old. >>>>> >>>>> I would have expected the rsyslog imifile-state files to >>>>> automatically reduce in number as the number of audit files >>>>> decreases due to the above housekeeping task but this is not the case. >>>>> >>>>> The number of imifile-state files just keeps on increasing and >>>>> this is problematic as we have limits around the number of open >>>>> files that rsyslog can maintain. >>>>> >>>>> I would just like to know what should be happening here and what >>>>> is normal. >>>>> >>>>> Does rsyslog ever automatically remove redundant state files or >>>>> are they left in place with the user expected to implement a cron >>>>> based routine to delete them manually? >>>>> >>>>> Any advice appreciated. >>>>> >>>>> Thank you. >>>>> >>>>> Regards, >>>>> >>>>> Cossy Cosmas >>>>> Payments and Transaction Management Services Diebold Nixdorf >>>>> >>>>> Advanced notice of annual leave: >>>>> 20th June - 24th June >>>>> 22nd August - 4th September >>>>> >>>>> Mobile: +44 7717 863755 >>>>> One The Boulevard, Cain Road, >>>>> Bracknell, Berkshire, RG12 1WP >>>>> >>>>> cossy.cos...@dieboldnixdorf.com<mailto:cossy.cosmas@dieboldnixdorf<mailto:cossy.cos...@dieboldnixdorf.com%3cmailto:cossy.cosmas@dieboldnixdorf> >>>>> .com >>>>>> >>>>> DieboldNixdorf.com >>>>> >>>>> [1_twitter_logo_24px]<https://urldefense.proofpoint.com/v2/url?u=h >>>>> ttps-3A__twitter.com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n >>>>> 6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIg >>>>> zlx3E3y8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=LUATZyF1IN8aM >>>>> KCSuxYkfho4Vg6eU041XiNrdpyD3so&e= >>>>> > [2_facebook_logo_24px] >>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook >>>>> .com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbA >>>>> dbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOF >>>>> BB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=X2WIo3MSLhpeBcW6VFX4Sy2SM0Wbk >>>>> IVr7xwShNubav4&e= >>>>> > [3_youtube_logo_24px] >>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube. >>>>> com_DieboldNixdorf&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAd >>>>> baH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y8U6dczMOFB >>>>> B4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=6wy_XQvegYq7h1tm5hOX8BXR_r-hKk >>>>> 9D3osb2Lz0Nro&e= >>>>> > [4_linkedin_logo_24px] >>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A_ > _w >>> ww. >>>>> linkedin.com_company_diebold&d=DwIFAg&c=7my1DiYA8Epq5UwiA7n6nQ&r=S >>>>> v0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=_ln9W7In6NJgIgzlx3E3y >>>>> 8U6dczMOFBB4D7C0kaTcB3luvx2uBUcdM2AK0b5Hys8&s=j7-qHkmmXOVmynEwlaLn >>>>> BhLfDyOBUuON7TbZ8mexxOw&e= >>>>> > [5_blog_logo_24px] <http://blog.dieboldnixdorf.com/> >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon >>>>> .net >>>>> _mailman_listinfo_rsyslog&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0V >>>>> nMLZ >>>>> bAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZ >>>>> nfHT >>>>> VWykmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=FY_T0UNmcLGdLKOQg3GcZNwP-6fDEVW >>>>> HKy0 >>>>> aplakdEE&e= >>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.co >>>>> m_pr >>>>> ofessional-2Dservices_&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnML >>>>> ZbAd >>>>> baH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfH >>>>> TVWy >>>>> kmkZNx0bHELIbg9Zzft9Q6GbmLh4t&s=UF7Hr7h98tGBxyIDcwVhf3axMFQhzWkQSA >>>>> h6T5 45tFs&e= What's up with rsyslog? Follow >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_r >>>>> gerh >>>>> ards&d=DwIFaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOYEE >>>>> LYL_ >>>>> Sa9QWHeuqpB2AY&m=DHsPogUuazI8EUh7DqtQalde8tqZnfHTVWykmkZNx0bHELIbg >>>>> 9Zzf t9Q6GbmLh4t&s=3pzUn1DEzQh35h5Uyo_0LY3g6k70GXY3BXy3KSLL55k&e= >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>> POST if you DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.adiscon. >>>> net_mailman_listinfo_rsyslog&d=DwIDaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv >>>> 0VnMLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=sxc7AQopnmGFMKWghbBfpgW >>>> Z12-Poz1N_cc1QFstrQY89dU3XyXsXKwfGegHwe7M&s=rzTUj0oTW4-e_oUoIYX4jeb >>>> Yaqd3P_UpB1-1FLHOpc8&e= >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com >>>> _professional-2Dservices_&d=DwIDaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0Vn >>>> MLZbAdbaH6yPjH_FOYEELYL_Sa9QWHeuqpB2AY&m=sxc7AQopnmGFMKWghbBfpgWZ12 >>>> -Poz1N_cc1QFstrQY89dU3XyXsXKwfGegHwe7M&s=TMJoikgVk5fM5cl3ptYLaXW3xy >>>> WNmZsHFKA9tZ4WKQU&e= What's up with rsyslog? Follow >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rg >>>> erhards&d=DwIDaQ&c=7my1DiYA8Epq5UwiA7n6nQ&r=Sv0VnMLZbAdbaH6yPjH_FOY >>>> EELYL_Sa9QWHeuqpB2AY&m=sxc7AQopnmGFMKWghbBfpgWZ12-Poz1N_cc1QFstrQY8 >>>> 9dU3XyXsXKwfGegHwe7M&s=UTBgWBv11s5ZXE2glYcGUDiPgCafELIViaeXI_sU1eE& >>>> e= NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>>> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>> POST if you DON'T LIKE THAT. >> > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
