Happening again: $ sudo du -sh * 9.8M bin 26M boot 220K dev 6.9M etc 3.5G home 129M lib 22M lib64 20K lost+found 8.0K media 8.0K mnt 6.4G opt du: cannot access `proc/2713/task/9576': No such file or directory du: cannot access `proc/9490/task/9490/fd/4': No such file or directory du: cannot access `proc/9490/task/9490/fdinfo/4': No such file or directory du: cannot access `proc/9490/fd/4': No such file or directory du: cannot access `proc/9490/fdinfo/4': No such file or directory 0 proc 116K root 13M sbin 4.0K selinux 4.0K srv 0 sys 296K tmp 1.3G usr 39G var
$ sudo df -kh Filesystem Size Used Avail Use% Mounted on /dev/md2 913G 212G 655G 25% / tmpfs 16G 0 16G 0% /dev/shm /dev/md0 97M 31M 62M 34% /boot I am seeing the same baheviour on a second system. On Tue, Jun 25, 2013 at 12:04 PM, Xuri Nagarin <[email protected]> wrote: > My suspicion was something to do with dynamic file creation but looking at > the code, both static and dynamic file creation use the same function of > file "open": > > From tools/omfile.c > ------------xxxxxxxxxxxxxxxxx--------------- > fd = open((char*) newFileName, > O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, > 556 pData->fCreateMode); > ------------xxxxxxxxxxxxxxxxx--------------- > > "O_APPEND" should take care of file being rotated while rsyslog is trying > to write to it. > > > > > On Tue, Jun 25, 2013 at 11:47 AM, Xuri Nagarin <[email protected]> wrote: > >> Not a typo or error, in RHEL the rsyslogd start up script in init.d has >> an explicit variable "PIDFILE" set as: >> PIDFILE=/var/run/syslogd.pid >> >> :) >> >> I did not write the script, using whatever was bundled in the RPM I >> grabbed from Adiscon. >> >> To confirm: >> $ ps ax | grep rsyslog >> 11331 ? Sl 61:18 /sbin/rsyslogd -i /var/run/syslogd.pid -x >> >> >> >> >> >> On Tue, Jun 25, 2013 at 11:41 AM, Soham Chakraborty < >> [email protected]> wrote: >> >>> Hi, >>> >>> In the logrotate config, change it to /var/run/rsyslog.pid and test. On a >>> quick skim, it looks like a typo. >>> >>> Soham >>> >>> >>> On Tue, Jun 25, 2013 at 11:07 PM, David Lang <[email protected]> wrote: >>> >>> > well, one thing that looks wrong is that logrotate is looking for >>> > /var/run/syslog.pid, but on my systems the pid is in >>> /var/log/rsyslog.pid >>> > >>> > are you sure that rsyslog is actually getting the HUP? >>> > >>> > can you try sending it a HUP manually and see if it closes the files? >>> > >>> > David Lang >>> > >>> > >>> > On Tue, 25 Jun 2013, Xuri Nagarin wrote: >>> > >>> > Date: Tue, 25 Jun 2013 11:34:08 -0700 >>> >> From: Xuri Nagarin <[email protected]> >>> >> Reply-To: rsyslog-users <[email protected]> >>> >> To: rsyslog-users <[email protected]> >>> >> Subject: Re: [rsyslog] HUP-ing rsyslog does not free up disk space >>> >> >>> >> >>> >> Hi David, >>> >> >>> >> The master conf file is simple with few additions that are >>> >> self-explanatory. The other conf in rsyslog.d, I have added comments >>> >> inline >>> >> in the conf that explain the config logic. >>> >> >>> >> >>> >> /etc/rsyslog.conf >>> >> ==============================**==============================** >>> >> =========== >>> >> module(load="impstats" interval="600" severity="7" >>> >> >>> >> log.syslog="off" /* need to turn log stream logging off! */ >>> >> log.file="/var/log/rsyslog-**stats.log") >>> >> >>> >> $ModLoad imuxsock.so # Unix sockets >>> >> $ModLoad imklog.so # Kernel logger >>> >> $MainMsgQueueSize 1000000 >>> >> $OMFileIOBufferSize 512k >>> >> $MaxMessageSize 8k >>> >> $MainMsgQueueWorkerThreads 64 >>> >> $umask 0000 >>> >> $FileOwner joe >>> >> $FileGroup joe >>> >> $DirOwner joe >>> >> $DirGroup joe >>> >> $DirCreateMode 0755 >>> >> $FileCreateMode 0644 >>> >> >>> >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>> >> $IncludeConfig /etc/rsyslog.d/*.conf >>> >> >>> >> $RuleSet local >>> >> *.info;mail.none;authpriv.**none;cron.none >>> >> /var/log/messages >>> >> authpriv.* >>> /var/log/secure >>> >> mail.* >>> -/var/log/maillog >>> >> cron.* /var/log/cron >>> >> *.emerg * >>> >> uucp,news.crit >>> /var/log/spooler >>> >> local7.* >>> /var/log/boot.log >>> >> $DefaultRuleset local >>> >> ==============================**==============================** >>> >> =========== >>> >> >>> >> /etc/rsyslog.d/cef.conf >>> >> ==============================**==============================** >>> >> =========== >>> >> # Senders are four Arcsight Logger devices that send logs in CEF >>> format >>> >> >>> >> #template for writing CEF formatted logs >>> >> template(name="cefdynfile" type="string" >>> >> string="/var/log/joe/%$!**vendor%/%$!product%/logfile") >>> >> >>> >> #template for writing logs from non-CEF sources >>> >> template(name="noncefdynfile" type="string" >>> >> string="/var/log/joe/noncef/%**hostname%/%programname%/**logfile") >>> >> >>> >> ruleset(name="tcpcef") { >>> >> >>> >> #CEF uses the pipe delimiter, fields 2 and 3 are product vendor and >>> >> product >>> >> type respectively >>> >> set $!vendor = field($msg, 124, 2); >>> >> set $!product = field($msg, 124, 3); >>> >> >>> >> # Rules to write CEF formatted logs to disk and send logs by app type >>> to >>> >> their flume destinations that are listening on the same box >>> >> >>> >> if $syslogtag=="CEF:" then { action (type="omfile" ASyncWriting="on" >>> >> IOBufferSize="8192K" FileOwner="joe" FileGroup="joe" DirOwner="joe" >>> >> DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644" >>> >> DynaFile="cefdynfile") } >>> >> if $!product == "app1" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5161" Protocol="tcp") stop } >>> >> if $!product == "app2" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5146" Protocol="tcp") stop } >>> >> if $!product == "app3" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5172" Protocol="tcp") stop } >>> >> if $!product == "app4" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5162" Protocol="tcp") stop } >>> >> if $!product == "app5" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5166" Protocol="tcp") stop } >>> >> if $!product == "app6" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5163" Protocol="tcp") stop } >>> >> if $!product == "app7" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5164" Protocol="tcp") stop } >>> >> if $!product == "app8" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5177" Protocol="tcp") stop } >>> >> if $!product == "app9" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5144" Protocol="tcp") stop } >>> >> if $!product == "app10" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5145" Protocol="tcp") stop } >>> >> if $!product == "app11" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5148" Protocol="tcp") stop } >>> >> if $!product == "app12" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5180" Protocol="tcp") stop } >>> >> if $!product == "app13" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5147" Protocol="tcp") stop } >>> >> if $!product == "app14" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5149" Protocol="tcp") stop } >>> >> if $!product == "app15" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5150" Protocol="tcp") stop } >>> >> if $!product == "app16" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5151" Protocol="tcp") stop } >>> >> if $!product == "app17" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5152" Protocol="tcp") stop } >>> >> if $!product == "app18" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5153" Protocol="tcp") stop } >>> >> if $!product == "app19" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5155" Protocol="tcp") stop } >>> >> if $!product == "app20" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5156" Protocol="tcp") stop } >>> >> if $!product == "app21" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5157" Protocol="tcp") stop } >>> >> if $!product == "app22" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5158" Protocol="tcp") stop } >>> >> if $!product == "app23" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5159" Protocol="tcp") stop } >>> >> if $!product == "app24" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5160" Protocol="tcp") stop } >>> >> if $!product == "app25" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5178" Protocol="tcp") stop } >>> >> if $!product == "app26" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5165" Protocol="tcp") stop } >>> >> if $!product == "app27" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5165" Protocol="tcp") stop } >>> >> if $!product == "app28" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5167" Protocol="tcp") stop } >>> >> if $!product == "app29" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5167" Protocol="tcp") stop } >>> >> if $!product == "app30" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5179" Protocol="tcp") stop } >>> >> if $!product == "app31" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5169" Protocol="tcp") stop } >>> >> if $!product == "app32" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5170" Protocol="tcp") stop } >>> >> if $!product == "app33" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5171" Protocol="tcp") stop } >>> >> if $!product == "app34" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5174" Protocol="tcp") stop } >>> >> if $!product == "app35" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5173" Protocol="tcp") stop } >>> >> if $!product == "app36" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5175" Protocol="tcp") stop } >>> >> if $!product == "app37" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5176" Protocol="tcp") stop } >>> >> if $!product == "app38" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5154" Protocol="tcp") stop } >>> >> if $!product == "app39" then { action (type="omfwd" Target="127.0.0.1" >>> >> Port="5181" Protocol="tcp") stop } >>> >> >>> >> # Unfortunately, the four Arcsight Loggers also send us garbage so >>> >> whatever >>> >> could not be parsed/classified correctly by the rules above, gets >>> parked >>> >> in >>> >> a catchall file. >>> >> >>> >> if $fromhost-ip == '10.1.1.100' or $fromhost-ip == '10.1.1.101' or >>> >> $fromhost-ip == '10.1.1.102' or $fromhost-ip == '10.1.1.103' then { >>> action >>> >> (type="omfile" FileOwner="joe" FileGroup="joe" DirOwner="joe" >>> >> DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644" >>> >> file="/var/log/joe/fallback/**logfile") >>> >> & action (type="omfwd" Target="127.0.0.1" Port="5182" >>> Protocol="tcp" >>> >> ) >>> >> stop } >>> >> >>> >> # Take care of all the non-CEF / BSD Syslog formatted streams coming >>> in >>> >> else { >>> >> action (type="omfile" ASyncWriting="on" IOBufferSize="8192K" >>> >> FileOwner="joe" FileGroup="joe" DirOwner="joe" DirGroup="joe" >>> >> DirCreateMode="0755" FileCreateMode="0644" DynaFile="noncefdynfile") >>> >> & action (type="omfwd" Target="127.0.0.1" Port="5182" >>> Protocol="tcp") >>> >> } >>> >> } >>> >> >>> >> module(load="imtcp" ) # needs to be done just once >>> >> input(type="imtcp" port="514" ruleset="tcpcef") >>> >> ==============================**==============================** >>> >> =========== >>> >> >>> >> >>> >> Here's the logrotate code: >>> >> ==============================**==============================** >>> >> =========== >>> >> /var/log/joe/*/*/* >>> >> /var/log/joe/*/*/*/* >>> >> { >>> >> missingok >>> >> size 1G >>> >> rotate 0 >>> >> sharedscripts >>> >> postrotate >>> >> if [ -f /var/run/syslog.pid ]; then \ >>> >> kill -HUP `cat /var/run/syslog.pid`; \ >>> >> fi; >>> >> endscript >>> >> } >>> >> ==============================**==============================** >>> >> =========== >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> On Tue, Jun 25, 2013 at 10:12 AM, David Lang <[email protected]> wrote: >>> >> >>> >> On Tue, 25 Jun 2013, Xuri Nagarin wrote: >>> >>> >>> >>> On RHEL 6.2 64-bit, I have Rsyslog 7.4.1 (actually, the issue has >>> >>> existed >>> >>> >>> >>>> for earlier versions too). >>> >>>> >>> >>>> After logrotation, the logrotate script HUPs the rsyslogd pid but >>> the >>> >>>> disk >>> >>>> space doesn't free up until I restart rsyslog. After a few hours >>> "df" >>> >>>> reports a full filesystem whereas "ls" shows much smaller file >>> sizes. In >>> >>>> between this confusion, rsyslog seems to stop receiving log streams. >>> >>>> >>> >>>> Is there a way for rsyslog to switch file handles when it is HUP-ed? >>> >>>> >>> >>>> >>> >>> rsyslog is already supposed to close and re-open files when it's >>> HUP-ed, >>> >>> and it seems to be working for me and many others. >>> >>> >>> >>> could you send your config file so we can see if there is anything >>> >>> unusual >>> >>> in it? >>> >>> >>> >>> David Lang >>> >>> ______________________________****_________________ >>> >>> rsyslog mailing list >>> >>> http://lists.adiscon.net/****mailman/listinfo/rsyslog< >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>> >>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog< >>> http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> >>> > >>> >>> http://www.rsyslog.com/****professional-services/< >>> http://www.rsyslog.com/**professional-services/> >>> >>> <http://**www.rsyslog.com/professional-**services/< >>> http://www.rsyslog.com/professional-services/> >>> >>> > >>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you >>> >>> DON'T LIKE THAT. >>> >>> >>> >>> ______________________________**_________________ >>> >> rsyslog mailing list >>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< >>> http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> >> http://www.rsyslog.com/**professional-services/< >>> http://www.rsyslog.com/professional-services/> >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> >> DON'T LIKE THAT. >>> >> >>> >> ______________________________**_________________ >>> > rsyslog mailing list >>> > http://lists.adiscon.net/**mailman/listinfo/rsyslog< >>> http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> > http://www.rsyslog.com/**professional-services/< >>> http://www.rsyslog.com/professional-services/> >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> > DON'T LIKE THAT. >>> > >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
