More fun info: $ sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 50 | grep deleted rsyslogd 11331 root 20w REG 9,2 117175430149 8847368 /var/log/joe/CISCO/app7/logfile.1 (deleted) rsyslogd 11331 root 9w REG 9,2 53533745904 8839187 /var/log/joe/Microsoft/app1/logfile.1 (deleted) rsyslogd 11331 root 10w REG 9,2 16268001760 8839237 /var/log/joe/Microsoft/app2/logfile.1 (deleted) rsyslogd 11331 root 18w REG 9,2 1831944964 8847369 /var/log/joe/Microsoft/app3/logfile.1 (deleted)
On Tue, Jun 25, 2013 at 12:10 PM, Xuri Nagarin <[email protected]> wrote: > Happening again: > > $ sudo du -sh * > 9.8M bin > 26M boot > 220K dev > 6.9M etc > 3.5G home > 129M lib > 22M lib64 > 20K lost+found > 8.0K media > 8.0K mnt > 6.4G opt > du: cannot access `proc/2713/task/9576': No such file or directory > du: cannot access `proc/9490/task/9490/fd/4': No such file or directory > du: cannot access `proc/9490/task/9490/fdinfo/4': No such file or directory > du: cannot access `proc/9490/fd/4': No such file or directory > du: cannot access `proc/9490/fdinfo/4': No such file or directory > 0 proc > 116K root > 13M sbin > 4.0K selinux > 4.0K srv > 0 sys > 296K tmp > 1.3G usr > 39G var > > $ sudo df -kh > Filesystem Size Used Avail Use% Mounted on > /dev/md2 913G 212G 655G 25% / > tmpfs 16G 0 16G 0% /dev/shm > /dev/md0 97M 31M 62M 34% /boot > > I am seeing the same baheviour on a second system. > > > > > > > > On Tue, Jun 25, 2013 at 12:04 PM, Xuri Nagarin <[email protected]> wrote: > >> My suspicion was something to do with dynamic file creation but looking >> at the code, both static and dynamic file creation use the same function of >> file "open": >> >> From tools/omfile.c >> ------------xxxxxxxxxxxxxxxxx--------------- >> fd = open((char*) newFileName, >> O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, >> 556 pData->fCreateMode); >> ------------xxxxxxxxxxxxxxxxx--------------- >> >> "O_APPEND" should take care of file being rotated while rsyslog is trying >> to write to it. >> >> >> >> >> On Tue, Jun 25, 2013 at 11:47 AM, Xuri Nagarin <[email protected]> wrote: >> >>> Not a typo or error, in RHEL the rsyslogd start up script in init.d has >>> an explicit variable "PIDFILE" set as: >>> PIDFILE=/var/run/syslogd.pid >>> >>> :) >>> >>> I did not write the script, using whatever was bundled in the RPM I >>> grabbed from Adiscon. >>> >>> To confirm: >>> $ ps ax | grep rsyslog >>> 11331 ? Sl 61:18 /sbin/rsyslogd -i /var/run/syslogd.pid -x >>> >>> >>> >>> >>> >>> On Tue, Jun 25, 2013 at 11:41 AM, Soham Chakraborty < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> In the logrotate config, change it to /var/run/rsyslog.pid and test. On >>>> a >>>> quick skim, it looks like a typo. >>>> >>>> Soham >>>> >>>> >>>> On Tue, Jun 25, 2013 at 11:07 PM, David Lang <[email protected]> wrote: >>>> >>>> > well, one thing that looks wrong is that logrotate is looking for >>>> > /var/run/syslog.pid, but on my systems the pid is in >>>> /var/log/rsyslog.pid >>>> > >>>> > are you sure that rsyslog is actually getting the HUP? >>>> > >>>> > can you try sending it a HUP manually and see if it closes the files? >>>> > >>>> > David Lang >>>> > >>>> > >>>> > On Tue, 25 Jun 2013, Xuri Nagarin wrote: >>>> > >>>> > Date: Tue, 25 Jun 2013 11:34:08 -0700 >>>> >> From: Xuri Nagarin <[email protected]> >>>> >> Reply-To: rsyslog-users <[email protected]> >>>> >> To: rsyslog-users <[email protected]> >>>> >> Subject: Re: [rsyslog] HUP-ing rsyslog does not free up disk space >>>> >> >>>> >> >>>> >> Hi David, >>>> >> >>>> >> The master conf file is simple with few additions that are >>>> >> self-explanatory. The other conf in rsyslog.d, I have added comments >>>> >> inline >>>> >> in the conf that explain the config logic. >>>> >> >>>> >> >>>> >> /etc/rsyslog.conf >>>> >> ==============================**==============================** >>>> >> =========== >>>> >> module(load="impstats" interval="600" severity="7" >>>> >> >>>> >> log.syslog="off" /* need to turn log stream logging off! */ >>>> >> log.file="/var/log/rsyslog-**stats.log") >>>> >> >>>> >> $ModLoad imuxsock.so # Unix sockets >>>> >> $ModLoad imklog.so # Kernel logger >>>> >> $MainMsgQueueSize 1000000 >>>> >> $OMFileIOBufferSize 512k >>>> >> $MaxMessageSize 8k >>>> >> $MainMsgQueueWorkerThreads 64 >>>> >> $umask 0000 >>>> >> $FileOwner joe >>>> >> $FileGroup joe >>>> >> $DirOwner joe >>>> >> $DirGroup joe >>>> >> $DirCreateMode 0755 >>>> >> $FileCreateMode 0644 >>>> >> >>>> >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >>>> >> $IncludeConfig /etc/rsyslog.d/*.conf >>>> >> >>>> >> $RuleSet local >>>> >> *.info;mail.none;authpriv.**none;cron.none >>>> >> /var/log/messages >>>> >> authpriv.* >>>> /var/log/secure >>>> >> mail.* >>>> -/var/log/maillog >>>> >> cron.* /var/log/cron >>>> >> *.emerg * >>>> >> uucp,news.crit >>>> /var/log/spooler >>>> >> local7.* >>>> /var/log/boot.log >>>> >> $DefaultRuleset local >>>> >> ==============================**==============================** >>>> >> =========== >>>> >> >>>> >> /etc/rsyslog.d/cef.conf >>>> >> ==============================**==============================** >>>> >> =========== >>>> >> # Senders are four Arcsight Logger devices that send logs in CEF >>>> format >>>> >> >>>> >> #template for writing CEF formatted logs >>>> >> template(name="cefdynfile" type="string" >>>> >> string="/var/log/joe/%$!**vendor%/%$!product%/logfile") >>>> >> >>>> >> #template for writing logs from non-CEF sources >>>> >> template(name="noncefdynfile" type="string" >>>> >> string="/var/log/joe/noncef/%**hostname%/%programname%/**logfile") >>>> >> >>>> >> ruleset(name="tcpcef") { >>>> >> >>>> >> #CEF uses the pipe delimiter, fields 2 and 3 are product vendor and >>>> >> product >>>> >> type respectively >>>> >> set $!vendor = field($msg, 124, 2); >>>> >> set $!product = field($msg, 124, 3); >>>> >> >>>> >> # Rules to write CEF formatted logs to disk and send logs by app >>>> type to >>>> >> their flume destinations that are listening on the same box >>>> >> >>>> >> if $syslogtag=="CEF:" then { action (type="omfile" ASyncWriting="on" >>>> >> IOBufferSize="8192K" FileOwner="joe" FileGroup="joe" DirOwner="joe" >>>> >> DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644" >>>> >> DynaFile="cefdynfile") } >>>> >> if $!product == "app1" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5161" Protocol="tcp") stop } >>>> >> if $!product == "app2" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5146" Protocol="tcp") stop } >>>> >> if $!product == "app3" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5172" Protocol="tcp") stop } >>>> >> if $!product == "app4" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5162" Protocol="tcp") stop } >>>> >> if $!product == "app5" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5166" Protocol="tcp") stop } >>>> >> if $!product == "app6" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5163" Protocol="tcp") stop } >>>> >> if $!product == "app7" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5164" Protocol="tcp") stop } >>>> >> if $!product == "app8" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5177" Protocol="tcp") stop } >>>> >> if $!product == "app9" then { action (type="omfwd" Target="127.0.0.1" >>>> >> Port="5144" Protocol="tcp") stop } >>>> >> if $!product == "app10" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5145" Protocol="tcp") stop } >>>> >> if $!product == "app11" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5148" Protocol="tcp") stop } >>>> >> if $!product == "app12" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5180" Protocol="tcp") stop } >>>> >> if $!product == "app13" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5147" Protocol="tcp") stop } >>>> >> if $!product == "app14" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5149" Protocol="tcp") stop } >>>> >> if $!product == "app15" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5150" Protocol="tcp") stop } >>>> >> if $!product == "app16" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5151" Protocol="tcp") stop } >>>> >> if $!product == "app17" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5152" Protocol="tcp") stop } >>>> >> if $!product == "app18" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5153" Protocol="tcp") stop } >>>> >> if $!product == "app19" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5155" Protocol="tcp") stop } >>>> >> if $!product == "app20" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5156" Protocol="tcp") stop } >>>> >> if $!product == "app21" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5157" Protocol="tcp") stop } >>>> >> if $!product == "app22" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5158" Protocol="tcp") stop } >>>> >> if $!product == "app23" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5159" Protocol="tcp") stop } >>>> >> if $!product == "app24" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5160" Protocol="tcp") stop } >>>> >> if $!product == "app25" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5178" Protocol="tcp") stop } >>>> >> if $!product == "app26" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5165" Protocol="tcp") stop } >>>> >> if $!product == "app27" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5165" Protocol="tcp") stop } >>>> >> if $!product == "app28" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5167" Protocol="tcp") stop } >>>> >> if $!product == "app29" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5167" Protocol="tcp") stop } >>>> >> if $!product == "app30" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5179" Protocol="tcp") stop } >>>> >> if $!product == "app31" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5169" Protocol="tcp") stop } >>>> >> if $!product == "app32" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5170" Protocol="tcp") stop } >>>> >> if $!product == "app33" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5171" Protocol="tcp") stop } >>>> >> if $!product == "app34" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5174" Protocol="tcp") stop } >>>> >> if $!product == "app35" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5173" Protocol="tcp") stop } >>>> >> if $!product == "app36" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5175" Protocol="tcp") stop } >>>> >> if $!product == "app37" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5176" Protocol="tcp") stop } >>>> >> if $!product == "app38" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5154" Protocol="tcp") stop } >>>> >> if $!product == "app39" then { action (type="omfwd" >>>> Target="127.0.0.1" >>>> >> Port="5181" Protocol="tcp") stop } >>>> >> >>>> >> # Unfortunately, the four Arcsight Loggers also send us garbage so >>>> >> whatever >>>> >> could not be parsed/classified correctly by the rules above, gets >>>> parked >>>> >> in >>>> >> a catchall file. >>>> >> >>>> >> if $fromhost-ip == '10.1.1.100' or $fromhost-ip == '10.1.1.101' or >>>> >> $fromhost-ip == '10.1.1.102' or $fromhost-ip == '10.1.1.103' then { >>>> action >>>> >> (type="omfile" FileOwner="joe" FileGroup="joe" DirOwner="joe" >>>> >> DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644" >>>> >> file="/var/log/joe/fallback/**logfile") >>>> >> & action (type="omfwd" Target="127.0.0.1" Port="5182" >>>> Protocol="tcp" >>>> >> ) >>>> >> stop } >>>> >> >>>> >> # Take care of all the non-CEF / BSD Syslog formatted streams coming >>>> in >>>> >> else { >>>> >> action (type="omfile" ASyncWriting="on" IOBufferSize="8192K" >>>> >> FileOwner="joe" FileGroup="joe" DirOwner="joe" DirGroup="joe" >>>> >> DirCreateMode="0755" FileCreateMode="0644" DynaFile="noncefdynfile") >>>> >> & action (type="omfwd" Target="127.0.0.1" Port="5182" >>>> Protocol="tcp") >>>> >> } >>>> >> } >>>> >> >>>> >> module(load="imtcp" ) # needs to be done just once >>>> >> input(type="imtcp" port="514" ruleset="tcpcef") >>>> >> ==============================**==============================** >>>> >> =========== >>>> >> >>>> >> >>>> >> Here's the logrotate code: >>>> >> ==============================**==============================** >>>> >> =========== >>>> >> /var/log/joe/*/*/* >>>> >> /var/log/joe/*/*/*/* >>>> >> { >>>> >> missingok >>>> >> size 1G >>>> >> rotate 0 >>>> >> sharedscripts >>>> >> postrotate >>>> >> if [ -f /var/run/syslog.pid ]; then \ >>>> >> kill -HUP `cat /var/run/syslog.pid`; \ >>>> >> fi; >>>> >> endscript >>>> >> } >>>> >> ==============================**==============================** >>>> >> =========== >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> On Tue, Jun 25, 2013 at 10:12 AM, David Lang <[email protected]> wrote: >>>> >> >>>> >> On Tue, 25 Jun 2013, Xuri Nagarin wrote: >>>> >>> >>>> >>> On RHEL 6.2 64-bit, I have Rsyslog 7.4.1 (actually, the issue has >>>> >>> existed >>>> >>> >>>> >>>> for earlier versions too). >>>> >>>> >>>> >>>> After logrotation, the logrotate script HUPs the rsyslogd pid but >>>> the >>>> >>>> disk >>>> >>>> space doesn't free up until I restart rsyslog. After a few hours >>>> "df" >>>> >>>> reports a full filesystem whereas "ls" shows much smaller file >>>> sizes. In >>>> >>>> between this confusion, rsyslog seems to stop receiving log >>>> streams. >>>> >>>> >>>> >>>> Is there a way for rsyslog to switch file handles when it is >>>> HUP-ed? >>>> >>>> >>>> >>>> >>>> >>> rsyslog is already supposed to close and re-open files when it's >>>> HUP-ed, >>>> >>> and it seems to be working for me and many others. >>>> >>> >>>> >>> could you send your config file so we can see if there is anything >>>> >>> unusual >>>> >>> in it? >>>> >>> >>>> >>> David Lang >>>> >>> ______________________________****_________________ >>>> >>> rsyslog mailing list >>>> >>> http://lists.adiscon.net/****mailman/listinfo/rsyslog< >>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>> >>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog< >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>> >>> > >>>> >>> http://www.rsyslog.com/****professional-services/< >>>> http://www.rsyslog.com/**professional-services/> >>>> >>> <http://**www.rsyslog.com/professional-**services/< >>>> http://www.rsyslog.com/professional-services/> >>>> >>> > >>>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad >>>> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you >>>> >>> DON'T LIKE THAT. >>>> >>> >>>> >>> ______________________________**_________________ >>>> >> rsyslog mailing list >>>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>> >> http://www.rsyslog.com/**professional-services/< >>>> http://www.rsyslog.com/professional-services/> >>>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad >>>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you >>>> >> DON'T LIKE THAT. >>>> >> >>>> >> ______________________________**_________________ >>>> > rsyslog mailing list >>>> > http://lists.adiscon.net/**mailman/listinfo/rsyslog< >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>> > http://www.rsyslog.com/**professional-services/< >>>> http://www.rsyslog.com/professional-services/> >>>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad >>>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> > DON'T LIKE THAT. >>>> > >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you DON'T LIKE THAT. >>>> >>> >>> >> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
