Re: [rsyslog] HUP-ing rsyslog does not free up disk space

David Lang Tue, 25 Jun 2013 12:52:50 -0700

On Tue, 25 Jun 2013, Xuri Nagarin wrote:

Yes, a manual HUP released them all but not instantaneously. Took a few
seconds.

given how many files you have open, It's not too surprising that it takes alittle time

A logrotate bug?

double check the contents (and timestamp) of /var/run/syslog.pid, it may not bewhat you are expecting

since the HUP does free the files, my suspicion is that rsyslog is not gettingthe HUP in the first place. This would point me at logrotate, or the way thatlogrotate is getting the PID


David Lang

On Tue, Jun 25, 2013 at 11:38 AM, David Lang <[email protected]> wrote:

If you manually do a kill -HUP 11311 do these files get released?


David Lang

On Tue, 25 Jun 2013, Xuri Nagarin wrote:

 More fun info:


$ sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 50 | grep
deleted
rsyslogd  11331      root   20w      REG                9,2 117175430149
8847368 /var/log/joe/CISCO/app7/**logfile.1 (deleted)
rsyslogd  11331      root    9w      REG                9,2  53533745904
8839187 /var/log/joe/Microsoft/app1/**logfile.1 (deleted)
rsyslogd  11331      root   10w      REG                9,2  16268001760
8839237 /var/log/joe/Microsoft/app2/**logfile.1 (deleted)
rsyslogd  11331      root   18w      REG                9,2   1831944964
8847369 /var/log/joe/Microsoft/app3/**logfile.1 (deleted)




On Tue, Jun 25, 2013 at 12:10 PM, Xuri Nagarin <[email protected]> wrote:

 Happening again:


$ sudo du -sh *
9.8M bin
26M boot
220K dev
6.9M etc
3.5G home
129M lib
22M lib64
20K lost+found
8.0K media
8.0K mnt
6.4G opt
du: cannot access `proc/2713/task/9576': No such file or directory
du: cannot access `proc/9490/task/9490/fd/4': No such file or directory
du: cannot access `proc/9490/task/9490/fdinfo/4'**: No such file or
directory
du: cannot access `proc/9490/fd/4': No such file or directory
du: cannot access `proc/9490/fdinfo/4': No such file or directory
0 proc
116K root
13M sbin
4.0K selinux
4.0K srv
0 sys
296K tmp
1.3G usr
39G var

$ sudo df -kh
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              913G  212G  655G  25% /
tmpfs                  16G     0   16G   0% /dev/shm
/dev/md0               97M   31M   62M  34% /boot

I am seeing the same baheviour on a second system.







On Tue, Jun 25, 2013 at 12:04 PM, Xuri Nagarin <[email protected]>
wrote:

 My suspicion was something to do with dynamic file creation but looking

at the code, both static and dynamic file creation use the same
function of
file "open":

From tools/omfile.c
------------xxxxxxxxxxxxxxxxx-**--------------
fd = open((char*) newFileName,
O_WRONLY|O_APPEND|O_CREAT|O_**NOCTTY|O_CLOEXEC,
 556                                 pData->fCreateMode);
------------xxxxxxxxxxxxxxxxx-**--------------

"O_APPEND" should take care of file being rotated while rsyslog is
trying
to write to it.




On Tue, Jun 25, 2013 at 11:47 AM, Xuri Nagarin <[email protected]>
wrote:

 Not a typo or error, in RHEL the rsyslogd start up script in init.d has

an explicit variable "PIDFILE" set as:
PIDFILE=/var/run/syslogd.pid

:)

I did not write the script, using whatever was bundled in the RPM I
grabbed from Adiscon.

To confirm:
$ ps ax | grep rsyslog
11331 ?        Sl    61:18 /sbin/rsyslogd -i /var/run/syslogd.pid -x





On Tue, Jun 25, 2013 at 11:41 AM, Soham Chakraborty <
[email protected]> wrote:

 Hi,


In the logrotate config, change it to /var/run/rsyslog.pid and test.
On
a
quick skim, it looks like a typo.

Soham


On Tue, Jun 25, 2013 at 11:07 PM, David Lang <[email protected]> wrote:

 well, one thing that looks wrong is that logrotate is looking for

/var/run/syslog.pid, but on my systems the pid is in

/var/log/rsyslog.pid


are you sure that rsyslog is actually getting the HUP?

can you try sending it a HUP manually and see if it closes the files?

David Lang


On Tue, 25 Jun 2013, Xuri Nagarin wrote:

 Date: Tue, 25 Jun 2013 11:34:08 -0700

From: Xuri Nagarin <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] HUP-ing rsyslog does not free up disk space


Hi David,

The master conf file is simple with few additions that are
self-explanatory. The other conf in rsyslog.d, I have added comments
inline
in the conf that explain the config logic.


/etc/rsyslog.conf
==============================****============================**
==**
===========
module(load="impstats" interval="600" severity="7"

log.syslog="off" /* need to turn log stream logging off! */
log.file="/var/log/rsyslog-****stats.log")

$ModLoad imuxsock.so    # Unix sockets
$ModLoad imklog.so      # Kernel logger
$MainMsgQueueSize 1000000
$OMFileIOBufferSize 512k
$MaxMessageSize 8k
$MainMsgQueueWorkerThreads 64
$umask 0000
$FileOwner joe
$FileGroup joe
$DirOwner joe
$DirGroup joe
$DirCreateMode 0755
$FileCreateMode 0644

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf

$RuleSet local
*.info;mail.none;authpriv.****none;cron.none
 /var/log/messages
authpriv.*

/var/log/secure

mail.*

 -/var/log/maillog

cron.*                                                  /var/log/cron

*.emerg                                                 *
uucp,news.crit

 /var/log/spooler

local7.*

 /var/log/boot.log

$DefaultRuleset local

==============================****============================**
==**
===========

/etc/rsyslog.d/cef.conf
==============================****============================**
==**
===========
# Senders are four Arcsight Logger devices that send logs in CEF

format

#template for writing CEF formatted logs
template(name="cefdynfile" type="string"
string="/var/log/joe/%$!****vendor%/%$!product%/logfile")

#template for writing logs from non-CEF sources
template(name="noncefdynfile" type="string"
string="/var/log/joe/noncef/%****hostname%/%programname%/****
logfile")

ruleset(name="tcpcef") {

#CEF uses the pipe delimiter, fields 2 and 3 are product vendor and
product
type respectively
set $!vendor = field($msg, 124, 2);
set $!product = field($msg, 124, 3);

# Rules to write CEF formatted logs to disk and send logs by app

type to

their flume destinations that are listening on the same box


if $syslogtag=="CEF:" then { action (type="omfile" ASyncWriting="on"
IOBufferSize="8192K" FileOwner="joe" FileGroup="joe" DirOwner="joe"
DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644"
DynaFile="cefdynfile")  }
if $!product == "app1" then { action (type="omfwd"
Target="127.0.0.1"
Port="5161" Protocol="tcp") stop }
if $!product == "app2" then { action (type="omfwd"
Target="127.0.0.1"
Port="5146" Protocol="tcp") stop }
if $!product == "app3" then { action (type="omfwd"
Target="127.0.0.1"
Port="5172" Protocol="tcp") stop }
if $!product == "app4" then { action (type="omfwd"
Target="127.0.0.1"
Port="5162" Protocol="tcp") stop }
if $!product == "app5" then { action (type="omfwd"
Target="127.0.0.1"
Port="5166" Protocol="tcp") stop }
if $!product == "app6" then { action (type="omfwd"
Target="127.0.0.1"
Port="5163" Protocol="tcp") stop }
if $!product == "app7" then { action (type="omfwd"
Target="127.0.0.1"
Port="5164" Protocol="tcp") stop }
if $!product == "app8" then { action (type="omfwd"
Target="127.0.0.1"
Port="5177" Protocol="tcp") stop }
if $!product == "app9" then { action (type="omfwd"
Target="127.0.0.1"
Port="5144" Protocol="tcp") stop }
if $!product == "app10" then { action (type="omfwd"

Target="127.0.0.1"

Port="5145" Protocol="tcp") stop }

if $!product == "app11" then { action (type="omfwd"

Target="127.0.0.1"

Port="5148" Protocol="tcp") stop }

if $!product == "app12" then { action (type="omfwd"

Target="127.0.0.1"

Port="5180" Protocol="tcp") stop }

if $!product == "app13" then { action (type="omfwd"

Target="127.0.0.1"

Port="5147" Protocol="tcp") stop }

if $!product == "app14" then { action (type="omfwd"

Target="127.0.0.1"

Port="5149" Protocol="tcp") stop }

if $!product == "app15" then { action (type="omfwd"

Target="127.0.0.1"

Port="5150" Protocol="tcp") stop }

if $!product == "app16" then { action (type="omfwd"

Target="127.0.0.1"

Port="5151" Protocol="tcp") stop }

if $!product == "app17" then { action (type="omfwd"

Target="127.0.0.1"

Port="5152" Protocol="tcp") stop }

if $!product == "app18" then { action (type="omfwd"

Target="127.0.0.1"

Port="5153" Protocol="tcp") stop }

if $!product == "app19" then { action (type="omfwd"

Target="127.0.0.1"

Port="5155" Protocol="tcp") stop }

if $!product == "app20" then { action (type="omfwd"

Target="127.0.0.1"

Port="5156" Protocol="tcp") stop }

if $!product == "app21" then { action (type="omfwd"

Target="127.0.0.1"

Port="5157" Protocol="tcp") stop }

if $!product == "app22" then { action (type="omfwd"

Target="127.0.0.1"

Port="5158" Protocol="tcp") stop }

if $!product == "app23" then { action (type="omfwd"

Target="127.0.0.1"

Port="5159" Protocol="tcp") stop }

if $!product == "app24" then { action (type="omfwd"

Target="127.0.0.1"

Port="5160" Protocol="tcp") stop }

if $!product == "app25" then { action (type="omfwd"

Target="127.0.0.1"

Port="5178" Protocol="tcp") stop }

if $!product == "app26" then { action (type="omfwd"

Target="127.0.0.1"

Port="5165" Protocol="tcp") stop }

if $!product == "app27" then { action (type="omfwd"

Target="127.0.0.1"

Port="5165" Protocol="tcp") stop }

if $!product == "app28" then { action (type="omfwd"

Target="127.0.0.1"

Port="5167" Protocol="tcp") stop }

if $!product == "app29" then { action (type="omfwd"

Target="127.0.0.1"

Port="5167" Protocol="tcp") stop }

if $!product == "app30" then { action (type="omfwd"

Target="127.0.0.1"

Port="5179" Protocol="tcp") stop }

if $!product == "app31" then { action (type="omfwd"

Target="127.0.0.1"

Port="5169" Protocol="tcp") stop }

if $!product == "app32" then { action (type="omfwd"

Target="127.0.0.1"

Port="5170" Protocol="tcp") stop }

if $!product == "app33" then { action (type="omfwd"

Target="127.0.0.1"

Port="5171" Protocol="tcp") stop }

if $!product == "app34" then { action (type="omfwd"

Target="127.0.0.1"

Port="5174" Protocol="tcp") stop }

if $!product == "app35" then { action (type="omfwd"

Target="127.0.0.1"

Port="5173" Protocol="tcp") stop }

if $!product == "app36" then { action (type="omfwd"

Target="127.0.0.1"

Port="5175" Protocol="tcp") stop }

if $!product == "app37" then { action (type="omfwd"

Target="127.0.0.1"

Port="5176" Protocol="tcp") stop }

if $!product == "app38" then { action (type="omfwd"

Target="127.0.0.1"

Port="5154" Protocol="tcp") stop }

if $!product == "app39" then { action (type="omfwd"

Target="127.0.0.1"

Port="5181" Protocol="tcp") stop }


# Unfortunately, the four Arcsight Loggers also send us garbage so
whatever
could not be parsed/classified correctly by the rules above, gets

parked

in

a catchall file.

if $fromhost-ip == '10.1.1.100' or $fromhost-ip == '10.1.1.101' or
$fromhost-ip == '10.1.1.102' or $fromhost-ip == '10.1.1.103' then {

action

(type="omfile" FileOwner="joe" FileGroup="joe" DirOwner="joe"

DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644"
file="/var/log/joe/fallback/****logfile")
     & action (type="omfwd" Target="127.0.0.1" Port="5182"

Protocol="tcp"

stop  }

# Take care of all the non-CEF / BSD Syslog formatted streams coming

in

else {

action (type="omfile" ASyncWriting="on" IOBufferSize="8192K"
FileOwner="joe" FileGroup="joe" DirOwner="joe" DirGroup="joe"
DirCreateMode="0755" FileCreateMode="0644" DynaFile="noncefdynfile")
     & action (type="omfwd" Target="127.0.0.1" Port="5182"

Protocol="tcp")

}

module(load="imtcp" ) # needs to be done just once
input(type="imtcp" port="514" ruleset="tcpcef")
==============================****============================**
==**
===========


Here's the logrotate code:
==============================****============================**
==**
===========
/var/log/joe/*/*/*
/var/log/joe/*/*/*/*
{
missingok
size 1G
rotate 0
sharedscripts
postrotate
if [ -f /var/run/syslog.pid ]; then \
kill -HUP `cat /var/run/syslog.pid`; \
fi;
endscript
}
==============================****============================**
==**
===========






On Tue, Jun 25, 2013 at 10:12 AM, David Lang <[email protected]> wrote:

 On Tue, 25 Jun 2013, Xuri Nagarin wrote:


 On RHEL 6.2 64-bit, I have Rsyslog 7.4.1 (actually, the issue has
existed

 for earlier versions too).


After logrotation, the logrotate script HUPs the rsyslogd pid but

the

disk

space doesn't free up until I restart rsyslog. After a few hours

"df"

reports a full filesystem whereas "ls" shows much smaller file

sizes. In

between this confusion, rsyslog seems to stop receiving log

streams.

Is there a way for rsyslog to switch file handles when it is

HUP-ed?


 rsyslog is already supposed to close and re-open files when it's

HUP-ed,

and it seems to be working for me and many others.


could you send your config file so we can see if there is anything
unusual
in it?

David Lang
______________________________******_________________
rsyslog mailing list
http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
<

http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>

<http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>

http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>

 
http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>

http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>

<http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>

http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>

 What's up with rsyslog? Follow https://twitter.com/rgerhards

NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

DON'T LIKE THAT.


 ______________________________****_________________

rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
<

http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>

http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>

http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>

What's up with rsyslog? Follow https://twitter.com/rgerhards

NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if

you

DON'T LIKE THAT.


 ______________________________****_________________

rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
<

http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>

http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
<

http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>

What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.

 ______________________________**_________________

rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if
you DON'T LIKE THAT.

 ______________________________**_________________

rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

 ______________________________**_________________

rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] HUP-ing rsyslog does not free up disk space

Reply via email to