On Thu, 9 Jan 2020, Daniel Rubio via rsyslog wrote:
I've got 2 problems :)
1.- Yesterday, In one of our rsyslog servers, we started to see massive
too many open files messages:
Jan 8 21:05:39 logcenter rsyslogd: file
'/logs/2020/frontal/web/01/apacheaccess_08_S1.log': open error: Too many
open files [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/2433 ]
Jan 8 21:05:40 logcenter rsyslogd: file
'/logs/2020/munihosting/web/01/apacheaccess_08_S1.log': open error: Too
many open files [v8.24.0-41.el7_7.2 try http://www.rsyslog.com/e/2433 ]
...
This morning I restarted the rsyslog server and now it seems to be all
ok but... How can I raise this max value up? Is the default config for
the operating system (CentOs 7.6)? or there is something we have to
configure in rsyslog config?
this is a Centos 7 / systemd setting that you will have to change.
The second problem is related with this one...
2.-On my rsyslog clients I have this config for the output:
ruleset(name="sendToLogserver") {
action( type="omfwd" target="logcenter.intranet.dtgna" port="514"
protocol="tcp" queue.type="LinkedList" queue.size="500000"
queue.filename="q_sendToLogserver" queue.highwatermark="290000" queue.lowwa
termark="50000" queue.maxdiskspace="2g" queue.saveonshutdown="on"
action.resumeRetryCount="-1" action.resumeInterval="20")
action( type="omfwd" target="logcenter02.intranet.dtgna" port="514"
protocol="tcp" queue.type="LinkedList" queue.size="500000"
queue.filename="q_sendToLogserver02" queue.highwatermark="290000" queue.l
owwatermark="50000" queue.maxdiskspace="2g" queue.saveonshutdown="on"
action.resumeRetryCount="-1" action.resumeInterval="20"
action.execOnlyWhenPreviousIsSuspended="on")
stop
}
we would need to see the config on the receiving side as well (and may need to
see more of the config)
The questions are:
-If the primary rsyslog server was giving those open files errors, the
client shouldn't have to start send the messages to the secondary server?
that will only happen if the receiver blocks, which depends on how it's
configured.
even then, since you have a queue configured on each sender, it will only
failover to the second output if it can't deliver the message to the queue of
the first output (i.e. the queue completely fills up), since you have watermark
settings configured to throw away messages if the queue gets too full, it will
never queue up.
-What happened to the disappeared messages? how we could trace it? it's
a config problem?
it's not possible to trace a message after the fact (unless you have debug
logging turned out, but that is a huge performance hit and eats a LOT of space.
it does look like a config problem. if you are wanting to failover when a server
is down, you probably don't want to have a queue on each action, you probably
want a queue on the ruleset instead
if you are really wanting to avoid loosing logs, you don't want to set watermark
levels that will cause you to throw logs away, and you probably want to look at
relp rather than just TCP read
https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
-During the problem, the client server.log rotated, I suppose that there
wouldn't have to be a problem because those lines where in the rsyslog
queue, isn't it?
possibly, but it's possible that the logs were in-flight when a tcp connection
was lost, or dropped from the queue due to the watermark settings.
I'm very worried about those lines lost, It happended to various log
files :(
rsyslog doesn't do things for specific files, it works on the combined log
stream, so when logs are lost, they will not just be lost for one file
PS:Rsyslog version in server and clients is the last red-hat official
update, 8.24.0-41.el7_7.2
This is over 3 years old (but with some fixes backported by RedHat, but we don't
know which ones), so the community cannot support this version. But your issues
don't seem to be version related, just config related.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
