Hello,
I'm using central logging using RELP+TLS with disk assisted queues on the
client. When I:
1) stop rsyslog on the server (or just drop all packets at the firewall to
simulate network loss)
2) send a few messages to syslog with "logger" on the client (rsyslog
notices the relp server is down and puts the messages in the queue)
3) then restart rsyslog on the client
the first message (sometimes I've noticed two) sent to logger while the
server was unreachable is lost and does not get sent to the central server.
If I *do not* restart rsyslog on the client while the central server is
unreachable, the message is not lost. This happens with both LinkedList and
Disk queues.
It's not uncommon that the client might be restarted during a time when it
has already lost the network connection to the central server. Am I missing
a config that might fix this?
Here are my example logs showing the message loss:
I start off with both central server and rsyslog client running normally
with RELP working fine:
============================================
root@rsyslog-test:/etc# logger upnow
root@rsyslog-test:/etc# service rsyslog restart
rsyslog stop/waiting
rsyslog start/running, process 23909
root@rsyslog-test:/etc# logger upnow2
Works fine. Central server receives both messages and sees the client
restart:
============================================
root@relp-server /etc # tail /var/log/relp_log
2013-07-19T17:07:55.757080-04:00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:08:15.894360-04:00 rsyslog-test logger: upnow2
Now I stop the central log server:
============================================
root@relp-server /etc # service rsyslog stop
rsyslog stop/waiting
Log some messages to syslog on the client with "logger" and stop rsyslog on
the client:
============================================
root@rsyslog-test:/etc# logger downnow1
root@rsyslog-test:/etc# logger downnow2
root@rsyslog-test:/etc# logger downnow3
root@rsyslog-test:/etc# service rsyslog stop
rsyslog stop/waiting
Since I have all messages being logged locally on the client I can see that
my messages were sent to syslog and syslog detected the central server was
down after I logged the 'downnow2' message:
============================================
root@rsyslog-test:/etc# tail /var/log/rsyslog-test
2013-07-19T17:07:20.413878-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:07:55.757080-04:00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:08:15.894360-04:00 rsyslog-test logger: upnow2
2013-07-19T17:08:31.045190-04:00 rsyslog-test logger: downnow1
2013-07-19T17:08:48.446359-04:00 rsyslog-test logger: downnow2
2013-07-19T17:08:48.446601-04:00 rsyslog-test rsyslogd-2353:
omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error -53:
Error in the push function.]', object 'conn to srvr 10.10.5.1:20514' -
action may not work as intended [try http://www.rsyslog.com/e/2353 ]
2013-07-19T17:08:54.453092-04:00 rsyslog-test logger: downnow3
2013-07-19T17:09:13.302262-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] exiting on signal 15.
I can see the messages except the first 'downnow1' are in the disk queue
file on the client:
============================================
root@rsyslog-test:/etc# cat /var/spool/rsyslog/queue/relpy.00000001
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:3:
+iFacility:2:1:5:
+msgFlags:2:1:1:
+ttGenTime:2:10:1374268128:
+tRcvdAt:3:34:2:2013:7:19:17:8:48:446601:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:8:48:446601:6:-:4:0:
+pszTAG:1:14:rsyslogd-2353::
+pszRawMsg:1:215:omrelp[10.10.5.1:20514]: error 'TLS record write failed
[gnutls error -53: Error in the push function.]', object 'conn to srvr
10.10.5.1:20514' - action may not work as intended [try
http://www.rsyslog.com/e/2353 ]:
+pszHOSTNAME:1:17:rsyslog-test:
+pszInputName:1:8:rsyslogd:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:1:0:
>End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:5:
+iFacility:2:1:1:
+msgFlags:2:1:4:
+ttGenTime:2:10:1374268134:
+tRcvdAt:3:34:2:2013:7:19:17:8:54:453092:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:8:54:453092:6:-:4:0:
+pszTAG:1:7:logger::
+pszRawMsg:1:36:<13>Jul 19 17:08:54 logger: downnow3:
+pszInputName:1:8:imuxsock:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:2:27:
>End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:6:
+iFacility:2:1:5:
+msgFlags:2:1:1:
+ttGenTime:2:10:1374268153:
+tRcvdAt:3:34:2:2013:7:19:17:9:13:302262:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:9:13:302262:6:-:4:0:
+pszTAG:1:9:rsyslogd::
+pszRawMsg:1:115: [origin software="rsyslogd" swVersion="7.5.2"
x-pid="23909" x-info="http://www.rsyslog.com";] exiting on signal 15.:
+pszHOSTNAME:1:17:rsyslog-test:
+pszInputName:1:8:rsyslogd:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:1:0:
>End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:5:
+iFacility:2:1:1:
+msgFlags:2:1:4:
+ttGenTime:2:10:1374268128:
+tRcvdAt:3:34:2:2013:7:19:17:8:48:446359:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:8:48:446359:6:-:4:0:
+pszTAG:1:7:logger::
+pszRawMsg:1:36:<13>Jul 19 17:08:48 logger: downnow2:
+pszInputName:1:8:imuxsock:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:2:27:
>End
.
Now I restart rsyslog on the central server and the client, and log a few
more messages on the client:
============================================
root@relp-server /etc # service rsyslog start
rsyslog start/running, process 331
============================================
root@rsyslog-test:/etc# service rsyslog start
rsyslog start/running, process 23971
root@rsyslog-test:/etc# logger upagain
root@rsyslog-test:/etc# logger upagain2
root@rsyslog-test:/etc# logger upagain3
I check the central server logs and all messages except the lost "downnow1"
message are successfully received:
============================================
root@relp-server /etc # tail -n 20 /var/log/relp_log
2013-07-19T17:07:55.757080-04:00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:08:15.894360-04:00 rsyslog-test logger: upnow2
2013-07-19T17:08:48.446601-04:00 rsyslog-test rsyslogd-2353:
omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error -53:
Error in the push function.]', object 'conn to srvr 10.10.5.1:20514' -
action may not work as intended [try http://www.rsyslog.com/e/2353 ]
2013-07-19T17:08:54.453092-04:00 rsyslog-test logger: downnow3
2013-07-19T17:09:13.302262-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:48.446359-04:00 rsyslog-test logger: downnow2
2013-07-19T17:10:15.245473-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23971"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:10:35.210411-04:00 rsyslog-test logger: upagain
2013-07-19T17:10:41.354935-04:00 rsyslog-test logger: upagain2
2013-07-19T17:10:48.602667-04:00 rsyslog-test logger: upagain3
Here is the relp server config:
============================================
root@relp-server /etc # cat rsyslog.conf
$PrivDropToUser syslog
$PrivDropToGroup syslog
$AbortOnUncleanConfig off
$umask 0000 # Ensure umask is reset
$DirCreateMode 0700
$FileCreateMode 0644
$DirOwner syslog
$DirGroup syslog
# DynaFile settings
$FileOwner syslog
$FileGroup syslog
$FailOnChownFailure on
$DynaFileCacheSize 100 # a cache of 100 files at most
$MaxOpenFiles 200000
$MaxMessageSize 64k
$MainMsgQueueSize 100000
$DropTrailingLFOnReception on
$Escape8BitCharactersOnReceive off
$EscapeControlCharactersOnReceive on
$RepeatedMsgReduction off # log every message
$DropMsgsWithMaliciousDnsPTRRecords off
module(load="imuxsock")
module(load="imrelp" ruleset="relp")
input(type="imrelp" port="20514" tls="on"
tls.caCert="/etc/rsyslog.certs/ca.pem"
tls.myCert="/etc/rsyslog.certs/server.cert"
tls.myPrivKey="/etc/rsyslog.certs/server.key"
tls.authMode="name"
tls.permittedpeer=["rsyslog-test.mydomain.com"]
)
ruleset (name="relp") {
$RulesetCreateMainQueue on
action(type="omfile" file="/var/log/relp_log")
}
$IncludeConfig /etc/rsyslog.d/
*.* /var/log/syslog
And the client config:
============================================
root@rsyslog-test:/etc# cat rsyslog.conf
module(load="imuxsock")
module(load="omrelp")
module(load="imtcp")
input(type="imtcp" port="514")
$WorkDirectory /var/spool/rsyslog/queue # default location for work (spool)
files
$MaxMessageSize 64k
$MainMsgQueueFileName mainq # set file name, also enables disk mode
$MainMsgQueueType LinkedList
$MainMsgQueueSaveOnShutDown on
$MainMsgQueueMaxDiskSpace 40g
$MainMsgQueueSize 8000000
$ActionSendResendLastMsgOnReconnect on
action(type="omrelp" target="10.10.5.1" port="20514" tls="on"
tls.compression="on"
tls.caCert="/etc/rsyslog.certs/ca.pem"
tls.myCert="/etc/rsyslog.certs/rsyslog-test.cert"
tls.myPrivKey="/etc/rsyslog.certs/rsyslog-test.key"
tls.authmode="name"
tls.permittedpeer=["log.mydomain.com"]
queue.filename="relpy"
queue.maxdiskspace="10g"
queue.saveonshutdown="on"
queue.type="linkedlist"
queue.maxfilesize="20m"
action.resumeretrycount="-1"
)
*.* /var/log/rsyslog-test
--
Dave
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
