Hello,
I'm using central logging using RELP+TLS with disk assisted queues on
the client. When I:
1) stop rsyslog on the server (or just drop all packets at the
firewall to simulate network loss)
2) send a few messages to syslog with "logger" on the client (rsyslog
notices the relp server is down and puts the messages in the queue)
3) then restart rsyslog on the client
the first message (sometimes I've noticed two) sent to logger while
the server was unreachable is lost and does not get sent to the
central server. If I *do not* restart rsyslog on the client while the
central server is unreachable, the message is not lost. This happens
with both LinkedList and Disk queues.
It's not uncommon that the client might be restarted during a time
when it has already lost the network connection to the central server.
Am I missing a config that might fix this?
Here are my example logs showing the message loss:
I start off with both central server and rsyslog client running
normally with RELP working fine:
============================================
root@rsyslog-test:/etc# logger upnow
root@rsyslog-test:/etc# service rsyslog restart
rsyslog stop/waiting
rsyslog start/running, process 23909
root@rsyslog-test:/etc# logger upnow2
Works fine. Central server receives both messages and sees the client
restart:
============================================
root@relp-server /etc # tail /var/log/relp_log
2013-07-19T17:07:55.757080-04:00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:08:15.894360-04:00 rsyslog-test logger: upnow2
Now I stop the central log server:
============================================
root@relp-server /etc # service rsyslog stop
rsyslog stop/waiting
Log some messages to syslog on the client with "logger" and stop
rsyslog on the client:
============================================
root@rsyslog-test:/etc# logger downnow1
root@rsyslog-test:/etc# logger downnow2
root@rsyslog-test:/etc# logger downnow3
root@rsyslog-test:/etc# service rsyslog stop
rsyslog stop/waiting
Since I have all messages being logged locally on the client I can see
that my messages were sent to syslog and syslog detected the central
server was down after I logged the 'downnow2' message:
============================================
root@rsyslog-test:/etc# tail /var/log/rsyslog-test
2013-07-19T17:07:20.413878-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:07:55.757080-04:00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:08:15.894360-04:00 rsyslog-test logger: upnow2
2013-07-19T17:08:31.045190-04:00 rsyslog-test logger: downnow1
2013-07-19T17:08:48.446359-04:00 rsyslog-test logger: downnow2
2013-07-19T17:08:48.446601-04:00 rsyslog-test rsyslogd-2353:
omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error
-53: Error in the push function.]', object 'conn to srvr
10.10.5.1:20514' - action may not work as intended [try
http://www.rsyslog.com/e/2353 ]
2013-07-19T17:08:54.453092-04:00 rsyslog-test logger: downnow3
2013-07-19T17:09:13.302262-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] exiting on signal 15.
I can see the messages except the first 'downnow1' are in the disk
queue file on the client:
============================================
root@rsyslog-test:/etc# cat /var/spool/rsyslog/queue/relpy.00000001
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:3:
+iFacility:2:1:5:
+msgFlags:2:1:1:
+ttGenTime:2:10:1374268128:
+tRcvdAt:3:34:2:2013:7:19:17:8:48:446601:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:8:48:446601:6:-:4:0:
+pszTAG:1:14:rsyslogd-2353::
+pszRawMsg:1:215:omrelp[10.10.5.1:20514]: error 'TLS record write
failed [gnutls error -53: Error in the push function.]', object 'conn
to srvr 10.10.5.1:20514' - action may not work as intended [try
http://www.rsyslog.com/e/2353 ]:
+pszHOSTNAME:1:17:rsyslog-test:
+pszInputName:1:8:rsyslogd:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:1:0:
>End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:5:
+iFacility:2:1:1:
+msgFlags:2:1:4:
+ttGenTime:2:10:1374268134:
+tRcvdAt:3:34:2:2013:7:19:17:8:54:453092:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:8:54:453092:6:-:4:0:
+pszTAG:1:7:logger::
+pszRawMsg:1:36:<13>Jul 19 17:08:54 logger: downnow3:
+pszInputName:1:8:imuxsock:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:2:27:
>End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:6:
+iFacility:2:1:5:
+msgFlags:2:1:1:
+ttGenTime:2:10:1374268153:
+tRcvdAt:3:34:2:2013:7:19:17:9:13:302262:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:9:13:302262:6:-:4:0:
+pszTAG:1:9:rsyslogd::
+pszRawMsg:1:115: [origin software="rsyslogd" swVersion="7.5.2"
x-pid="23909" x-info="http://www.rsyslog.com";] exiting on signal 15.:
+pszHOSTNAME:1:17:rsyslog-test:
+pszInputName:1:8:rsyslogd:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:1:0:
>End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:5:
+iFacility:2:1:1:
+msgFlags:2:1:4:
+ttGenTime:2:10:1374268128:
+tRcvdAt:3:34:2:2013:7:19:17:8:48:446359:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:17:8:48:446359:6:-:4:0:
+pszTAG:1:7:logger::
+pszRawMsg:1:36:<13>Jul 19 17:08:48 logger: downnow2:
+pszInputName:1:8:imuxsock:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:2:27:
>End
.
Now I restart rsyslog on the central server and the client, and log a
few more messages on the client:
============================================
root@relp-server /etc # service rsyslog start
rsyslog start/running, process 331
============================================
root@rsyslog-test:/etc# service rsyslog start
rsyslog start/running, process 23971
root@rsyslog-test:/etc# logger upagain
root@rsyslog-test:/etc# logger upagain2
root@rsyslog-test:/etc# logger upagain3
I check the central server logs and all messages except the lost
"downnow1" message are successfully received:
============================================
root@relp-server /etc # tail -n 20 /var/log/relp_log
2013-07-19T17:07:55.757080-04:00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:08:15.894360-04:00 rsyslog-test logger: upnow2
2013-07-19T17:08:48.446601-04:00 rsyslog-test rsyslogd-2353:
omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error
-53: Error in the push function.]', object 'conn to srvr
10.10.5.1:20514' - action may not work as intended [try
http://www.rsyslog.com/e/2353 ]
2013-07-19T17:08:54.453092-04:00 rsyslog-test logger: downnow3
2013-07-19T17:09:13.302262-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com";] exiting on signal 15.
2013-07-19T17:08:48.446359-04:00 rsyslog-test logger: downnow2
2013-07-19T17:10:15.245473-04:00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23971"
x-info="http://www.rsyslog.com";] start
2013-07-19T17:10:35.210411-04:00 rsyslog-test logger: upagain
2013-07-19T17:10:41.354935-04:00 rsyslog-test logger: upagain2
2013-07-19T17:10:48.602667-04:00 rsyslog-test logger: upagain3
Here is the relp server config:
============================================
root@relp-server /etc # cat rsyslog.conf
$PrivDropToUser syslog
$PrivDropToGroup syslog
$AbortOnUncleanConfig off
$umask 0000 # Ensure umask is reset
$DirCreateMode 0700
$FileCreateMode 0644
$DirOwner syslog
$DirGroup syslog
# DynaFile settings
$FileOwner syslog
$FileGroup syslog
$FailOnChownFailure on
$DynaFileCacheSize 100 # a cache of 100 files at most
$MaxOpenFiles 200000
$MaxMessageSize 64k
$MainMsgQueueSize 100000
$DropTrailingLFOnReception on
$Escape8BitCharactersOnReceive off
$EscapeControlCharactersOnReceive on
$RepeatedMsgReduction off # log every message
$DropMsgsWithMaliciousDnsPTRRecords off
module(load="imuxsock")
module(load="imrelp" ruleset="relp")
input(type="imrelp" port="20514" tls="on"
tls.caCert="/etc/rsyslog.certs/ca.pem"
tls.myCert="/etc/rsyslog.certs/server.cert"
tls.myPrivKey="/etc/rsyslog.certs/server.key"
tls.authMode="name"
tls.permittedpeer=["rsyslog-test.mydomain.com"]
)
ruleset (name="relp") {
$RulesetCreateMainQueue on
action(type="omfile" file="/var/log/relp_log")
}
$IncludeConfig /etc/rsyslog.d/
*.* /var/log/syslog
And the client config:
============================================
root@rsyslog-test:/etc# cat rsyslog.conf
module(load="imuxsock")
module(load="omrelp")
module(load="imtcp")
input(type="imtcp" port="514")
$WorkDirectory /var/spool/rsyslog/queue # default location for work
(spool) files
$MaxMessageSize 64k
$MainMsgQueueFileName mainq # set file name, also enables disk mode
$MainMsgQueueType LinkedList
$MainMsgQueueSaveOnShutDown on
$MainMsgQueueMaxDiskSpace 40g
$MainMsgQueueSize 8000000
$ActionSendResendLastMsgOnReconnect on
action(type="omrelp" target="10.10.5.1" port="20514" tls="on"
tls.compression="on"
tls.caCert="/etc/rsyslog.certs/ca.pem"
tls.myCert="/etc/rsyslog.certs/rsyslog-test.cert"
tls.myPrivKey="/etc/rsyslog.certs/rsyslog-test.key"
tls.authmode="name"
tls.permittedpeer=["log.mydomain.com"]
queue.filename="relpy"
queue.maxdiskspace="10g"
queue.saveonshutdown="on"
queue.type="linkedlist"
queue.maxfilesize="20m"
action.resumeretrycount="-1"
)
*.* /var/log/rsyslog-test
--
Dave
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
