I thought that in that case, it would duplicate messages, not loose them.
David Lang
On Thu, 1 Aug 2013, Rainer Gerhards wrote:
That's not a bug but design, if the client shuts down, we lose state.
There are ways around that, but the implementation needs a sponsor.
Rainer
Sent from phone, thus brief.
Am 01.08.2013 14:45 schrieb "David Anderson" <[email protected]>:
Thanks David and Andre for your replies. I have filed bug 470:
http://bugzilla.adiscon.com/**show_bug.cgi?id=470<http://bugzilla.adiscon.com/show_bug.cgi?id=470>
--
Dave
On 8/1/13 6:12 AM, Andre Lorbach wrote:
I am going to look into this soon, I need to setup a working test
environment first.
Perhaps you can create a new bug and provide initial sample config files:
http://bugzilla.adiscon.com/**buglist.cgi?product=rsyslog&**list_id=190<http://bugzilla.adiscon.com/buglist.cgi?product=rsyslog&list_id=190>
best regards,
Andre
-----Original Message-----
From:
[email protected].**com<[email protected]>[mailto:
rsyslog-
[email protected]] On Behalf Of David Lang
Sent: Thursday, August 01, 2013 9:13 AM
To: rsyslog-users
Subject: Re: [rsyslog] Message loss with relp and queues
FYI, Rainer is on vacation for several weeks. I'm answering the
questions that
I can, but this is one that I can't touch.
Andre is the person from adiscon who's watching the list, so I've been
keeping track of threads like this (so they don't get lost) and hoping
that he
can step in.
David Lang
On Fri, 26 Jul 2013, David Anderson wrote:
Sorry to "bump" this, but I'm wondering if I'm just not setting
something in my configuration properly that is causing this message
loss, or if this is a bug. Can anyone replicate or comment?
Thanks,
--
Dave
On 7/19/13 4:43 PM, David Anderson wrote:
Hello,
I'm using central logging using RELP+TLS with disk assisted queues on
the client. When I:
1) stop rsyslog on the server (or just drop all packets at the
firewall to simulate network loss)
2) send a few messages to syslog with "logger" on the client (rsyslog
notices the relp server is down and puts the messages in the queue)
3) then restart rsyslog on the client
the first message (sometimes I've noticed two) sent to logger while
the server was unreachable is lost and does not get sent to the
central
server.
If I *do not* restart rsyslog on the client while the central server
is unreachable, the message is not lost. This happens with both
LinkedList and Disk queues.
It's not uncommon that the client might be restarted during a time
when it has already lost the network connection to the central
server. Am I missing a config that might fix this?
Here are my example logs showing the message loss:
I start off with both central server and rsyslog client running
normally with RELP working fine:
==============================**==============
root@rsyslog-test:/etc# logger upnow
root@rsyslog-test:/etc# service rsyslog restart rsyslog stop/waiting
rsyslog start/running, process 23909 root@rsyslog-test:/etc# logger
upnow2
Works fine. Central server receives both messages and sees the client
restart:
==============================**==============
root@relp-server /etc # tail /var/log/relp_log
2013-07-19T17:07:55.757080-04:**00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com**";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com**";] start
2013-07-19T17:08:15.894360-04:**00 rsyslog-test logger: upnow2
Now I stop the central log server:
==============================**==============
root@relp-server /etc # service rsyslog stop rsyslog stop/waiting
Log some messages to syslog on the client with "logger" and stop
rsyslog on the client:
==============================**==============
root@rsyslog-test:/etc# logger downnow1 root@rsyslog-test:/etc#
logger downnow2 root@rsyslog-test:/etc# logger downnow3
root@rsyslog-test:/etc# service rsyslog stop rsyslog stop/waiting
Since I have all messages being logged locally on the client I can
see that
my messages were sent to syslog and syslog detected the central
server
was
down after I logged the 'downnow2' message:
==============================**==============
root@rsyslog-test:/etc# tail /var/log/rsyslog-test
2013-07-19T17:07:20.413878-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com**";] start
2013-07-19T17:07:55.757080-04:**00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com**";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com**";] start
2013-07-19T17:08:15.894360-04:**00 rsyslog-test logger: upnow2
2013-07-19T17:08:31.045190-04:**00 rsyslog-test logger: downnow1
2013-07-19T17:08:48.446359-04:**00 rsyslog-test logger: downnow2
2013-07-19T17:08:48.446601-04:**00 rsyslog-test rsyslogd-2353:
omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error
-53:
Error in the push function.]', object 'conn to srvr 10.10.5.1:20514'
-
action may not work as intended [try http://www.rsyslog.com/e/2353 ]
2013-07-19T17:08:54.453092-04:**00 rsyslog-test logger: downnow3
2013-07-19T17:09:13.302262-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com**";] exiting on signal 15.
I can see the messages except the first 'downnow1' are in the disk
queue
file on the client:
==============================**==============
root@rsyslog-test:/etc# cat /var/spool/rsyslog/queue/**relpy.00000001
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:3:
+iFacility:2:1:5:
+msgFlags:2:1:1:
+ttGenTime:2:10:1374268128:
+tRcvdAt:3:34:2:2013:7:19:17:**8:48:446601:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:**17:8:48:446601:6:-:4:0:
+pszTAG:1:14:rsyslogd-2353::
+pszRawMsg:1:215:omrelp[10.10.**5.1:20514 <http://10.10.5.1:20514>]:
error 'TLS record write
failed
[gnutls error -53: Error in the push function.]', object 'conn to
srvr
10.10.5.1:20514' - action may not work as intended [try
http://www.rsyslog.com/e/2353 ]:
+pszHOSTNAME:1:17:rsyslog-**test:
+pszInputName:1:8:rsyslogd:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:1:0:
End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:5:
+iFacility:2:1:1:
+msgFlags:2:1:4:
+ttGenTime:2:10:1374268134:
+tRcvdAt:3:34:2:2013:7:19:17:**8:54:453092:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:**17:8:54:453092:6:-:4:0:
+pszTAG:1:7:logger::
+pszRawMsg:1:36:<13>Jul 19 17:08:54 logger: downnow3:
+pszInputName:1:8:imuxsock:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:2:27:
End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:6:
+iFacility:2:1:5:
+msgFlags:2:1:1:
+ttGenTime:2:10:1374268153:
+tRcvdAt:3:34:2:2013:7:19:17:**9:13:302262:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:**17:9:13:302262:6:-:4:0:
+pszTAG:1:9:rsyslogd::
+pszRawMsg:1:115: [origin software="rsyslogd" swVersion="7.5.2"
x-pid="23909" x-info="http://www.rsyslog.com**";] exiting on signal
15.:
+pszHOSTNAME:1:17:rsyslog-**test:
+pszInputName:1:8:rsyslogd:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:1:0:
End
.
<Obj:1:msg:1:
+iProtocolVersion:2:1:0:
+iSeverity:2:1:5:
+iFacility:2:1:1:
+msgFlags:2:1:4:
+ttGenTime:2:10:1374268128:
+tRcvdAt:3:34:2:2013:7:19:17:**8:48:446359:6:-:4:0:
+tTIMESTAMP:3:34:2:2013:7:19:**17:8:48:446359:6:-:4:0:
+pszTAG:1:7:logger::
+pszRawMsg:1:36:<13>Jul 19 17:08:48 logger: downnow2:
+pszInputName:1:8:imuxsock:
+pszRcvFrom:1:17:rsyslog-test:
+pszRcvFromIP:1:9:127.0.0.1:
+offMSG:2:2:27:
End
.
Now I restart rsyslog on the central server and the client, and log a
few
more messages on the client:
==============================**==============
root@relp-server /etc # service rsyslog start
rsyslog start/running, process 331
==============================**==============
root@rsyslog-test:/etc# service rsyslog start
rsyslog start/running, process 23971
root@rsyslog-test:/etc# logger upagain
root@rsyslog-test:/etc# logger upagain2
root@rsyslog-test:/etc# logger upagain3
I check the central server logs and all messages except the lost
"downnow1"
message are successfully received:
==============================**==============
root@relp-server /etc # tail -n 20 /var/log/relp_log
2013-07-19T17:07:55.757080-04:**00 rsyslog-test logger: upnow
2013-07-19T17:08:13.439180-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23883"
x-info="http://www.rsyslog.com**";] exiting on signal 15.
2013-07-19T17:08:13.493691-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com**";] start
2013-07-19T17:08:15.894360-04:**00 rsyslog-test logger: upnow2
2013-07-19T17:08:48.446601-04:**00 rsyslog-test rsyslogd-2353:
omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error
-53:
Error in the push function.]', object 'conn to srvr 10.10.5.1:20514'
-
action may not work as intended [try http://www.rsyslog.com/e/2353 ]
2013-07-19T17:08:54.453092-04:**00 rsyslog-test logger: downnow3
2013-07-19T17:09:13.302262-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23909"
x-info="http://www.rsyslog.com**";] exiting on signal 15.
2013-07-19T17:08:48.446359-04:**00 rsyslog-test logger: downnow2
2013-07-19T17:10:15.245473-04:**00 rsyslog-test rsyslogd: [origin
software="rsyslogd" swVersion="7.5.2" x-pid="23971"
x-info="http://www.rsyslog.com**";] start
2013-07-19T17:10:35.210411-04:**00 rsyslog-test logger: upagain
2013-07-19T17:10:41.354935-04:**00 rsyslog-test logger: upagain2
2013-07-19T17:10:48.602667-04:**00 rsyslog-test logger: upagain3
Here is the relp server config:
==============================**==============
root@relp-server /etc # cat rsyslog.conf
$PrivDropToUser syslog
$PrivDropToGroup syslog
$AbortOnUncleanConfig off
$umask 0000 # Ensure umask is reset
$DirCreateMode 0700
$FileCreateMode 0644
$DirOwner syslog
$DirGroup syslog
# DynaFile settings
$FileOwner syslog
$FileGroup syslog
$FailOnChownFailure on
$DynaFileCacheSize 100 # a cache of 100 files at most
$MaxOpenFiles 200000
$MaxMessageSize 64k
$MainMsgQueueSize 100000
$DropTrailingLFOnReception on
$Escape8BitCharactersOnReceive off
$**EscapeControlCharactersOnRecei**ve on
$RepeatedMsgReduction off # log every message
$**DropMsgsWithMaliciousDnsPTRRec**ords off
module(load="imuxsock")
module(load="imrelp" ruleset="relp")
input(type="imrelp" port="20514" tls="on"
tls.caCert="/etc/rsyslog.**certs/ca.pem"
tls.myCert="/etc/rsyslog.**certs/server.cert"
tls.myPrivKey="/etc/rsyslog.**certs/server.key"
tls.authMode="name"
tls.permittedpeer=["rsyslog-**test.mydomain.com<http://rsyslog-test.mydomain.com>
"]
)
ruleset (name="relp") {
$RulesetCreateMainQueue on
action(type="omfile" file="/var/log/relp_log")
}
$IncludeConfig /etc/rsyslog.d/
*.* /var/log/syslog
And the client config:
==============================**==============
root@rsyslog-test:/etc# cat rsyslog.conf
module(load="imuxsock")
module(load="omrelp")
module(load="imtcp")
input(type="imtcp" port="514")
$WorkDirectory /var/spool/rsyslog/queue # default location for work
(spool)
files
$MaxMessageSize 64k
$MainMsgQueueFileName mainq # set file name, also enables disk mode
$MainMsgQueueType LinkedList
$MainMsgQueueSaveOnShutDown on
$MainMsgQueueMaxDiskSpace 40g
$MainMsgQueueSize 8000000
$**ActionSendResendLastMsgOnRecon**nect on
action(type="omrelp" target="10.10.5.1" port="20514" tls="on"
tls.compression="on"
tls.caCert="/etc/rsyslog.**certs/ca.pem"
tls.myCert="/etc/rsyslog.**certs/rsyslog-test.cert"
tls.myPrivKey="/etc/rsyslog.**certs/rsyslog-test.key"
tls.authmode="name"
tls.permittedpeer=["log.**mydomain.com <http://log.mydomain.com>"]
queue.filename="relpy"
queue.maxdiskspace="10g"
queue.saveonshutdown="on"
queue.type="linkedlist"
queue.maxfilesize="20m"
action.resumeretrycount="-1"
)
*.* /var/log/rsyslog-test
--
Dave
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T
LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T
LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
