That's not a bug but design, if the client shuts down, we lose state. There are ways around that, but the implementation needs a sponsor.
Rainer Sent from phone, thus brief. Am 01.08.2013 14:45 schrieb "David Anderson" <[email protected]>: > Thanks David and Andre for your replies. I have filed bug 470: > > http://bugzilla.adiscon.com/**show_bug.cgi?id=470<http://bugzilla.adiscon.com/show_bug.cgi?id=470> > > -- > Dave > > On 8/1/13 6:12 AM, Andre Lorbach wrote: > >> I am going to look into this soon, I need to setup a working test >> environment first. >> Perhaps you can create a new bug and provide initial sample config files: >> http://bugzilla.adiscon.com/**buglist.cgi?product=rsyslog&**list_id=190<http://bugzilla.adiscon.com/buglist.cgi?product=rsyslog&list_id=190> >> >> best regards, >> Andre >> >> -----Original Message----- >>> From: >>> [email protected].**com<[email protected]>[mailto: >>> rsyslog- >>> [email protected]] On Behalf Of David Lang >>> Sent: Thursday, August 01, 2013 9:13 AM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Message loss with relp and queues >>> >>> FYI, Rainer is on vacation for several weeks. I'm answering the >>> >> questions that >> >>> I can, but this is one that I can't touch. >>> >>> Andre is the person from adiscon who's watching the list, so I've been >>> keeping track of threads like this (so they don't get lost) and hoping >>> >> that he >> >>> can step in. >>> >>> David Lang >>> >>> On Fri, 26 Jul 2013, David Anderson wrote: >>> >>> Sorry to "bump" this, but I'm wondering if I'm just not setting >>>> something in my configuration properly that is causing this message >>>> loss, or if this is a bug. Can anyone replicate or comment? >>>> >>>> Thanks, >>>> -- >>>> Dave >>>> >>>> On 7/19/13 4:43 PM, David Anderson wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm using central logging using RELP+TLS with disk assisted queues on >>>>> the client. When I: >>>>> 1) stop rsyslog on the server (or just drop all packets at the >>>>> firewall to simulate network loss) >>>>> 2) send a few messages to syslog with "logger" on the client (rsyslog >>>>> notices the relp server is down and puts the messages in the queue) >>>>> 3) then restart rsyslog on the client >>>>> >>>>> the first message (sometimes I've noticed two) sent to logger while >>>>> the server was unreachable is lost and does not get sent to the >>>>> >>>> central >> >>> server. >>> >>>> If I *do not* restart rsyslog on the client while the central server >>>>> is unreachable, the message is not lost. This happens with both >>>>> LinkedList and Disk queues. >>>>> >>>>> It's not uncommon that the client might be restarted during a time >>>>> when it has already lost the network connection to the central >>>>> server. Am I missing a config that might fix this? >>>>> >>>>> Here are my example logs showing the message loss: >>>>> >>>>> I start off with both central server and rsyslog client running >>>>> normally with RELP working fine: >>>>> ==============================**============== >>>>> root@rsyslog-test:/etc# logger upnow >>>>> root@rsyslog-test:/etc# service rsyslog restart rsyslog stop/waiting >>>>> rsyslog start/running, process 23909 root@rsyslog-test:/etc# logger >>>>> upnow2 >>>>> >>>>> Works fine. Central server receives both messages and sees the client >>>>> restart: >>>>> ==============================**============== >>>>> root@relp-server /etc # tail /var/log/relp_log >>>>> 2013-07-19T17:07:55.757080-04:**00 rsyslog-test logger: upnow >>>>> 2013-07-19T17:08:13.439180-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23883" >>>>> x-info="http://www.rsyslog.com**";] exiting on signal 15. >>>>> 2013-07-19T17:08:13.493691-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23909" >>>>> x-info="http://www.rsyslog.com**";] start >>>>> 2013-07-19T17:08:15.894360-04:**00 rsyslog-test logger: upnow2 >>>>> >>>>> Now I stop the central log server: >>>>> ==============================**============== >>>>> root@relp-server /etc # service rsyslog stop rsyslog stop/waiting >>>>> >>>>> Log some messages to syslog on the client with "logger" and stop >>>>> rsyslog on the client: >>>>> ==============================**============== >>>>> root@rsyslog-test:/etc# logger downnow1 root@rsyslog-test:/etc# >>>>> logger downnow2 root@rsyslog-test:/etc# logger downnow3 >>>>> root@rsyslog-test:/etc# service rsyslog stop rsyslog stop/waiting >>>>> >>>>> >>>>> Since I have all messages being logged locally on the client I can >>>>> >>>> see that >> >>> my messages were sent to syslog and syslog detected the central >>>>> >>>> server >> >>> was >>> >>>> down after I logged the 'downnow2' message: >>>>> ==============================**============== >>>>> root@rsyslog-test:/etc# tail /var/log/rsyslog-test >>>>> 2013-07-19T17:07:20.413878-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23883" >>>>> x-info="http://www.rsyslog.com**";] start >>>>> 2013-07-19T17:07:55.757080-04:**00 rsyslog-test logger: upnow >>>>> 2013-07-19T17:08:13.439180-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23883" >>>>> x-info="http://www.rsyslog.com**";] exiting on signal 15. >>>>> 2013-07-19T17:08:13.493691-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23909" >>>>> x-info="http://www.rsyslog.com**";] start >>>>> 2013-07-19T17:08:15.894360-04:**00 rsyslog-test logger: upnow2 >>>>> 2013-07-19T17:08:31.045190-04:**00 rsyslog-test logger: downnow1 >>>>> 2013-07-19T17:08:48.446359-04:**00 rsyslog-test logger: downnow2 >>>>> 2013-07-19T17:08:48.446601-04:**00 rsyslog-test rsyslogd-2353: >>>>> omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error >>>>> >>>> -53: >> >>> Error in the push function.]', object 'conn to srvr 10.10.5.1:20514' >>>>> >>>> - >> >>> action may not work as intended [try http://www.rsyslog.com/e/2353 ] >>>>> 2013-07-19T17:08:54.453092-04:**00 rsyslog-test logger: downnow3 >>>>> 2013-07-19T17:09:13.302262-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23909" >>>>> x-info="http://www.rsyslog.com**";] exiting on signal 15. >>>>> >>>>> >>>>> I can see the messages except the first 'downnow1' are in the disk >>>>> >>>> queue >> >>> file on the client: >>>>> ==============================**============== >>>>> root@rsyslog-test:/etc# cat /var/spool/rsyslog/queue/**relpy.00000001 >>>>> <Obj:1:msg:1: >>>>> +iProtocolVersion:2:1:0: >>>>> +iSeverity:2:1:3: >>>>> +iFacility:2:1:5: >>>>> +msgFlags:2:1:1: >>>>> +ttGenTime:2:10:1374268128: >>>>> +tRcvdAt:3:34:2:2013:7:19:17:**8:48:446601:6:-:4:0: >>>>> +tTIMESTAMP:3:34:2:2013:7:19:**17:8:48:446601:6:-:4:0: >>>>> +pszTAG:1:14:rsyslogd-2353:: >>>>> +pszRawMsg:1:215:omrelp[10.10.**5.1:20514 <http://10.10.5.1:20514>]: >>>>> error 'TLS record write >>>>> >>>> failed >> >>> [gnutls error -53: Error in the push function.]', object 'conn to >>>>> >>>> srvr >> >>> 10.10.5.1:20514' - action may not work as intended [try >>>>> http://www.rsyslog.com/e/2353 ]: >>>>> +pszHOSTNAME:1:17:rsyslog-**test: >>>>> +pszInputName:1:8:rsyslogd: >>>>> +pszRcvFrom:1:17:rsyslog-test: >>>>> +pszRcvFromIP:1:9:127.0.0.1: >>>>> +offMSG:2:1:0: >>>>> >>>>>> End >>>>>> >>>>> . >>>>> <Obj:1:msg:1: >>>>> +iProtocolVersion:2:1:0: >>>>> +iSeverity:2:1:5: >>>>> +iFacility:2:1:1: >>>>> +msgFlags:2:1:4: >>>>> +ttGenTime:2:10:1374268134: >>>>> +tRcvdAt:3:34:2:2013:7:19:17:**8:54:453092:6:-:4:0: >>>>> +tTIMESTAMP:3:34:2:2013:7:19:**17:8:54:453092:6:-:4:0: >>>>> +pszTAG:1:7:logger:: >>>>> +pszRawMsg:1:36:<13>Jul 19 17:08:54 logger: downnow3: >>>>> +pszInputName:1:8:imuxsock: >>>>> +pszRcvFrom:1:17:rsyslog-test: >>>>> +pszRcvFromIP:1:9:127.0.0.1: >>>>> +offMSG:2:2:27: >>>>> >>>>>> End >>>>>> >>>>> . >>>>> <Obj:1:msg:1: >>>>> +iProtocolVersion:2:1:0: >>>>> +iSeverity:2:1:6: >>>>> +iFacility:2:1:5: >>>>> +msgFlags:2:1:1: >>>>> +ttGenTime:2:10:1374268153: >>>>> +tRcvdAt:3:34:2:2013:7:19:17:**9:13:302262:6:-:4:0: >>>>> +tTIMESTAMP:3:34:2:2013:7:19:**17:9:13:302262:6:-:4:0: >>>>> +pszTAG:1:9:rsyslogd:: >>>>> +pszRawMsg:1:115: [origin software="rsyslogd" swVersion="7.5.2" >>>>> x-pid="23909" x-info="http://www.rsyslog.com**";] exiting on signal >>>>> 15.: >>>>> +pszHOSTNAME:1:17:rsyslog-**test: >>>>> +pszInputName:1:8:rsyslogd: >>>>> +pszRcvFrom:1:17:rsyslog-test: >>>>> +pszRcvFromIP:1:9:127.0.0.1: >>>>> +offMSG:2:1:0: >>>>> >>>>>> End >>>>>> >>>>> . >>>>> <Obj:1:msg:1: >>>>> +iProtocolVersion:2:1:0: >>>>> +iSeverity:2:1:5: >>>>> +iFacility:2:1:1: >>>>> +msgFlags:2:1:4: >>>>> +ttGenTime:2:10:1374268128: >>>>> +tRcvdAt:3:34:2:2013:7:19:17:**8:48:446359:6:-:4:0: >>>>> +tTIMESTAMP:3:34:2:2013:7:19:**17:8:48:446359:6:-:4:0: >>>>> +pszTAG:1:7:logger:: >>>>> +pszRawMsg:1:36:<13>Jul 19 17:08:48 logger: downnow2: >>>>> +pszInputName:1:8:imuxsock: >>>>> +pszRcvFrom:1:17:rsyslog-test: >>>>> +pszRcvFromIP:1:9:127.0.0.1: >>>>> +offMSG:2:2:27: >>>>> >>>>>> End >>>>>> >>>>> . >>>>> >>>>> >>>>> Now I restart rsyslog on the central server and the client, and log a >>>>> >>>> few >> >>> more messages on the client: >>>>> ==============================**============== >>>>> root@relp-server /etc # service rsyslog start >>>>> rsyslog start/running, process 331 >>>>> ==============================**============== >>>>> root@rsyslog-test:/etc# service rsyslog start >>>>> rsyslog start/running, process 23971 >>>>> root@rsyslog-test:/etc# logger upagain >>>>> root@rsyslog-test:/etc# logger upagain2 >>>>> root@rsyslog-test:/etc# logger upagain3 >>>>> >>>>> I check the central server logs and all messages except the lost >>>>> >>>> "downnow1" >>> >>>> message are successfully received: >>>>> ==============================**============== >>>>> root@relp-server /etc # tail -n 20 /var/log/relp_log >>>>> 2013-07-19T17:07:55.757080-04:**00 rsyslog-test logger: upnow >>>>> 2013-07-19T17:08:13.439180-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23883" >>>>> x-info="http://www.rsyslog.com**";] exiting on signal 15. >>>>> 2013-07-19T17:08:13.493691-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23909" >>>>> x-info="http://www.rsyslog.com**";] start >>>>> 2013-07-19T17:08:15.894360-04:**00 rsyslog-test logger: upnow2 >>>>> 2013-07-19T17:08:48.446601-04:**00 rsyslog-test rsyslogd-2353: >>>>> omrelp[10.10.5.1:20514]: error 'TLS record write failed [gnutls error >>>>> >>>> -53: >> >>> Error in the push function.]', object 'conn to srvr 10.10.5.1:20514' >>>>> >>>> - >> >>> action may not work as intended [try http://www.rsyslog.com/e/2353 ] >>>>> 2013-07-19T17:08:54.453092-04:**00 rsyslog-test logger: downnow3 >>>>> 2013-07-19T17:09:13.302262-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23909" >>>>> x-info="http://www.rsyslog.com**";] exiting on signal 15. >>>>> 2013-07-19T17:08:48.446359-04:**00 rsyslog-test logger: downnow2 >>>>> 2013-07-19T17:10:15.245473-04:**00 rsyslog-test rsyslogd: [origin >>>>> software="rsyslogd" swVersion="7.5.2" x-pid="23971" >>>>> x-info="http://www.rsyslog.com**";] start >>>>> 2013-07-19T17:10:35.210411-04:**00 rsyslog-test logger: upagain >>>>> 2013-07-19T17:10:41.354935-04:**00 rsyslog-test logger: upagain2 >>>>> 2013-07-19T17:10:48.602667-04:**00 rsyslog-test logger: upagain3 >>>>> >>>>> >>>>> Here is the relp server config: >>>>> ==============================**============== >>>>> root@relp-server /etc # cat rsyslog.conf >>>>> $PrivDropToUser syslog >>>>> $PrivDropToGroup syslog >>>>> >>>>> $AbortOnUncleanConfig off >>>>> >>>>> $umask 0000 # Ensure umask is reset >>>>> $DirCreateMode 0700 >>>>> $FileCreateMode 0644 >>>>> $DirOwner syslog >>>>> $DirGroup syslog >>>>> >>>>> # DynaFile settings >>>>> $FileOwner syslog >>>>> $FileGroup syslog >>>>> $FailOnChownFailure on >>>>> $DynaFileCacheSize 100 # a cache of 100 files at most >>>>> >>>>> $MaxOpenFiles 200000 >>>>> >>>>> $MaxMessageSize 64k >>>>> >>>>> $MainMsgQueueSize 100000 >>>>> >>>>> $DropTrailingLFOnReception on >>>>> $Escape8BitCharactersOnReceive off >>>>> $**EscapeControlCharactersOnRecei**ve on >>>>> $RepeatedMsgReduction off # log every message >>>>> >>>>> $**DropMsgsWithMaliciousDnsPTRRec**ords off >>>>> >>>>> module(load="imuxsock") >>>>> module(load="imrelp" ruleset="relp") >>>>> >>>>> input(type="imrelp" port="20514" tls="on" >>>>> tls.caCert="/etc/rsyslog.**certs/ca.pem" >>>>> tls.myCert="/etc/rsyslog.**certs/server.cert" >>>>> tls.myPrivKey="/etc/rsyslog.**certs/server.key" >>>>> tls.authMode="name" >>>>> >>>>> tls.permittedpeer=["rsyslog-**test.mydomain.com<http://rsyslog-test.mydomain.com> >>>>> "] >>>>> ) >>>>> >>>>> ruleset (name="relp") { >>>>> $RulesetCreateMainQueue on >>>>> action(type="omfile" file="/var/log/relp_log") >>>>> } >>>>> >>>>> $IncludeConfig /etc/rsyslog.d/ >>>>> >>>>> *.* /var/log/syslog >>>>> >>>>> >>>>> >>>>> And the client config: >>>>> ==============================**============== >>>>> root@rsyslog-test:/etc# cat rsyslog.conf >>>>> module(load="imuxsock") >>>>> module(load="omrelp") >>>>> module(load="imtcp") >>>>> >>>>> input(type="imtcp" port="514") >>>>> >>>>> $WorkDirectory /var/spool/rsyslog/queue # default location for work >>>>> >>>> (spool) >>> >>>> files >>>>> >>>>> $MaxMessageSize 64k >>>>> >>>>> $MainMsgQueueFileName mainq # set file name, also enables disk mode >>>>> $MainMsgQueueType LinkedList >>>>> $MainMsgQueueSaveOnShutDown on >>>>> $MainMsgQueueMaxDiskSpace 40g >>>>> $MainMsgQueueSize 8000000 >>>>> >>>>> $**ActionSendResendLastMsgOnRecon**nect on >>>>> >>>>> action(type="omrelp" target="10.10.5.1" port="20514" tls="on" >>>>> tls.compression="on" >>>>> tls.caCert="/etc/rsyslog.**certs/ca.pem" >>>>> tls.myCert="/etc/rsyslog.**certs/rsyslog-test.cert" >>>>> tls.myPrivKey="/etc/rsyslog.**certs/rsyslog-test.key" >>>>> tls.authmode="name" >>>>> tls.permittedpeer=["log.**mydomain.com <http://log.mydomain.com>"] >>>>> queue.filename="relpy" >>>>> queue.maxdiskspace="10g" >>>>> queue.saveonshutdown="on" >>>>> queue.type="linkedlist" >>>>> queue.maxfilesize="20m" >>>>> action.resumeretrycount="-1" >>>>> ) >>>>> >>>>> *.* /var/log/rsyslog-test >>>>> >>>>> >>>>> -- >>>>> Dave >>>>> ______________________________**_________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> >>>> myriad >> >>> of >>> >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> >>>> DON'T >>> >>>> LIKE THAT. >>>>> >>>> ______________________________**_________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >>> myriad of >> >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> >>> DON'T >>> >>>> LIKE THAT. >>>> >>>> ______________________________**_________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> >> of >> >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
