Saw this error message below several times, not every time though: rsyslogd-2040: fatal error on disk queue 'action 22 queue[DA]', emergency switch to direct mode [try http://www.rsyslog.com/e/2040 ]
I also noticed that .qi file in the disk queue directory doesn't show up sometimes. Is this file required for disk queue to function correctly? I did notice that disk queue flushing was working when the files was absent in several cases. Thanks, Fajun On Thu, May 9, 2013 at 4:50 PM, David Lang <[email protected]> wrote: > On Thu, 9 May 2013, Fajun Chen wrote: > > Resending original reply without debugging logs since it was blocked >> waiting for approval (message size over 512k). >> >> Thanks, >> Fajun >> >> On Thu, May 9, 2013 at 2:52 PM, Fajun Chen <[email protected]> wrote: >> >> >>> >>> >>> On Thu, May 9, 2013 at 11:34 AM, David Lang <[email protected]> wrote: >>> >>> On Thu, 9 May 2013, Fajun Chen wrote: >>>> >>>> On Wed, May 8, 2013 at 9:22 PM, David Lang <[email protected]> wrote: >>>> >>>>> >>>>> On Wed, 8 May 2013, Fajun Chen wrote: >>>>> >>>>>> >>>>>> iptables block setting didn't work for some reason. >>>>>> >>>>>> >>>>>>> >>>>>>> what do the iptables rules on that system look like? >>>>>> >>>>>> without seeing them, my guess is that there is a rule already there >>>>>> that >>>>>> allows packets related to a known connection that are getting applied >>>>>> (and >>>>>> therefor accepting the packets) before the deny rule you are trying to >>>>>> put >>>>>> in place takes effect. >>>>>> >>>>>> >>>>> >>>>> The same problem exists on 5.8.6 with iptables blocking. One minor >>>>> detail: >>>>> the queued files reached the limit of 96M, it's reduced to 95M after >>>>> the >>>>> firewall was unblocked, but it stays at 95M on the client without >>>>> flushing. >>>>> I can use logger to send new log messages to the server, so network >>>>> connection is not an issue. >>>>> >>>>> 7.3.14 seems to be working with iptables blocking. >>>>> >>>>> >>>> hmm, I don't understand how it could be different for different versions >>>> of rsyslog. the iptables filtering should be happening by the OS and >>>> wouldn't care what version of software is running. >>>> >>> >>> >>> iptables filtering issue had been resolved by restarting rsyslog for the >>> firewall changes to take effect. >>> >> > Ok, that is almost certinly the 'established connection' thing that I was > speculating about. > > > rsyslog version has nothing to do with iptables filtering. What I referred >>> to was that rsyslog 5.8.6 doesn't flush queued files while 7.3.14 does >>> when >>> iptables filtering was changed from blocking to unblocking. >>> >> > ahh, Ok. > > At this point 5.8.6 is old enough that it's well past being supported, so > let's work on the current version. > > > >>>> >>>> As a alternative testing, I stopped rsylogd on the remote server and >>>> the >>>> >>>>> >>>>>> logs were queued on the client as expected. I started rsyslog on the >>>>>>> remote >>>>>>> server once the disk queue on the client is filled up. I did see the >>>>>>> queue >>>>>>> files were flushed to the remote server once rsyslog is back to >>>>>>> service. So >>>>>>> this seems to be related to rsyslog configuration change. >>>>>>> >>>>>>> >>>>>>> My guess (without knowing the code well) is that the queued >>>>>> messages are >>>>>> somehow queued for the specific destination (IIRC you had this queue >>>>>> setup >>>>>> as an action queue, not as the main queue, you posted your config, >>>>>> but I >>>>>> have already deleted those messages). I'd be curious to see if you >>>>>> have >>>>>> the >>>>>> same problem spilling the main queue to disk. >>>>>> >>>>>> >>>>> >>>>> Just for your reference, here's my rsyslog configuration: >>>>> >>>>> # start forwarding rule 1 of 1 >>>>> $ActionQueueType LinkedList >>>>> $ActionQueueFileName srvrfwd >>>>> $ActionResumeRetryCount -1 >>>>> $ActionQueueSaveOnShutdown on >>>>> $ActionQueueMaxDiskSpace 100000000 >>>>> $ActionQueueSize 200000 # Tried 100000 as well >>>>> $ActionQueueHighWaterMark 600 >>>>> $ActionQueueLowWaterMark 200 >>>>> $ActionQueueTimeoutEnqueue 1 >>>>> >>>>> #local5.* :omrelp:127.255.255.1:20514 # Invalid IP to trigger log >>>>> buffering >>>>> local5.* :omrelp:172.17.5.28:20514 # Real IP to trigger log >>>>> forwarding >>>>> # end forwarding rule 1 of 1 >>>>> >>>>> >>>>> >>>>> On the other hand, as I noted in the first report, when I changed >>>>>> rsyslog >>>>>> >>>>>> configuration before disk space limit is reached, the queued files >>>>>>> were >>>>>>> flushed to the remote server without issues. >>>>>>> >>>>>>> >>>>>>> very interesting, and probably a bug. >>>>>> >>>>>> >>>>> >>>>> Let me know if you need debugging logs to troubleshoot it. >>>>> >>>>> >>>> Ranier will need probably need to get involved with this, but he's super >>>> busy the next 2-3 weeks with a very high priority deadline (the "every >>>> waking hour" type of project) >>>> >>>> It wouldn't hurt to take a look at the debug logs for the copy started >>>> after the config change. >>>> >>>> >>> Rsyslog debugging log is attached here. This was collected by running >>> "rsyslogd -dn" when the remote server IP was set to valid. Please let me >>> know if you want me to submit bug tracking item. >>> >> > you can e-mail it to me directly > > > >>>> by the way, are you sure you are doing a full restart after the config >>>> change? a -HUP does not cause rsyslog to do a full restart and re-read >>>> it's >>>> config file, it just causes rsyslog to close and re-open it's outputs (a >>>> full restart takes a long time and can cause messages to be lost) >>>> >>> >>> >>> I did "service rsyslog restart" after the config change. "Kill timeout 5" >>> is set in /etc/init/rsyslog.conf. I'm not sure if this timeout setting >>> could make a difference. >>> >> > this should do it. but just to be sure, do a stop (make sure it's finished > shuttng down), then a start > > > >>>> >>>> We need the initial startup logs to be queued before remote logging >>>> >>>>> server >>>>>> >>>>>> is set. Switching from invalid IP to valid IP in rsyslog >>>>>>> configuration >>>>>>> was >>>>>>> chosen to meet this requirement. >>>>>>> >>>>>>> >>>>>>> Is there any chance of re-ordering the startup sequence to get the >>>>>> config >>>>>> first, then start rsyslog, then start everything else? kernel messages >>>>>> will >>>>>> get queued for quite a while, so they shouldn't be an issue. The only >>>>>> issue >>>>>> would be any other applications that need to write logs very early on. >>>>>> >>>>>> >>>>>> The problem is that we don't know remote logging server at startup, >>>>> so we >>>>> need the capability to buffer the logs until the remote server is set >>>>> by >>>>> user later. Understood that the logs could get lost after the disk >>>>> space >>>>> limit is reached. Is there any way to achieve this without rsyslog >>>>> configuration change? >>>>> >>>>> >>>> one possibility would be to just write the logs to a file and then use >>>> imfile to read this file later to send them upstream, but I'm not sure >>>> if >>>> imfile has gained the capability to get all it's data from the file yet. >>>> >>>> Historically, imfile only read the message content from the file, it >>>> generated the timestamp, hostname, priority, and severity information >>>> itself. I know there was talk about having an option to have imfile >>>> parse >>>> this from the file, but I don't know if it ever happened. >>>> >>>> If nothing else, you could write messages to a file with the >>>> RSYSLOG_ForwardFormat and then use netcat or similar to read the file >>>> and >>>> spit it out over the network later, but that wouldn't be able to use >>>> RELP >>>> to send it. I guess you could use netcat to send it to a UDP listener on >>>> localhost and then have the logs sent out via RELP from there. >>>> >>>> There should be some way to feed the logs to /dev/log, but I'm not sure >>>> exactly how to do that. >>>> >>>> Thanks for all your suggestions. Data completeness and integrity is very >>>> >>> important in our use cases. I'm not sure how some of the logging >>> information such as originial timestamp would change when it's routed >>> around. If this is confirmed to be a bug and can be fixed in 1-2 months, >>> I >>> would much rather to wait for the fix. >>> >> > Well, if you feed the data to syslog with the timestamp, it will preserve > the existing timestamp by default. > > David Lang > > Thanks, >>> Fajun >>> >>> >>>> >>>> On Wed, May 8, 2013 at 11:56 AM, David Lang <[email protected]> wrote: >>>> >>>>> >>>>>>> On Wed, 8 May 2013, Fajun Chen wrote: >>>>>>> >>>>>>> >>>>>>>> I upgraded ubuntu rsyslog to 7.3.14 and still got the same issue. >>>>>>>> >>>>>>>> >>>>>>>> My test procedure: >>>>>>>>> Clean log file. Set remote host IP to 127.255.255.1 (invalid IP) in >>>>>>>>> rsyslog conf. service rsyslog restart followed by logger in a >>>>>>>>> loop. The >>>>>>>>> disk queue files are buffered but are limited to 96M overall. Set >>>>>>>>> remote >>>>>>>>> host IP to valid IP. service rsyslog restart. I expect the queued >>>>>>>>> files to >>>>>>>>> be flushed to the remote host but these files are still in the >>>>>>>>> queuing >>>>>>>>> directory. >>>>>>>>> >>>>>>>>> >>>>>>>>> This may be a silly thought, but the fact that you are changing >>>>>>>>> the >>>>>>>>> >>>>>>>> configuration between these two steps could be part of the problem. >>>>>>>> >>>>>>>> I would suggest that instead of changing the config to >>>>>>>> enable/disable >>>>>>>> sending the logs that you instead keep the rsyslog config the same >>>>>>>> and set >>>>>>>> iptables rules to block and unblock the communications. >>>>>>>> >>>>>>>> ______________________________****_________________ >>>>>>> >>>>>> rsyslog mailing list >>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>> > >>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>> > >>>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>> >>> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
