Is the action queue thread tied to the ruleset? -- James
----- Reply message ----- From: "David Lang" <[email protected]> To: "rsyslog-users" <[email protected]> Subject: [rsyslog] client connectivity issues syslog-ng -> rsyslog 7.x Date: Fri, Jun 14, 2013 6:13 pm I've found that FixedArray queues are noticably faster than LinkedList. That probably won't completely solve the problem, but it may help. If I'm seeing this correctly, this is the message template $template appLogHadoopTemplate,"<%PRI%>%TIMESTAMP% %FROMHOST% app=%programname:R,ERE,1,DFLT:([A-Za-z0-9]+)-.*-.*_.*--end%|bucket=%programname:R,ERE,1,DFLT:.*-.*-.*_([A-Za-z0-9]+)--end%%msg%\n" and this is the filename template $template appLogDynFile,"/log/app-logs/%**programname:R,ERE,0,DFLT:[A-Za-z0-9]+--end%/%FROMHOST%/%$**YEAR%/%$MONTH%/%$DAY%/%PROGRAMNAME%.log" Both of these are heavy users of regex parsing, which is a pretty expensive operation One thing that I think I'm seeing, it that you do the same regex multiple times. With an ugly regex like this, you would probably gain significantly by setting a variable and using that set $!shortname = "%programname:R,ERE,1,DFLT:([A-Za-z0-9]+)-.*-.*_.*--end%" $template appLogHadoopTemplate,"<%PRI%>%TIMESTAMP% %FROMHOST% app=%$!shortname%|bucket=%$!shortname%%msg%\n" the other thing is, can you simplify the regex? it looks like you are trying to pull any alphanumeric value ahead of a -, but if that's the case, why are you matching things after that? will the following give you what you need? set $!shortname = "%programname:R,ERE,1,DFLT:^([A-Za-z0-9]+)-%" I anchor the regex to the beginning of the string, and then pull everything before the - David Lang On Fri, 14 Jun 2013, Timothy Ehlers wrote: > Date: Fri, 14 Jun 2013 17:50:05 -0500 > From: Timothy Ehlers <[email protected]> > Reply-To: rsyslog-users <[email protected]> > To: rsyslog-users <[email protected]> > Subject: Re: [rsyslog] client connectivity issues syslog-ng -> rsyslog 7.x > > Yeah it helped in staging but with the higher volume of prod ruleset 2 > seems to be the problem... I still only see 1 thread and its pegged at 100% > > $Ruleset appLog > $ActionQueueType LinkedList > $ActionQueueWorkerThreads 64 > $ActionQueueWorkerThreadMinimumMessages 10000 > $ActionQueueSize 400000 > *.* > ?appLogDynFile;appLogHadoopTemplate > > There must be something i do not understand about the Queue system. > > > On Fri, Jun 14, 2013 at 4:45 PM, David Lang <[email protected]> wrote: > >> Interesting, I did not expect that action queues would help this much, >> given that you have the rulesets bound to different interfaces, I would >> have expected that their output processing would be independant. >> >> But the fact that putting in action queues (I assume one queue per >> ruleset??) splits up the work so much says that I was wrong. >> >> Which output is the action 3 queue that's using so much more CPU than >> anything else? >> >> >> David Lang >> >> On Fri, 14 Jun 2013, Timothy Ehlers wrote: >> >> I put ActionQueus into the config and in staging it looks better now. >>> 12773 root 20 0 1919m 231m 1848 R 34.8 0.5 0:19.06 rs:action 3 >>> que >>> 12772 root 20 0 1919m 231m 1848 S 10.6 0.5 0:09.51 rs:action 2 >>> que >>> 12751 root 20 0 1919m 231m 1848 S 1.7 0.5 0:01.29 rs:main Q:Reg >>> 12742 root 20 0 1919m 231m 1848 S 0.7 0.5 0:00.72 in:imtcp >>> 12767 root 20 0 1919m 231m 1848 S 0.7 0.5 0:00.21 rs:action 5 >>> que >>> 12774 root 20 0 1919m 231m 1848 S 0.7 0.5 0:00.70 rs:action 4 >>> que >>> >>> I will try this in production with the Higher volume after a peer review. >>> >>> >>> On Fri, Jun 14, 2013 at 4:32 PM, Timothy Ehlers <[email protected]> >>> wrote: >>> >>> 52mb/sec inbound traffic >>>> Hadoop stream is showing: 25k msg per second.. i do not know how accurate >>>> this is. >>>> >>>> $OptimizeForUniprocessor off >>>> $MaxMessageSize 2048k >>>> >>>> # Rsyslog plugins >>>> $ModLoad immark # provides --MARK-- message capability >>>> $ModLoad imudp # provides UDP syslog reception >>>> $ModLoad imtcp # provides TCP syslog reception >>>> $ModLoad imuxsock # provides support for local system logging (e.g. >>>> via logger command) >>>> $ModLoad imklog # provides kernel logging support (previously >>>> done >>>> by rklogd) >>>> $ModLoad imrelp # Provides RELP syslog reception >>>> $ModLoad omrelp # Provides RELP syslog transmission >>>> >>>> # Rsyslog Stats >>>> $ModLoad impstats >>>> $PStatInterval 60 >>>> $PStatSeverity 7 >>>> >>>> # Queue configuration >>>> $ActionQueueSize 2000000 >>>> $MainMsgQueueSize 40000000 >>>> >>>> # File Creation Permissions >>>> $umask 0000 >>>> $DirCreateMode 0755 >>>> $FileCreateMode 0644 >>>> >>>> # Remote Log Processing Ruleset >>>> $PreserveFQDN on >>>> $template >>>> appLogDynFile,"/log/app-logs/%**programname:R,ERE,0,DFLT:[A-** >>>> Za-z0-9]+--end%/%FROMHOST%/%$**YEAR%/%$MONTH%/%$DAY%/%** >>>> PROGRAMNAME%.log" >>>> $template >>>> currLogStatsDynFile,"/log/app-**logs/logstats/%FROMHOST%/%$** >>>> YEAR%/%$MONTH%/%$DAY%/**logstats.log.%$HOUR%00" >>>> $template >>>> currAppLogDynFile,"/log/app-**logs/%msg:R,ERE,1,DFLT:^([A-** >>>> Za-z0-9._-]+)\|([A-Za-z0-9._-]**+)\|([A-Za-z0-9._]+)[-_]*([A-** >>>> Za-z0-9]*)([\^])--end%/%**FROMHOST%/%$YEAR%/%$MONTH%/%$** >>>> DAY%/%msg:R,ERE,1,DFLT:^([A-**Za-z0-9._-]+)\|([A-Za-z0-9._-]** >>>> +)\|([A-Za-z0-9._]+)[-_]*([A-**Za-z0-9]*)([\^])--end%-%msg:R,** >>>> ERE,2,DFLT:^([A-Za-z0-9._-]+)\**|([A-Za-z0-9._-]+)\|([A-Za-z0-** >>>> 9._]+)[-_]*([A-Za-z0-9]*)([\^]**)--end%-%msg:R,ERE,3,DFLT:^([** >>>> A-Za-z0-9._-]+)\|([A-Za-z0-9._**-]+)\|([A-Za-z0-9._]+)[-_]*([** >>>> A-Za-z0-9]*)([\^])--end%-%msg:**R,ERE,4,DFLT:^([A-Za-z0-9._-]+** >>>> )\|([A-Za-z0-9._-]+)\|([A-Za-**z0-9._]+)[-_]*([A-Za-z0-9]*)([** >>>> \^])--end%.log.%$HOUR%00" >>>> $template >>>> currAppLoggTemplate,"%msg:R,**ERE,1,DFLT:^[A-Za-z0-9._-]+\|[** >>>> A-Za-z0-9._-]+\|[A-Za-z0-9._]+**[-_]*[A-Za-z0-9]*[\^](.*)--**end%\n" >>>> $template currAppLoggTemplate2,"%msg%\n" >>>> $template currentappLogHadoopTemplate,"<**%PRI%>%TIMESTAMP:date-rfc3164% >>>> %FROMHOST% %msg%\n" >>>> $template currentappLogNewHadoopTemplate**,"<%PRI%>%TIMESTAMP% >>>> %FROMHOST% >>>> app=%msg:R,ERE,1,DFLT:^([A-Za-**z0-9._-]+)\|([A-Za-z0-9._-]+)\** >>>> |([A-Za-z0-9.]+)[-_]*([A-Za-**z0-9]*)--end%|bucket=%msg:R,** >>>> ERE,4,DFLT:^([A-Za-z0-9._-]+)\**|([A-Za-z0-9._-]+)\|([A-Za-z0-** >>>> 9.]+)[-_]*([A-Za-z0-9]*)--end% >>>> %msg%\n" >>>> $template appLogHadoopTemplate,"<%PRI%>%**TIMESTAMP% %FROMHOST% >>>> app=%programname:R,ERE,1,DFLT:**([A-Za-z0-9]+)-.*-.*_.*--end%|** >>>> bucket=%programname:R,ERE,1,**DFLT:.*-.*-.*_([A-Za-z0-9]+)--** >>>> end%%msg%\n" >>>> $template >>>> remoteMessagesDynFile,"/log/**system-logs/%FROMHOST%/%$YEAR%** >>>> /%$MONTH%/%$DAY%/messages" >>>> $template >>>> remoteSecureDynFile,"/log/**secure-system-logs/%FROMHOST%/** >>>> %$YEAR%/%$MONTH%/%$DAY%/**secure" >>>> $template >>>> remoteMaillogDynFile,"/log/**system-logs/%FROMHOST%/%$YEAR%** >>>> /%$MONTH%/%$DAY%/maillog" >>>> $template >>>> remoteEmergDynFile,"/log/**system-logs/%FROMHOST%/%$YEAR%** >>>> /%$MONTH%/%$DAY%/emergency" >>>> $template >>>> remoteCronDynFile,"/log/**system-logs/%FROMHOST%/%$YEAR%** >>>> /%$MONTH%/%$DAY%/cron" >>>> $template >>>> remoteSpoolerDynFile,"/log/**system-logs/%FROMHOST%/%$YEAR%** >>>> /%$MONTH%/%$DAY%/spooler" >>>> $template >>>> remoteBootDynFile,"/log/**system-logs/%FROMHOST%/%$YEAR%** >>>> /%$MONTH%/%$DAY%/boot.log" >>>> >>>> $Ruleset appLog >>>> *.* >>>> ?appLogDynFile;**appLogHadoopTemplate >>>> # Forward to Hadoop >>>> #*.* @@ >>>> wmhdcollector01s.stag.**timstesting.net:5003<http://wmhdcollector01s.stag.timstesting.net:5003> >>>> ; >>>> >>>> $Ruleset currAppLog >>>> *.* >>>> ?currAppLogDynFile;**currAppLoggTemplate >>>> # Forward to Hadoop >>>> *.* >>>> @@hadoopcollectors.prod.**timstesting.net:5003;** >>>> currentappLogHadoopTemplate >>>> >>>> $Ruleset currLogStats >>>> *.* ?currLogStatsDynFile >>>> # Forward to Hadoop >>>> #*.* >>>> @@hadoopcollectors.prod.**timstesting.net:5003;** >>>> currentappLogHadoopTemplate >>>> >>>> # Remote System Log Processing Ruleset >>>> $Ruleset remoteSysLogs >>>> # Log all kernel messages to the console. >>>> # Logging much else clutters up the screen. >>>> #kern.* /dev/console >>>> >>>> # Log anything (except mail) of level info or higher. >>>> # Don't log private authentication messages! >>>> $DirCreateMode 0755 >>>> $FileCreateMode 0644 >>>> *.info;local1.none;local6.**none;mail.none;authpriv.none;**cron.none >>>> ?remoteMessagesDynFile >>>> >>>> # The authpriv file has restricted access. >>>> $DirCreateMode 0700 >>>> $FileCreateMode 0600 >>>> authpriv.* >>>> ?remoteSecureDynFile >>>> >>>> # Log all the mail messages in one place. >>>> $DirCreateMode 0755 >>>> $FileCreateMode 0644 >>>> mail.* >>>> ?remoteMaillogDynFile >>>> >>>> >>>> # Log cron stuff >>>> $DirCreateMode 0755 >>>> $FileCreateMode 0644 >>>> cron.* >>>> ?remoteCronDynFile >>>> >>>> # Everybody gets emergency messages >>>> $DirCreateMode 0755 >>>> $FileCreateMode 0644 >>>> *.emerg >>>> ?remoteEmergDynFile >>>> >>>> # Save news errors of level crit and higher in a special file. >>>> $DirCreateMode 0755 >>>> $FileCreateMode 0644 >>>> uucp,news.crit >>>> ?remoteSpoolerDynFile >>>> >>>> # Save boot messages also to boot.log >>>> $DirCreateMode 0755 >>>> $FileCreateMode 0644 >>>> local7.* >>>> ?remoteBootDynFile >>>> >>>> # Local Log Processing Ruleset >>>> $Ruleset local >>>> # Log all kernel messages to the console. >>>> # Logging much else clutters up the screen. >>>> #kern.* /dev/console >>>> >>>> # Log anything (except mail) of level info or higher. >>>> # Don't log private authentication messages! >>>> *.info;local1.none;local6.**none;mail.none;authpriv.none;**cron.none >>>> /var/log/messages >>>> syslog.=debug >>>> /log/rsyslog-stats >>>> >>>> # The authpriv file has restricted access. >>>> authpriv.* /var/log/secure >>>> >>>> # Log all the mail messages in one place. >>>> mail.* -/var/log/maillog >>>> >>>> >>>> # Log cron stuff >>>> cron.* /var/log/cron >>>> >>>> # Everybody gets emergency messages >>>> *.emerg :omusrmsg:* >>>> >>>> # Save news errors of level crit and higher in a special file. >>>> uucp,news.crit /var/log/spooler >>>> >>>> # Save boot messages also to boot.log >>>> local7.* /var/log/boot.log >>>> >>>> # Assign default Ruleset >>>> $DefaultRuleset local >>>> >>>> # New AppLog Process RELP Collector >>>> $InputRELPServerBindRuleset appLog >>>> $InputRELPServerRun 20514 >>>> >>>> # Current AppLog TCP Collector >>>> $InputTCPServerBindRuleset currAppLog >>>> $InputTCPServerRun 20516 >>>> >>>> # Current LogStats TCP Collector >>>> $InputTCPServerBindRuleset currLogStats >>>> $InputTCPServerRun 20518 >>>> >>>> # SystemLog TCP Collector >>>> $InputTCPServerBindRuleset remoteSysLogs >>>> $InputTCPServerRun 20515 >>>> >>>> # SystemLog UDP Collector >>>> $InputUDPServerBindRuleset remoteSysLogs >>>> $UDPServerRun 514 >>>> >>>> >>> >>> >>> >>> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
