On Wed, Jun 12, 2013 at 10:42 PM, David Lang <[email protected]> wrote:
> On Wed, 12 Jun 2013, Roberto Giordani wrote:
>
> Hello David,
>>
>> your optimization is the right way, Ican use only 1 if statement for
>> client!!!
>> The last step is something like this that doesn't work for the reasons
>> below
>>
>> if $fromhost-ip == "10.10.1.7" then {
>> set $!path="PROD/APP";
>> set $!server="client1";
>> set $!owner="robert";
>> }
>>
>> if $fromhost-ip == "10.10.1.8" then {
>> set $!path="PROD/APP";
>> set $!server="client2";
>> set $!owner="robert";
>> }
>>
>> if $fromhost-ip == "10.10.2.10" then {
>> set $!path="PROD/WEB";
>> set $!server="client7";
>> set $!owner="tom";
>> }
>>
>> if $fromhost-ip == "10.10.2.11" then {
>> set $!path="PROD/WEB";
>> set $!server="client8";
>> set $!owner="tom";
>> }
>>
>> template(name="filename" type="string" stringB="/rsyslog-data/%$!**
>> path%/%$!server%/%syslogtag%.**log")
>>
>> local6.debug action(type="omfile" DirOwner="%$!owner%"
>> DirCreateMode="0750" FileCreateMode="0444" DynaFile="filename" )
>>
>> ------------------------------**------------------------------**
>> -----------
>>
>> As you can see I need to use the OWNER variable to assign the right owner
>> to the output file.
>> Unfortunately:
>> 1) I can't insert TEMPLATE into IF
>>
>
> Correct, but you should not need to, by defining the path variable and
> then defining the template later it should work.
>
>
yup - I don't see any need to conditionally define templates. Define them,
use them based on condition.
>
> 2) I can't use the OWNER variable into ACTION
>>
>
> This sounds like it's a bug, we'll need to have Rainer take a look at that.
This feature is simply not present (and was never before asked for). It
also has quite some runtime implications, as the lookup is costly. So some
kind of caching needs to be put in place. In other words: not something to
quickly add.
>
>
> You told me about output ruleset but reading documentation it seems it
>> doesn't useful for my purpose.
>>
>
> well, they would possibly be useful to short-circuit the long series of if
> statements.
>
>
a word on ruleset use: it is NOT primarily an optimization tool (it was in
previous releases). It is a tool for complex configurations, where you
would either like to have something like subroutines or - and this is
probably the most important use case - where you want to bind listeners to
specific rulesets to keep message processing simple.
> even without resolving the owner you could do:
>
>
> template(name="filename" type="string" string="/rsyslog-data/%$!path%**
> /%$!server%/%syslogtag%.log")
>
> if $fromhost-ip == "10.10.1.7" then {
> set $!path="PROD/APP";
> set $!server="client1";
> local6.debug action(type="omfile" DirOwner="robert"
> DirCreateMode="0750" FileCreateMode="0444" DynaFile="filename" )
> stop
>
> }
>
> if $fromhost-ip == "10.10.1.8" then {
> set $!path="PROD/APP";
> set $!server="client2";
> set $!owner="robert";
> local6.debug action(type="omfile" DirOwner="robert"
> DirCreateMode="0750" FileCreateMode="0444" DynaFile="filename" )
> stop
>
> }
>
> if $fromhost-ip == "10.10.2.10" then {
> set $!path="PROD/WEB";
> set $!server="client7";
> local6.debug action(type="omfile" DirOwner="tom"
> DirCreateMode="0750" FileCreateMode="0444" DynaFile="filename" )
> stop
>
> }
>
> if $fromhost-ip == "10.10.2.11" then {
> set $!path="PROD/WEB";
> set $!server="client8";
> local6.debug action(type="omfile" DirOwner="tom"
> DirCreateMode="0750" FileCreateMode="0444" DynaFile="filename" )
> stop
> }
>
>
speaking performance-wise "else" is your freind. if one of those ifs
matches, proper use of if ... then ... else if ... reduces processing time.
But the "stop" in here alread takes care of this. Just wanted to mention
(for other cases where you cannot stop message processing).
> you could replace the repeated local6.debug statements with
> call(ruleset_robert) or call(ruleset_tom)
>
>
yup, that's a use case!
> that would look cleaner, but not be any faster than this configuration.
>
>
right. With 7.4 the performance toll is minimal (maybe 50 CPU cycles).
> This configuration will be a bit faster than your original in that logs
> from client1 only go through one fromhost-ip test, the 'stop' causes
> rsyslog to skip all the remaining tests.
>
I am still trying to find some more time to look in-depth at this. With
sufficient different inbound connections, core utilization is usually much
more even. I wonder why not here...
Rainer
>
> David Lang
>
>
> Resolving "owner" variable I can do a stress test.
>>
>> Thanks.
>>
>>
>> Roberto.
>> On 06/11/2013 10:12 PM, David Lang wrote:
>>
>>> On Tue, 11 Jun 2013, David Lang wrote:
>>>
>>> On Tue, 11 Jun 2013, Roberto Giordani wrote:
>>>>
>>>> Hello David,
>>>>> here there are the first actions for each client conf used by rsyslog
>>>>> server
>>>>>
>>>>> if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_server_01_Server" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client01/**PROD_server_01_Server.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_01A_R" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client01/**PROD_SRV_server_01_R.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_01B_R" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client01/**PROD_SRV_server_01_R.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_01A_S" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client01/**PROD_SRV_server_01_S.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.7" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_01B_S" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client01/**PROD_SRV_server_01_S.log")
>>>>> .
>>>>> .
>>>>> .
>>>>> .
>>>>> ------------------------------**------------
>>>>>
>>>>> if $fromhost-ip == "10.10.1.8" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_server_02_Server" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client02/**PROD_server_02_Server.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.8" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_02A_R" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client02/**PROD_SRV_server_02_R.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.8" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_02B_R" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client02/**PROD_SRV_server_02_R.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.8" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_02A_S" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client02/**PROD_SRV_server_02_S.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.8" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_02B_S" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client02/**PROD_SRV_server_02_S.log")
>>>>> .
>>>>> .
>>>>> .
>>>>> .
>>>>> ------------------------------**-----------------
>>>>>
>>>>> if $fromhost-ip == "10.10.1.30" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_server_23_Server" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client23/**PROD_server_23_Server.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.30" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_23A_R" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client23/**PROD_SRV_server_23_R.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.30" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_23B_R" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client23/**PROD_SRV_server_23_R.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.30" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_23A_S" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client23/**PROD_SRV_server_23_S.log")
>>>>>
>>>>> if $fromhost-ip == "10.10.1.30" and $syslogfacility-text == "local6"
>>>>> and $syslogseverity-text == "debug" and $syslogtag ==
>>>>> "PROD_SRV_server_23B_S" then action(type="omfile" DirOwner="robert"
>>>>> DirCreateMode="0750" FileCreateMode="0444"
>>>>> File="/rsyslog-data/client23/**PROD_SRV_server_23_S.log")
>>>>> .
>>>>> .
>>>>> .
>>>>> .
>>>>> and so on to 480 actions
>>>>>
>>>>> On your benchmark test which is the best ratio between #messages/s
>>>>> and #actions?
>>>>>
>>>>
>>>> it's not that there is a best ratio, it's just that there is overhead
>>>> to actions, overhead to if statements, etc
>>>>
>>>> As you have it written, you have very complex conditions for each action
>>>>
>>>> I think you would be better off re-writing your ruleset something
>>>> along these lines
>>>>
>>>> if $fromhost='10.10.1.7' then set $!client='client01'
>>>> if $fromhost='10.10.1.8' then set $!client='client02'
>>>> if $fromhost='10.10.1.9' then set $!client='client03'
>>>> if $fromhost='10.10.1.10' then set $!client='client04'
>>>> .
>>>> .
>>>> .
>>>>
>>>> template(name='filename' type="string"
>>>> stringB="/rsyslog-data/%$!**client%/%syslogtag%.log"
>>>>
>>>> local6.debug action(type="omfile" DirOwner="robert"
>>>> DirCreateMode="0750" FileCreateMode="0444" DynaFile="filename" )
>>>>
>>>>
>>>> now, this is not going to give the exact same results (your ruleset
>>>> won't log anything if the syslogtag is not one of the known ones,
>>>> while mine will), but if that's a problem, a second set of if
>>>> statements to set another variable would satisfy that.
>>>>
>>>> It may end up making sense to create a ruleset "output" that contains
>>>> the action line and then change the if statements to
>>>>
>>>> if $fromhost='10.10.1.7' then {
>>>> set $!client='client01'
>>>> call output
>>>> stop
>>>> }
>>>>
>>>> this would short circuit the tests, but add the overhead of a ruleset
>>>> call. I don't know if 27 rules is enough to justify this.
>>>>
>>>> What you really want is the table lookup function, but that hasn't
>>>> been funded yet :-(
>>>>
>>>> it would reduce your config to
>>>>
>>>> set $!client = lookup("clienttable", "%fromhost-ip")
>>>> set $!tag = lookup("tagtable", "%syslogtag")
>>>> template(name='filename' type="string"
>>>> stringB="/rsyslog-data/%$!**client%/%$!tag%.log"
>>>> local6.debug action(type="omfile" DirOwner="robert"
>>>> DirCreateMode="0750" FileCreateMode="0444" DynaFile="filename" )
>>>>
>>>
>>> One other optimization that I would look at is to use arrays. this is
>>> documented at
>>> http://www.rsyslog.com/tag/**dynafile/<http://www.rsyslog.com/tag/dynafile/>
>>>
>>> This solves the problem of odd syslogtag values.
>>>
>>> As far as the names go, I would suggest that you consider doing some
>>> name resolution games.
>>>
>>> If you setup a special DNS server (or hosts file) to map the IPaddresses
>>> to the client01 names that you want, allow rsyslog to do the DNS lookups
>>> on the incoming logs (with TCP this shouldn't be too bad as rsyslog
>>> should only have to do this once per connection), and then you could use
>>> the template:
>>>
>>> template(name='filename' type="string"
>>> string="/rsyslog-data/%**fromhost%/%syslogtag%.log"
>>>
>>> and completely eliminate the if statements to define the client name.
>>>
>>> DNS is something to avoid, but not at the cost of a ton of if statements
>>> in your filter.
>>>
>>> David Lang
>>>
>>
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
