Hi David,
documentation is definitely a sore point of rsyslog (and most other
opensource projects for that matter).
Writing good documentation is hard.
A notable exception I remember (even if its is obsolete by now), is
upstart which shipped its upstart cookbook.
The original documentation is unfortunately no longer available, only
a wayback snapshot
https://web.archive.org/web/20230322064449/https://upstart.ubuntu.com/cookbook/
I personally like this style of documentation.
Am Do., 2. Nov. 2023 um 12:41 Uhr schrieb David Lang via rsyslog
<rsyslog@lists.adiscon.com>:
>
> We have received complaints about rsyslog documentation repeatedly, We have
> a lot of detail, but it's all written for someone already fairly familiar
> with things.
>
> Here is a 3am first pass from me at writing an overview of how rsyslog works,
> with the idea that this could be made pretty with diagrams, click through
> links
> to more specific pages with detail, etc.
>
> I'm replying to the github issue to see if the user who complained about the
> documentation and RainerScript would find this more useful, but also to
> rsyslog-users to get feedback from others on this.
>
> some of the sections here should possibly be broken into sub-pages (some
> sub-pages already exist that cover some of these and can/should be
> simplified),
> or it make make sense to have a simple version on an overview page with the
> ability to click down for the gory details.
>
> David Lang
>
>
>
> Rsyslog architecture is very straightforward, but in it's simplicity it hides
> a lot of flexibility.
>
> Rsyslog has one or more inputs that each receive one or more messages and
> pass the batch of messages to a ruleset
>
> Each input runs the incoming log through a stack of possible parser modules
> until it hits one that reports success in parsing the log (pointer to parser
> module documentation and the default stack)
>
> Multiple inputs can feed to the same ruleset (by default, all inputs feed
> to
> the Default ruleset which uses the 'main' queue) [1]
>
> Worker threads pull batches of logs from a queue, then process the logs in the
> batch using the statements in a ruleset
>
> Conceptually, it really is that trivial. As always, looking at details makes
> it
> seem more complicated.
>
>
>
> Rsyslog config file(s)
>
> Rsyslog reads in the config file and all included files and combines them
> before
> evaluating anything (see -o option for how files are combined), which file a
> statement is in has no impact (other than as part of the ordering of
> statements).
> (insert link to Rainer's recent post on mis-use of config includes??)
>
> At startup time, Rsyslog evaluates the combined config and implements all
> module
> loading, input definition, template definitions and other global settings.
>
> All other statements get put into the default ruleset unless a ruleset is
> specified. None of these statements are evaluated (beyond syntax checking) at
> startup.
>
> The Rsyslog team believes very strongly in maintaining backwards compatibility
> (a config that works should never break or change behavior when rsyslog is
> updated to a new version) as such there are multiple ways of doing the same
> thing, and some ways are no longer recommended. When you see that something
> is depriciated, that means it is recommended not to use it in a new
> configuration for confusion/feature reasons, not that it is scheduled to go
> away/break in a new version.
>
> The config statments that existed prior to v6 of rsyslog were an evolution of
> the syslog format from the 90's, doing complex things by setting a bunch of
> values that then got used by later statements. By v5 of rsyslog, this was
> resulting in such complex interactions that even core developers were having
> trouble understanding what complex configs did. V6 introduced RainerScript,
> which deliberately requires you to specify all options rather than
> 'inheriting'
> them from prior statements. This can be significantly more verbose as it
> requires you to specify all options each time, but makes it much clearer
> exactly
> what is happening. There are times when the old syntax is shorter and more
> obvious to use than the new syntax, and in those cases, it's recommended to
> use
> the old syntax. But if the old syntax requires multiple lines to do something,
> you are probably better off using the new syntax.
>
>
>
> Rulesets are the heart of log processing, defining what happens with each log
> message. The statements in a ruleset are evaluated for every log message as it
> is processed.
>
> Rulesets and Actions can have a queue defined for them (insert link to queue
> turn lane post, possibnly with updates). The 'default' ruleset uses the 'main'
> queue.
>
> The contents of a ruleset are a series of statements, which can be:
> 1. call an action to use an output module
> 1a. legacy formats:
> /var/log/messages (write to a file)
> @1.2.3.4 (send to a remote system via UDP
> @@1.2.3.4 (send to a remote system via TCP)
> 1b. action() format
> 2. set/clear variables (link to functions)
> 3. call a message modification module (can modify the log message being
> processed and set variables) commonly used to parse messages
> 4. call another ruleset and then retur
> 5. statement block
> { statement statement }
> usually used after a filter to have the filter apply to multiple
> statements
> & <statememt>
> apply the last filter to this statement [2]
> 6. stop processing this log message
> 6a. ~ [2]
> 6b. stop
> ignore all following statements in the ruleset. Rsyslog will warn you
> if you have statements after an unconditional stop
> 7. apply a filter
> 7a. legacy syslog facility.serverity filters
> i.e. mail.info /var/log/mail
> 7b. rsyslog property based filters [2]
> i.e. :msg, contains, "foo" <statemnent>
> 7c. expression based filters (if..then..else with continue) (link to
> functions and conditionals)
> i.d. if $msg contains("foo") then <statement>
> 8. atomic stats update (see impstats module)
> 9. foreach execute a block of statements on each value in a variable array
>
>
> Variable types:
> built-in/legacy properties start with $ or $$ (link to property page)
> user modifyable variables exist as a tree internally represented as a json
> structure. There are three trees that can be used:
> "normal variables" start with $!
> "local variables" start with $. (exist so that you can include all $!
> variables in a template without including everything)
> "global variables" start with $\ (persist past the log message where
> they were set, performance pigs)
>
> Templates are used by output modules. They are used to create larger strings
> that use variable values for use with the module. These allow you to change
> the format of the output, what file or database table the log gets written to
> and other similar things. The details of what the result of the template
> means varies from one module to another. There is a common misconception that
> a template can be used to match and parse a log message. Templates are output
> only.
>
>
>
> [1] it may make sense to have a 'are you sure' message at startup about inputs
> that feed to rulesets that don't have a queue defined for the ruleset. I don't
> think a new warning would be a breaking change
>
> [2] supported for backwards compatibility, use is discouraged
>
> (note, there should possibly be two versions of this, one showing the
> straighforward, single-message-at-a-time process, and a second one that shows
> the advanced, batch supporting, version that includes showing where locking
> happens, the atomic stats and foreach would be in the advanced version)
>
>
> On Wed, 1 Nov 2023, computerquip-work wrote:
>
> > The documentation is so poor that it's almost unusable. Using RainerScript
> > is hands down the most painful thing I've ever used in software.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
