multi-line logs are difficult to handle, it would be far easier on you if you
can turn them into single-line logs as early in processing as possible.
There is a lot of business analytics value in logs. the 'easy' way is to throw
it into Splunk or ElasticSearch and depend on queries there, but that ends up
being rather inefficient. I like to get the logs into those tools to make them
easy to explore, but once you figure out what you want to know you can be far
more efficient in the gathering of your metrics.
you can use something like Simple Event Correlator to turn a series of events
into counts that you can then graph, and once you have graphable numbers, then
something like the holt-winters algorithm that RRDtool implements can predict
normal values and alert you when you stray (and the beauty of holt-winters is
that the same numerical value can produce a 'unexpecteedly high' alert at 3am
sunday morning, 'unexpectedly low' at 10am monday, and be in the normal range at
3pm on monday)
Rsyslog is not an analysis engine, but it's a very good routing/reformating
engine for single-line logs (it can do some handling of multi-line logs, but
that tends to just push the failure down to the next component)
One thing to remember is that rsyslog is a 'best effort' logging, there are ways
to make it handle failures, but there remain failures that can cause logs to be
lost. Don't use rsyslog as the only path for content that will cost you money
if it's lost.
https://www.usenix.org/publications/login/david-lang-series
https://www.usenix.org/publications/login/april14/lang
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
http://ristov.users.sourceforge.net/publications/cogsima15-sec-web.pdf
David Lang
On Fri, 9 Jul 2021, Jim Van Meggelen via rsyslog
wrote:
Date: Fri, 9 Jul 2021 07:42:28 -0500 (CDT)
From: Jim Van Meggelen via rsyslog <[email protected]>
To: rsyslog-users <[email protected]>
Cc: Jim Van Meggelen <[email protected]>
Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs
during development
Daniel,
I'm pretty sure you and I have had at least one yap at some conference or
another. Could be I just attended a talk of yours.
I saw your name here and thought "I'm pretty sure I've met him somewhere", and
that was somewhat of a pleasant shock, because I've been digging into rsyslog for some
stuff I've been thinking about, and it's in a similar vein to what you're talking about
here (feeling multi-line data into analytics to help make some sense of it), and frankly
it's nice to hear someone else in the same line of work is thinking similar things with
respect to these log files (which are chock full of detailed data).
I don't know if what we're after is in fact the same (most folks seem to use
logging for error handling, whereas I'm thinking more about gleaning business
analytics from the data).
It feels like there's gold in all those log files. It'd be interesting to see
how it could be mined.
Regards,
Jim
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
