David, Thank you for your generosity in taking the time to write out such a thoughtful response. Wow.
You have given me some really interesting ideas to pursue. I am grateful. I am going to chew on this for a bit. I have some reading to do (including a few articles written by you!). Thanks again, and very warm regards, Jim -- Jim Van Meggelen ClearlyCore Inc. +1-416-639-6001 (DID) +1-877-253-2716 (Canada) +1-866-644-7729 (USA) +1-416-425-6111 x6001 [email protected] [ http://www.clearlycore.com/ | http://www.clearlycore.com ] Asterisk: The Definitive Guide FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] ----- Original Message ----- > From: "David Lang" <[email protected]> > To: "rsyslog-users" <[email protected]> > Cc: "Jim Van Meggelen" <[email protected]> > Sent: Friday, 9 July, 2021 18:38:08 > Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs > during development > multi-line logs are difficult to handle, it would be far easier on you if you > can turn them into single-line logs as early in processing as possible. > > There is a lot of business analytics value in logs. the 'easy' way is to throw > it into Splunk or ElasticSearch and depend on queries there, but that ends up > being rather inefficient. I like to get the logs into those tools to make them > easy to explore, but once you figure out what you want to know you can be far > more efficient in the gathering of your metrics. > > you can use something like Simple Event Correlator to turn a series of events > into counts that you can then graph, and once you have graphable numbers, then > something like the holt-winters algorithm that RRDtool implements can predict > normal values and alert you when you stray (and the beauty of holt-winters is > that the same numerical value can produce a 'unexpecteedly high' alert at 3am > sunday morning, 'unexpectedly low' at 10am monday, and be in the normal range > at > 3pm on monday) > > Rsyslog is not an analysis engine, but it's a very good routing/reformating > engine for single-line logs (it can do some handling of multi-line logs, but > that tends to just push the failure down to the next component) > > One thing to remember is that rsyslog is a 'best effort' logging, there are > ways > to make it handle failures, but there remain failures that can cause logs to > be > lost. Don't use rsyslog as the only path for content that will cost you money > if it's lost. > > https://www.usenix.org/publications/login/david-lang-series > https://www.usenix.org/publications/login/april14/lang > https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david > http://ristov.users.sourceforge.net/publications/cogsima15-sec-web.pdf > > David Lang > > > > On Fri, 9 Jul 2021, Jim Van Meggelen via rsyslog > wrote: > >> Date: Fri, 9 Jul 2021 07:42:28 -0500 (CDT) >> From: Jim Van Meggelen via rsyslog <[email protected]> >> To: rsyslog-users <[email protected]> >> Cc: Jim Van Meggelen <[email protected]> >> Subject: Re: [rsyslog] using Kibana / OpenSearch Dashboards to analyze logs >> during development >> >> Daniel, >> >> I'm pretty sure you and I have had at least one yap at some conference or >> another. Could be I just attended a talk of yours. >> >> I saw your name here and thought "I'm pretty sure I've met him somewhere", >> and >> that was somewhat of a pleasant shock, because I've been digging into rsyslog >> for some stuff I've been thinking about, and it's in a similar vein to what >> you're talking about here (feeling multi-line data into analytics to help >> make >> some sense of it), and frankly it's nice to hear someone else in the same >> line >> of work is thinking similar things with respect to these log files (which are >> chock full of detailed data). >> >> I don't know if what we're after is in fact the same (most folks seem to use >> logging for error handling, whereas I'm thinking more about gleaning business >> analytics from the data). >> >> It feels like there's gold in all those log files. It'd be interesting to see >> how it could be mined. >> >> Regards, >> >> Jim >> >> _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
