On Tue, Sep 21, 2021 at 10:30:54PM -0400, Alex <mysqlstud...@gmail.com> wrote:
> Hi, > > > Why you shouldn’t log into db > > > > https://medium.com/@marton.waszlavik/why-you-shouldnt-log-into-db-e700c2cb0c8c > > > > I'm not suggesting that this person is correct, just > > mentioning it. After all, there are many successful > > companies with products that put staggering quantities > > of log data into some kind of database for operational > > analysis. They can't all be wrong, but they won't be > > using SQL databases. > > Yes, great point. Is something like mongodb better suited for doing > these things? That article only warned about combining write-heavy logging and read-heavy operational/transactional data in the same database. It advised that the logging for a read-heavy operational database should be sent elsewhere (like ELK or a separate database). But if the database's only function is to store logs, then it doesn't matter. Elasticsearch isn't a database exctly. It's more like just a lot of indices. And some people recommend not using it as your actual primary datasource for that reason. So MongoDB would be a good place for the data that Elasticsearch indexes. But it depends on how precious the logs are. And it depends on the scale. MySQL might be perfectly adequate for your needs, especially if logging is its only function. Perhaps the rsyslog+MySQL+LogAnalyzer approach will be appropriate. It sounds more like what you were asking for. But if you run a very busy bunch of mail servers, and there will lots of logs to store and analyze, an ELK stack, possibly with MongoDB or similar, might be more appropriate. > > Rsyslog is probably fine. I'm not aware of any web > > front-end, but I doubt that one is really needed. > > If your database is ready, it's probably just a > > matter of adding a few lines to its configuration file. > > > > If you are referring to a web front-end for examining > > the logs in the SQL database, you might want to rethink > > that and investigate the ELK Stack instead > > (Elasticsearch, Logstash, Kibana). It'll be overkill, > > but well worth learning, as it will be useful for all > > manner of log analysis needs. > > I meant a web front-end for querying historical data based on subject > or sender or "all emails sent between these dates" or "today's > rejected email to Joe" to see if an email he was expecting was instead > rejected. Chances are that the rsyslog+MySQL+LogAnalyzer approach will have a gentler learning curve, so maybe you should try that first, and see if it does what you need. If not, try the ELK stack which can probably do everything you can imagine. If you expect to need to analyse lots of logs from lots of different types of systems in future, it's probably worth the additional effort of learning to use the ELK stack. Also, the ELK stack will have a huge community of users which will be an invaluable resource. And it has its own conferences. LogAnalyzer won't have as many users, and there might not be much for dealing with Postfix logs specifically, but perhaps you can configure it to do what you need, whereas there are bound to be ELK stack users that analyze Postfix logs, so any plugins or configuration needed will have already been created. That might be reason enough to use the ELK stack, even if the scale doesn't warrant it. Bear in mind, I don't really know what I'm talking about. I never heard of LogAnalyzer before today, and I've never used the ELK stack myself. I've only heard good things about it. > Thanks, > Alex cheers, raf
