tlsmgr timeout
Hi, Can someone help me troubleshoot why I'm periodically receiving these messages? It results in postfix stop responding to connections altogether. Sep 21 19:18:41 xavier postfix-116/smtpd[2485484]: warning: problem talking to server private/tlsmgr: Connection timed out Sep 21 19:25:38 xavier postfix-117/smtpd[2491054]: warning: problem talking to server private/tlsmgr: Connection timed out This is a from a multi-instance postfix implementation. It does appear to occur on multiple instances. This is also a very infrequent occurrence, but I have no idea what happens leading up to it or what causes it. I have the following in master.cf for both instances: tlsmgrunix - - n 1000? 1 tlsmgr Ideas greatly appreciated. Thanks, Alex
logging directly to database?
Hi, I'm interested in having postfix log directly to a mariadb or mongodb database so I can then query it for different info like sender, recipient and subject, etc. Does anyone know the best way to go about doing this? Any particular reason output to sql hasn't been implemented in postfix already? I see there are several github projects available, but many haven't been updated in many years. Any guidance on the best implementation, pros/cons, and perhaps even an existing web front-end to do this would be greatly appreciated. Thanks, Alex
Re: logging directly to database?
I'm interested in having postfix log directly to a mariadb or mongodb database so I can then query it for different info like sender, recipient and subject, etc. Does anyone know the best way to go about doing this? I don't know if this is the best way, but one option is to send all mail logs from rsyslog to a custom script, have that script parse the log line and save it to your database. postfix -> rsyslog -> script -> sql For Redhat flavor OS: /etc/rsyslog.d/custom.conf: mail.* ^/etc/rsyslog.d/your_script_that_saves_to_sql
Re: logging directly to database?
On Tue, Sep 21, 2021 at 08:49:24PM -0400, Alex wrote: > Hi, > > I'm interested in having postfix log directly to a mariadb or mongodb > database so I can then query it for different info like sender, > recipient and subject, etc. Does anyone know the best way to go about > doing this? By default, postfix logs via the syslog protocol, so using a syslog-compatible logging service that can send log messages to a database would be the way to go. I searched for "syslog to database" and found: Writing syslog messages to MySQL, PostgreSQL or any other supported Database https://www.rsyslog.com/doc/v8-stable/tutorials/database.html The default syslog facility is "mail". You might want to change it to something like "local7" (or something you're not already using) with Postfix's syslog_facility parameter so that only postfix's log messages are sent to the database, rather than all mail-related log messages. Mind you, I also found this: Why you shouldn’t log into db https://medium.com/@marton.waszlavik/why-you-shouldnt-log-into-db-e700c2cb0c8c I'm not suggesting that this person is correct, just mentioning it. After all, there are many successful companies with products that put staggering quantities of log data into some kind of database for operational analysis. They can't all be wrong, but they won't be using SQL databases. > Any particular reason output to sql hasn't been implemented in postfix > already? Probably because it's a job best suited to a logging service. And SQL databases are only one of many possible destinations. What if someone preferred e.g. elasticsearch, and wanted to use something like logstash, fluentd, flume, splunk, graylog, etc. The only practical way for Postfix to support all of these things is to do what it's doing, and use the well-known syslog protocol, knowing that all logging systems will almost certainly support that protocol. Postfix can also log directly to file (mainly for use in containers), and many logging systems can probably(?) handle grabbing logs from a file as well. > I see there are several github projects available, but many haven't > been updated in many years. Any guidance on the best implementation, > pros/cons, and perhaps even an existing web front-end to do this would > be greatly appreciated. Rsyslog is probably fine. I'm not aware of any web front-end, but I doubt that one is really needed. If your database is ready, it's probably just a matter of adding a few lines to its configuration file. If you are referring to a web front-end for examining the logs in the SQL database, you might want to rethink that and investigate the ELK Stack instead (Elasticsearch, Logstash, Kibana). It'll be overkill, but well worth learning, as it will be useful for all manner of log analysis needs. > Thanks, > Alex cheers, raf
Re: logging directly to database?
Hi, > > I'm interested in having postfix log directly to a mariadb or mongodb > > database so I can then query it for different info like sender, > > recipient and subject, etc. Does anyone know the best way to go about > > doing this? > > I don't know if this is the best way, but one option is to send all mail > logs from rsyslog to a custom script, have that script parse the log > line and save it to your database. > > postfix -> rsyslog -> script -> sql > > For Redhat flavor OS: > /etc/rsyslog.d/custom.conf: > mail.* ^/etc/rsyslog.d/your_script_that_saves_to_sql Yes, thanks. I realize I can do that - it's the "your_script_that_saves_to_sql" part that would be very helpful :-) There's also great difficulty with correlating log lines from postscreen, smtpd. local, etc, and tracing all of the processes from the initial CONNECT through to the eventual delivery. Thanks, Alex
Re: logging directly to database?
Hi, > Why you shouldn’t log into db > > https://medium.com/@marton.waszlavik/why-you-shouldnt-log-into-db-e700c2cb0c8c > > I'm not suggesting that this person is correct, just > mentioning it. After all, there are many successful > companies with products that put staggering quantities > of log data into some kind of database for operational > analysis. They can't all be wrong, but they won't be > using SQL databases. Yes, great point. Is something like mongodb better suited for doing these things? > Rsyslog is probably fine. I'm not aware of any web > front-end, but I doubt that one is really needed. > If your database is ready, it's probably just a > matter of adding a few lines to its configuration file. > > If you are referring to a web front-end for examining > the logs in the SQL database, you might want to rethink > that and investigate the ELK Stack instead > (Elasticsearch, Logstash, Kibana). It'll be overkill, > but well worth learning, as it will be useful for all > manner of log analysis needs. I meant a web front-end for querying historical data based on subject or sender or "all emails sent between these dates" or "today's rejected email to Joe" to see if an email he was expecting was instead rejected. Thanks, Alex
Re: logging directly to database?
On Wed, Sep 22, 2021 at 12:24:25PM +1000, raf wrote: > On Tue, Sep 21, 2021 at 08:49:24PM -0400, Alex wrote: > > > Hi, > > > > I'm interested in having postfix log directly to a mariadb or mongodb > > database so I can then query it for different info like sender, > > recipient and subject, etc. Does anyone know the best way to go about > > doing this? > > By default, postfix logs via the syslog protocol, so > using a syslog-compatible logging service that can send > log messages to a database would be the way to go. I > searched for "syslog to database" and found: > > Writing syslog messages to MySQL, PostgreSQL or any other supported Database > https://www.rsyslog.com/doc/v8-stable/tutorials/database.html > > The default syslog facility is "mail". You might want > to change it to something like "local7" (or something > you're not already using) with Postfix's syslog_facility > parameter so that only postfix's log messages are sent > to the database, rather than all mail-related log > messages. > > Mind you, I also found this: > > Why you shouldn’t log into db > > https://medium.com/@marton.waszlavik/why-you-shouldnt-log-into-db-e700c2cb0c8c > > I'm not suggesting that this person is correct, just > mentioning it. After all, there are many successful > companies with products that put staggering quantities > of log data into some kind of database for operational > analysis. They can't all be wrong, but they won't be > using SQL databases. > > > Any particular reason output to sql hasn't been implemented in postfix > > already? > > Probably because it's a job best suited to a logging > service. And SQL databases are only one of many > possible destinations. What if someone preferred e.g. > elasticsearch, and wanted to use something like > logstash, fluentd, flume, splunk, graylog, etc. > > The only practical way for Postfix to support all of > these things is to do what it's doing, and use the > well-known syslog protocol, knowing that all logging > systems will almost certainly support that protocol. > Postfix can also log directly to file (mainly for use > in containers), and many logging systems can > probably(?) handle grabbing logs from a file as well. > > > I see there are several github projects available, but many haven't > > been updated in many years. Any guidance on the best implementation, > > pros/cons, and perhaps even an existing web front-end to do this would > > be greatly appreciated. > > Rsyslog is probably fine. I'm not aware of any web > front-end, but I doubt that one is really needed. > If your database is ready, it's probably just a > matter of adding a few lines to its configuration file. > > If you are referring to a web front-end for examining > the logs in the SQL database, you might want to rethink > that and investigate the ELK Stack instead > (Elasticsearch, Logstash, Kibana). It'll be overkill, > but well worth learning, as it will be useful for all > manner of log analysis needs. Here's another rsyslog+mysql approach that has a web interface (phpLogCon). It was in the web search I did earlier but I hadn't gotten to it yet: Set up rsyslog to store syslog messages in MySQL https://www.techrepublic.com/blog/linux-and-open-source/set-up-rsyslog-to-store-syslog-messages-in-mysql/ But phpLogCon has renamed to LogAnalyzer and moved: https://loganalyzer.adiscon.com/ cheers, raf
Re: tlsmgr timeout
On Tue, Sep 21, 2021 at 07:57:12PM -0400, Alex wrote: > Can someone help me troubleshoot why I'm periodically receiving these > messages? It results in postfix stop responding to connections > altogether. > > Sep 21 19:18:41 xavier postfix-116/smtpd[2485484]: warning: problem > talking to server private/tlsmgr: Connection timed out > Sep 21 19:25:38 xavier postfix-117/smtpd[2491054]: warning: problem > talking to server private/tlsmgr: Connection timed out What databases if any are using for TLS session caches? Are any of the instances inadvertently sharing their TLS session caches databases? smtp_tls_session_cache_database smtpd_tls_session_cache_database Also, what is the configure random source, and what OS? tls_random_source -- Viktor.
Re: logging directly to database?
On Tue, Sep 21, 2021 at 10:30:54PM -0400, Alex wrote: > Hi, > > > Why you shouldn’t log into db > > > > https://medium.com/@marton.waszlavik/why-you-shouldnt-log-into-db-e700c2cb0c8c > > > > I'm not suggesting that this person is correct, just > > mentioning it. After all, there are many successful > > companies with products that put staggering quantities > > of log data into some kind of database for operational > > analysis. They can't all be wrong, but they won't be > > using SQL databases. > > Yes, great point. Is something like mongodb better suited for doing > these things? That article only warned about combining write-heavy logging and read-heavy operational/transactional data in the same database. It advised that the logging for a read-heavy operational database should be sent elsewhere (like ELK or a separate database). But if the database's only function is to store logs, then it doesn't matter. Elasticsearch isn't a database exctly. It's more like just a lot of indices. And some people recommend not using it as your actual primary datasource for that reason. So MongoDB would be a good place for the data that Elasticsearch indexes. But it depends on how precious the logs are. And it depends on the scale. MySQL might be perfectly adequate for your needs, especially if logging is its only function. Perhaps the rsyslog+MySQL+LogAnalyzer approach will be appropriate. It sounds more like what you were asking for. But if you run a very busy bunch of mail servers, and there will lots of logs to store and analyze, an ELK stack, possibly with MongoDB or similar, might be more appropriate. > > Rsyslog is probably fine. I'm not aware of any web > > front-end, but I doubt that one is really needed. > > If your database is ready, it's probably just a > > matter of adding a few lines to its configuration file. > > > > If you are referring to a web front-end for examining > > the logs in the SQL database, you might want to rethink > > that and investigate the ELK Stack instead > > (Elasticsearch, Logstash, Kibana). It'll be overkill, > > but well worth learning, as it will be useful for all > > manner of log analysis needs. > > I meant a web front-end for querying historical data based on subject > or sender or "all emails sent between these dates" or "today's > rejected email to Joe" to see if an email he was expecting was instead > rejected. Chances are that the rsyslog+MySQL+LogAnalyzer approach will have a gentler learning curve, so maybe you should try that first, and see if it does what you need. If not, try the ELK stack which can probably do everything you can imagine. If you expect to need to analyse lots of logs from lots of different types of systems in future, it's probably worth the additional effort of learning to use the ELK stack. Also, the ELK stack will have a huge community of users which will be an invaluable resource. And it has its own conferences. LogAnalyzer won't have as many users, and there might not be much for dealing with Postfix logs specifically, but perhaps you can configure it to do what you need, whereas there are bound to be ELK stack users that analyze Postfix logs, so any plugins or configuration needed will have already been created. That might be reason enough to use the ELK stack, even if the scale doesn't warrant it. Bear in mind, I don't really know what I'm talking about. I never heard of LogAnalyzer before today, and I've never used the ELK stack myself. I've only heard good things about it. > Thanks, > Alex cheers, raf
