Re: Can't log in from Evolution or Roundcube
> Op 8 jan. 2022 om 06:30 heeft Ken Wright het > volgende geschreven: > > On Fri, 2022-01-07 at 23:59 -0500, Dave McGuire wrote: >> On 1/7/22 11:58 PM, Ken Wright wrote: > When I try to log in to my mail server (ubuntu 20.04, Postfix > 3.4.13, Dovecot 2.3.7.2) I get a response saying "Source stream > returned no data”. At least to me, that's not particularly > informative. Is it any more informative to anyone else? The last time I hit that, I'm pretty sure it was because I was going to port 80 instead of port 443 to reach Roundcube. >>> >>> I'm using port 143 for receiving and 587 for sending; I didn't >>> think 443 was for email. Am I mistaken? (Not at all unlikely!) >> >>Nono, for your web browser's connection to Roundcube. I could be >> barking up the wrong tree here, but I'm pretty sure that's the error >> I hit. > > Thanks for the clarification. I just tried Roundcube again, and got > the error "Connection to storage server failed." If I’m not mistaken, that means Roundcube couldn’t connect to the mail server. There can be a million reasons for that, of course. Roundcube doesn’t show the actual error to the user, but does log it somewhere. > I also checked the > nginx script for Roundcube and commented out the references to port 80, > then restarted nginx. Same error. So I tend to think it's a server > issue, not a client issue. Does that make any sense? > > Ken > >
Re: Non-user logins?
Am 08.01.22 um 05:27 schrieb Dave McGuire: > trying to mess with other peoples' stuff. I run fail2ban to catch those > log entries and block the source IP address for a month on the first > failed login. At any one time I have between 12,000 and 15,000 well, I don't know how _your_ users are connected to the internet, but in germany most people has at least daily changing IPs out of larger pools (when connected via xDSL) or even sometimes shares ip-addresses with others (when connected via tv-cable or mobile - having a private network-address, which is natted), so it's possible to get/use an IP, which was used before by some script-kiddies... so everyone, who's blocking such requests for more than some minutes/hours should be aware of maybe blocking legitimate user-logins... btw.: setting up a new mail-client and making any mistake by reading it from old install or writing it into new install also leads to a months-blocking with above restrictive handling... (any may drive this user mad) so anyone, who has no experience with blocking should be really careful with it. d.
Re: Non-user logins?
On 08/01/2022 14:26, dc...@dvl.werbittewas.de wrote: Am 08.01.22 um 05:27 schrieb Dave McGuire: trying to mess with other peoples' stuff. I run fail2ban to catch those log entries and block the source IP address for a month on the first failed login. At any one time I have between 12,000 and 15,000 well, I don't know how _your_ users are connected to the internet, but in germany most people has at least daily changing IPs out of larger pools (when connected via xDSL) or even sometimes shares ip-addresses with others (when connected via tv-cable or mobile - having a private network-address, which is natted), so it's possible to get/use an IP, which was used before by some script-kiddies... so everyone, who's blocking such requests for more than some minutes/hours should be aware of maybe blocking legitimate user-logins... btw.: setting up a new mail-client and making any mistake by reading it from old install or writing it into new install also leads to a months-blocking with above restrictive handling... (any may drive this user mad) so anyone, who has no experience with blocking should be really careful with it. d. yes, blocking on the first wrong password sounds like overkill. But it does depend on user base. For a small mail server with few known users it could be workable. But even on small servers I'd recommend blocking for a small time (like up to an hour) after a small number of failures (example 3). Then if this pattern repeats (for example 3 times) within a longer period (for example up to a day), blocking for a longer period (example 1 week) using the recidive jail. Mileage will vary depending on user base and number of support requests generated. The point about fail2ban is that it slows down attackers stopping infinite and fast repeating attacks from the same ip. That should be in combination with a good password policy which reduces the probability of any single attack guessing the password. It doesn't necessarily have to zero out attacks. As Dave has experimented, to bypass fail2ban all the attacker has to do is use a different ip. 10-15K blocks in place at any time seem very high compared to the few attacks I see. I'd hazard a guess that the restrictive fail2ban policy is causing the attacker(s) to try immediately from a new ip and isn't generating a great deal more security than a slightly less restrictive policy which lures the attacker into trying a few times more from the same ip with longer intervals between the attempts. John
Re: Can't log in from Evolution or Roundcube
On 1/8/22 12:28 AM, Ken Wright wrote: When I try to log in to my mail server (ubuntu 20.04, Postfix 3.4.13, Dovecot 2.3.7.2) I get a response saying "Source stream returned no data”. At least to me, that's not particularly informative. Is it any more informative to anyone else? The last time I hit that, I'm pretty sure it was because I was going to port 80 instead of port 443 to reach Roundcube. I'm using port 143 for receiving and 587 for sending; I didn't think 443 was for email. Am I mistaken? (Not at all unlikely!) Nono, for your web browser's connection to Roundcube. I could be barking up the wrong tree here, but I'm pretty sure that's the error I hit. Thanks for the clarification. I just tried Roundcube again, and got the error "Connection to storage server failed." I also checked the nginx script for Roundcube and commented out the references to port 80, then restarted nginx. Same error. So I tend to think it's a server issue, not a client issue. Does that make any sense? It makes sense, but that's a different error than the one you got before. This new error looks like a Roundcube configuration problem, having to do with either its connection to Dovecot or possibly a back-end database server. (MySQL?) -Dave -- Dave McGuire, AK4HZ New Kensington, PA
banning, was Re: Non-user logins?
On 1/8/22 8:26 AM, dc...@dvl.werbittewas.de wrote: trying to mess with other peoples' stuff. I run fail2ban to catch those log entries and block the source IP address for a month on the first failed login. At any one time I have between 12,000 and 15,000 well, I don't know how _your_ users are connected to the internet, but in germany most people has at least daily changing IPs out of larger pools (when connected via xDSL) or even sometimes shares ip-addresses with others (when connected via tv-cable or mobile - having a private network-address, which is natted), so it's possible to get/use an IP, which was used before by some script-kiddies... Obviously. However, my users are nearly all on static IP addresses. btw.: setting up a new mail-client and making any mistake by reading it from old install or writing it into new install also leads to a months-blocking with above restrictive handling... (any may drive this user mad) Again, "obviously". May mail server is not new; I was not the OP on this thread who came here looking for help. so anyone, who has no experience with blocking should be really careful with it. That's good advice for everything, not just blocking. My first experience with blocking was on a Cisco AGS in 1994, buddy. Not a n00b. -Dave -- Dave McGuire, AK4HZ New Kensington, PA
banning, was Re: Non-user logins?
On 1/8/22 8:57 AM, John Fawcett wrote: yes, blocking on the first wrong password sounds like overkill. But it does depend on user base. For a small mail server with few known users it could be workable. It may be overkill for your network, but it's not overkill for mine. But even on small servers I'd recommend blocking for a small time (like up to an hour) after a small number of failures (example 3). Then if this pattern repeats (for example 3 times) within a longer period (for example up to a day), blocking for a longer period (example 1 week) using the recidive jail. My mail server is a small one with about 135 users, a corporate network. NONE of them actually type their password when they're checking their mail. They type it exactly once when they set up the account in their mail client(s), like everyone else in the world for the past decade or more. Mileage will vary depending on user base and number of support requests generated. The point about fail2ban is that it slows down attackers stopping infinite and fast repeating attacks from the same ip. That should be in combination with a good password policy which reduces the probability of any single attack guessing the password. It doesn't necessarily have to zero out attacks. As Dave has experimented, to bypass fail2ban all the attacker has to do is use a different ip. 10-15K blocks in place at any time seem very high compared to the few attacks I see. Sigh. I don't "experiment" with production networks. I set up a banning policy that works for the attack patterns that I see in my logs, and that work with my user base. As I explained to the other guy who decided that how I run my network is wrong, I'm not new at this. Any attack mitigation strategy has to begin with observation and rational thought. Network security is no place for guesswork or assumptions. I'd hazard a guess that the restrictive fail2ban policy is causing the attacker(s) to try immediately from a new ip and isn't generating a great deal more security than a slightly less restrictive policy which lures the attacker into trying a few times more from the same ip with longer intervals between the attempts. I wasn't asking for a critique of my configuration; I explained my approach to a new user who came here looking for help. Which is the last time I'll do THAT on this list, by the way. But since you brought it up, the attack pattern that I see most frequently is a single IP address trying different, obviously algorithmically-generated usernames at long intervals, many seconds to many hours, and in some cases a day or more. This started around 2014 or so, and has persisted and grown. The approach that you describe above seems to make sense on the surface, but looking at actual logs doesn't support the idea. The "attackers" in this case are almost exclusively little programs thrown together by kids and run from zombie Windows boxes on clueless users' home networks organized as botnets. This pattern is visible when the same sequences of generated usernames start appearing from different IP addresses within hours of each other. Some of the more advanced stuff has the adaptive behaviors that you describe, but not many. These script k1ddiez are trolling for low-hanging fruit to get bank account info etc out of peoples' mail accounts. These are almost never focused attacks from one motivated, knowledgeable person trying hard to get into one user's mail. So, hazard all the guesses you like, I will continue to develop banning strategies for my servers which fit the attacks that I observe through direct analysis of my logs. Now, please don't misunderstand, I actually do appreciate the thought and intention behind your unsolicited advice. But, for future reference, don't assume that someone doesn't know what they're doing just because their approach differs from either your approach for your own network, or your guesses about theirs. -Dave, pre-coffee -- Dave McGuire, AK4HZ New Kensington, PA
Re: Can't log in from Evolution or Roundcube
On Sat, 2022-01-08 at 11:05 -0500, Dave McGuire wrote: > On 1/8/22 12:28 AM, Ken Wright wrote: > > > > > > When I try to log in to my mail server (ubuntu 20.04, > > > > > > Postfix 3.4.13, Dovecot 2.3.7.2) I get a response saying > > > > > > "Source stream returned no data”. At least to me, that's > > > > > > not particularly informative. Is it any more informative > > > > > > to anyone else? > > > > > > > > > > The last time I hit that, I'm pretty sure it was because > > > > > I was going to port 80 instead of port 443 to reach > > > > > Roundcube. > > > > > > > > I'm using port 143 for receiving and 587 for sending; I didn't > > > > think 443 was for email. Am I mistaken? (Not at all > > > > unlikely!) > > > > > > Nono, for your web browser's connection to Roundcube. I > > > could be barking up the wrong tree here, but I'm pretty sure > > > that's the error I hit. > > > > Thanks for the clarification. I just tried Roundcube again, and > > got the error "Connection to storage server failed." I also > > checked the nginx script for Roundcube and commented out the > > references to port 80, then restarted nginx. Same error. So I > > tend to think it's a server issue, not a client issue. Does that > > make any sense? > > It makes sense, but that's a different error than the one you got > having to do with either its connection to Dovecot or possibly a > back-end database server. (MySQL?) MariaDB. Now it's time for me to clarify. The "source stream returned no data" error is in Evolution; the "connection to storage server failed" is in Roundcube. So I'm seeing similar errors in two different email clients trying to get to the same server. I know there are any number of reasons for a failed connection to Dovecot, but I just don't have the experience to figure this one out. Ken
Re: banning, was Re: Non-user logins?
On Sat, 2022-01-08 at 11:22 -0500, Dave McGuire wrote: > I wasn't asking for a critique of my configuration; I explained my > approach to a new user who came here looking for help. > > Which is the last time I'll do THAT on this list, by the way. Dave, on behalf of all the n00bs who come here seeking help, I ask you to please reconsider this. Quite simply, we (I) need all the help we can get! Ken
Re: banning, was Re: Non-user logins?
On 08/01/2022 17:22, Dave McGuire wrote: On 1/8/22 8:57 AM, John Fawcett wrote: yes, blocking on the first wrong password sounds like overkill. But it does depend on user base. For a small mail server with few known users it could be workable. It may be overkill for your network, but it's not overkill for mine. But even on small servers I'd recommend blocking for a small time (like up to an hour) after a small number of failures (example 3). Then if this pattern repeats (for example 3 times) within a longer period (for example up to a day), blocking for a longer period (example 1 week) using the recidive jail. My mail server is a small one with about 135 users, a corporate network. NONE of them actually type their password when they're checking their mail. They type it exactly once when they set up the account in their mail client(s), like everyone else in the world for the past decade or more. Mileage will vary depending on user base and number of support requests generated. The point about fail2ban is that it slows down attackers stopping infinite and fast repeating attacks from the same ip. That should be in combination with a good password policy which reduces the probability of any single attack guessing the password. It doesn't necessarily have to zero out attacks. As Dave has experimented, to bypass fail2ban all the attacker has to do is use a different ip. 10-15K blocks in place at any time seem very high compared to the few attacks I see. Sigh. I don't "experiment" with production networks. I set up a banning policy that works for the attack patterns that I see in my logs, and that work with my user base. As I explained to the other guy who decided that how I run my network is wrong, I'm not new at this. Any attack mitigation strategy has to begin with observation and rational thought. Network security is no place for guesswork or assumptions. I'd hazard a guess that the restrictive fail2ban policy is causing the attacker(s) to try immediately from a new ip and isn't generating a great deal more security than a slightly less restrictive policy which lures the attacker into trying a few times more from the same ip with longer intervals between the attempts. I wasn't asking for a critique of my configuration; I explained my approach to a new user who came here looking for help. Which is the last time I'll do THAT on this list, by the way. But since you brought it up, the attack pattern that I see most frequently is a single IP address trying different, obviously algorithmically-generated usernames at long intervals, many seconds to many hours, and in some cases a day or more. This started around 2014 or so, and has persisted and grown. The approach that you describe above seems to make sense on the surface, but looking at actual logs doesn't support the idea. The "attackers" in this case are almost exclusively little programs thrown together by kids and run from zombie Windows boxes on clueless users' home networks organized as botnets. This pattern is visible when the same sequences of generated usernames start appearing from different IP addresses within hours of each other. Some of the more advanced stuff has the adaptive behaviors that you describe, but not many. These script k1ddiez are trolling for low-hanging fruit to get bank account info etc out of peoples' mail accounts. These are almost never focused attacks from one motivated, knowledgeable person trying hard to get into one user's mail. So, hazard all the guesses you like, I will continue to develop banning strategies for my servers which fit the attacks that I observe through direct analysis of my logs. Now, please don't misunderstand, I actually do appreciate the thought and intention behind your unsolicited advice. But, for future reference, don't assume that someone doesn't know what they're doing just because their approach differs from either your approach for your own network, or your guesses about theirs. -Dave, pre-coffee Dave no one is telling you how to run your server or offering you unsolicited advice but on a public mailing list there can be multiple points of view about a topic and there may be not be a single right way to do something. There is no reason then to take this as someone telling you that how you run your network is wrong or to avoid participating in future. John
Re: banning
Am 08.01.22 um 17:22 schrieb Dave McGuire: > I wasn't asking for a critique of my configuration; I explained my > approach to a new user who came here looking for help. huh? well, I don't think that anyone wanted to say anything about _your_ configuration, but wanted to supplement, that your configuration is maybe good for _your_ systems, but other systems may need other approaches. especially if answering to someone, who experiences new things and asking for help, it should be normal, that someone even pick answers from others and supplement things from their own view. this has nothing to do with blaming the author of the answer, but should show the asking person, that the answer given maybe incomplete (not wrong!) for the concrete situation, because it only shows a single view to a problem. > Which is the last time I'll do THAT on this list, by the way. this would be bad in my opinion. we're all doing something within our own world and if we have the possibility to look at other worlds and how people solve problems there, the knowlegde gets better and even if then someone asks us in a private situation with no additonal listeners, we can refer to solutions, which are different from our own one. at least newbies should see: there's nearly nowhere one answer. d.
Re: Can't log in from Evolution or Roundcube
On 1/8/22 11:27 AM, Ken Wright wrote: MariaDB. Now it's time for me to clarify. The "source stream returned no data" error is in Evolution; the "connection to storage server failed" is in Roundcube. So I'm seeing similar errors in two different email clients trying to get to the same server. I know there are any number of reasons for a failed connection to Dovecot, but I just don't have the experience to figure this one out. Understood. There's enough expertise here to get you going. First, ignore the superficial similarity of the errors and diagnose/address them individually. Get one mail client working via IMAP. For Evolution's "source stream returned no data", ignore my previous suggestion about web browser SSL vs. non-SSL, as that's not relevant to Evolution. The error looked very familiar to me at first; it probably came from the same library as whatever I was working with when I hit that. (probably libssl) Go into the mail account configuration in Evolution and check the settings there. I don't use Evolution so I can't direct you more specifically, but what to pay attention to here is the connection settings and port numbers. I'm guessing (hazardously) that the port number or SSL method is incorrect. Make sure to distinguish between SSL and TLS (STARTTLS). Concentrate on getting that working first; don't get distracted from it. After that's working, then move on to Roundcube. Look in Roundcube's config.inc.php file. Where that file is located is system-dependend; mine is in /opt/local/etc/roundcube, which is specific to SmartOS. Parameter "$config['db_dsnw']" is the DSN for your database connection. This is the format of that configuration variable: $config['db_dsnw'] = 'mysql://USERNAME:PASSWORD@SERVER/DATABASE'; You're running MariaDB, which is a fork of MySQL, so I'm guessing that Roundcube doesn't differentiate between the two, so the "mysql" above is probably correct. Check the docs if that fails. Obviously, the words in uppercase must be correct for your installation. "SERVER" might be "localhost" for you. (it isn't for me) One quick thing to check: Did you issue a "flush privileges" command to MariaDB after creating the account for Roundcube to use? See how far that gets you and report back. -Dave -- Dave McGuire, AK4HZ New Kensington, PA
Sv: banning
I would say, lock accounts to for example IP address, ASN or GeoIP. This can be accomplished simply by a custom login handler, which also checks IP against database. And first time users, and those who change country/ISP/IP have to simply logon to a web interface (where 2FA can be required and also Captcha) and add their IPs/ASNs/Geo's. For a larger user base, I would recommend GeoIP and no web interface, save country of first login to database, and subsequent logins must originate from same country. Users that want to reset have to contact support. If you are a web hotel who only sell service to a specific country at all, lock the ports in firewall to that GeoIP. For smaller user base, like 50-100 users, I would recommend locking to ASN and providing a web interface where multiple ASN can be added. So people syncing from mobile and home can succeed. For very small user base, like 10's of users, just plan lock to IP. By connecting IP to accounts, you greatly reduce the attack surface.
Re: Can't log in from Evolution or Roundcube
Dave McGuire schreef op 2022-01-08 18:20: On 1/8/22 11:27 AM, Ken Wright wrote: MariaDB. Now it's time for me to clarify. The "source stream returned no data" error is in Evolution; the "connection to storage server failed" is in Roundcube. So I'm seeing similar errors in two different email clients trying to get to the same server. I know there are any number of reasons for a failed connection to Dovecot, but I just don't have the experience to figure this one out. Understood. There's enough expertise here to get you going. First, ignore the superficial similarity of the errors and diagnose/address them individually. Get one mail client working via IMAP. For Evolution's "source stream returned no data", ignore my previous suggestion about web browser SSL vs. non-SSL, as that's not relevant to Evolution. The error looked very familiar to me at first; it probably came from the same library as whatever I was working with when I hit that. (probably libssl) Go into the mail account configuration in Evolution and check the settings there. I don't use Evolution so I can't direct you more specifically, but what to pay attention to here is the connection settings and port numbers. I'm guessing (hazardously) that the port number or SSL method is incorrect. Make sure to distinguish between SSL and TLS (STARTTLS). Concentrate on getting that working first; don't get distracted from it. After that's working, then move on to Roundcube. Look in Roundcube's config.inc.php file. Where that file is located is system-dependend; mine is in /opt/local/etc/roundcube, which is specific to SmartOS. Parameter "$config['db_dsnw']" is the DSN for your database connection. This is the format of that configuration variable: $config['db_dsnw'] = 'mysql://USERNAME:PASSWORD@SERVER/DATABASE'; You're running MariaDB, which is a fork of MySQL, so I'm guessing that Roundcube doesn't differentiate between the two, so the "mysql" above is probably correct. Check the docs if that fails. Obviously, the words in uppercase must be correct for your installation. "SERVER" might be "localhost" for you. (it isn't for me) AFAIK, the error "Connection to storage server failed" only occurs when Roundcube can't connect to IMAP (in this case, Dovecot). If there's a database issue, Roudcube should show the message "Unable to connect to the database!". One quick thing to check: Did you issue a "flush privileges" command to MariaDB after creating the account for Roundcube to use? FWIW, that hasn't been needed for quite some time when not directly manipulating mysql.users (i.e. using `CREATE USER`). See how far that gets you and report back. -Dave -- With kind regards, William Edwards
Re: Can't log in from Evolution or Roundcube
On 1/8/22 3:04 PM, William Edwards wrote: After that's working, then move on to Roundcube. Look in Roundcube's config.inc.php file. Where that file is located is system-dependend; mine is in /opt/local/etc/roundcube, which is specific to SmartOS. Parameter "$config['db_dsnw']" is the DSN for your database connection. This is the format of that configuration variable: $config['db_dsnw'] = 'mysql://USERNAME:PASSWORD@SERVER/DATABASE'; You're running MariaDB, which is a fork of MySQL, so I'm guessing that Roundcube doesn't differentiate between the two, so the "mysql" above is probably correct. Check the docs if that fails. Obviously, the words in uppercase must be correct for your installation. "SERVER" might be "localhost" for you. (it isn't for me) AFAIK, the error "Connection to storage server failed" only occurs when Roundcube can't connect to IMAP (in this case, Dovecot). If there's a database issue, Roudcube should show the message "Unable to connect to the database!". Rummaging through the source code, I see that you are indeed correct. Thank you for that clarification. One quick thing to check: Did you issue a "flush privileges" command to MariaDB after creating the account for Roundcube to use? FWIW, that hasn't been needed for quite some time when not directly manipulating mysql.users (i.e. using `CREATE USER`). I had heard that, but not knowing: - what release of MySQL obviated that long-standing requirement - when MariaDB forked from MySQL - what release of anything the OP is running - how the OP chose to manipulate the grant tables ...I chose to make the recommendation for the sake of safety, as it's cheap to try. Now maybe we can get back to the OP's actual problems, which nobody else seems to be interested in today. -Dave -- Dave McGuire, AK4HZ New Kensington, PA
Re: banning, was Re: Non-user logins?
On 1/8/22 12:12 PM, John Fawcett wrote: On 08/01/2022 17:22, Dave McGuire wrote: On 1/8/22 8:57 AM, John Fawcett wrote: yes, blocking on the first wrong password sounds like overkill. But it does depend on user base. For a small mail server with few known users it could be workable. It may be overkill for your network, but it's not overkill for mine. But even on small servers I'd recommend blocking for a small time (like up to an hour) after a small number of failures (example 3). Then if this pattern repeats (for example 3 times) within a longer period (for example up to a day), blocking for a longer period (example 1 week) using the recidive jail. My mail server is a small one with about 135 users, a corporate network. NONE of them actually type their password when they're checking their mail. They type it exactly once when they set up the account in their mail client(s), like everyone else in the world for the past decade or more. Mileage will vary depending on user base and number of support requests generated. The point about fail2ban is that it slows down attackers stopping infinite and fast repeating attacks from the same ip. That should be in combination with a good password policy which reduces the probability of any single attack guessing the password. It doesn't necessarily have to zero out attacks. As Dave has experimented, to bypass fail2ban all the attacker has to do is use a different ip. 10-15K blocks in place at any time seem very high compared to the few attacks I see. Sigh. I don't "experiment" with production networks. I set up a banning policy that works for the attack patterns that I see in my logs, and that work with my user base. As I explained to the other guy who decided that how I run my network is wrong, I'm not new at this. Any attack mitigation strategy has to begin with observation and rational thought. Network security is no place for guesswork or assumptions. I'd hazard a guess that the restrictive fail2ban policy is causing the attacker(s) to try immediately from a new ip and isn't generating a great deal more security than a slightly less restrictive policy which lures the attacker into trying a few times more from the same ip with longer intervals between the attempts. I wasn't asking for a critique of my configuration; I explained my approach to a new user who came here looking for help. Which is the last time I'll do THAT on this list, by the way. But since you brought it up, the attack pattern that I see most frequently is a single IP address trying different, obviously algorithmically-generated usernames at long intervals, many seconds to many hours, and in some cases a day or more. This started around 2014 or so, and has persisted and grown. The approach that you describe above seems to make sense on the surface, but looking at actual logs doesn't support the idea. The "attackers" in this case are almost exclusively little programs thrown together by kids and run from zombie Windows boxes on clueless users' home networks organized as botnets. This pattern is visible when the same sequences of generated usernames start appearing from different IP addresses within hours of each other. Some of the more advanced stuff has the adaptive behaviors that you describe, but not many. These script k1ddiez are trolling for low-hanging fruit to get bank account info etc out of peoples' mail accounts. These are almost never focused attacks from one motivated, knowledgeable person trying hard to get into one user's mail. So, hazard all the guesses you like, I will continue to develop banning strategies for my servers which fit the attacks that I observe through direct analysis of my logs. Now, please don't misunderstand, I actually do appreciate the thought and intention behind your unsolicited advice. But, for future reference, don't assume that someone doesn't know what they're doing just because their approach differs from either your approach for your own network, or your guesses about theirs. -Dave, pre-coffee Dave no one is telling you how to run your server or offering you unsolicited advice but on a public mailing list there can be multiple points of view about a topic and there may be not be a single right way to do something. There is no reason then to take this as someone telling you that how you run your network is wrong or to avoid participating in future. Well thanks for that, but perhaps you should re-read what you sent that prompted my (admittedly rather sour) reply. -Dave -- Dave McGuire, AK4HZ New Kensington, PA
Re: Non-user logins?
On Fri, 7 Jan 2022, Ken Wright wrote: On Fri, 2022-01-07 at 23:27 -0500, Dave McGuire wrote: On 1/7/22 11:24 PM, Ken Wright wrote: So, if anyone can tell me what's going on with all these logins, I'd be much obliged! I see them all the time on the mail servers I run. Typical kids trying to mess with other peoples' stuff. I run fail2ban to catch those log entries and block the source IP address for a month on the first failed login. At any one time I have between 12,000 and 15,000 addresses in my blocked list for IMAP. Dave, that's exactly the kind of answer I was looking for. Fail2ban, huh? I'll have to check that out. Thanks again! Yup, these SMTP AUTH BFD attempts come in thick and heavy. Another resource to preempt these attacks is Spamhaus's XBL blacklist. At my site, there was a 99.2% hit rate and very low false positives. Even those FPs led to some useful discoveries that the client had a malware they didn't know about. http://www.blocklist.de/en/index.html also run a DBS RBL list and I've had zero FPs after years of use. I think you can even get Fail2ban report to your attackers to this site to add to the crowdsourcing. Joseph Tam
Patch: assertion failed in doveadm fts lookup
Hi
I'm reposting this patch, which I have been applying locally since I
originally posted it. Hopefully it can be considered for inclusion in
the official release. It was tested on dovecot 2.3.16 but applies
against the latest version 2.3.17.1.
This is the "assertion failed" that it addresses
doveadm fts lookup -u j...@voipsupport.it body "text to look for"
doveadm(j...@voipsupport.it): Panic: file mail-storage.c: line 2108
(mailbox_get_open_status): assertion failed: (box->opened)
doveadm(j...@voipsupport.it): Error: Raw backtrace:
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x42)
[0x7ff624fa7072] ->
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e)
[0x7ff624fa718e] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xffc3b)
[0x7ff624fb3c3b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xffc71)
[0x7ff624fb3c71] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x5536f)
[0x7ff624f0936f] ->
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0x3fc31)
[0x7ff6250bcc31] ->
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x540d)
[0x7ff62244f40d] ->
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x4d)
[0x7ff6246b624d] ->
/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x2e8c)
[0x7ff62240ce8c] -> doveadm(+0x31ded) [0x55f0c7f3aded] ->
doveadm(+0x32862) [0x55f0c7f3b862] ->
doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55f0c7f3c6fd] ->
doveadm(doveadm_cmd_run_ver2+0x4d8) [0x55f0c7f4d158] ->
doveadm(doveadm_cmd_try_run_ver2+0x3a) [0x55f0c7f4d1aa] ->
doveadm(main+0x1f6) [0x55f0c7f2b606] ->
/lib64/libc.so.6(__libc_start_main+0xd5) [0x7ff624bc8b75] ->
doveadm(_start+0x2e) [0x55f0c7f2ba0e]
Aborted (core dumped)
Here's the patch:
diff -ur dovecot-2.3.17.1-orig/src/plugins/fts/doveadm-fts.c
dovecot-2.3.17.1-new/src/plugins/fts/doveadm-fts.c
--- dovecot-2.3.17.1-orig/src/plugins/fts/doveadm-fts.c 2021-12-03
12:48:47.0 +0100
+++ dovecot-2.3.17.1-new/src/plugins/fts/doveadm-fts.c 2022-01-09
01:33:23.396342025 +0100
@@ -47,6 +47,14 @@
i_array_init(&result.scores, 16);
box = mailbox_alloc(info->ns->list, info->vname, 0);
+ mailbox_set_reason(box,"fts search");
+ if (mailbox_open(box) < 0) {
+ i_error("Couldn't open mailbox: %s",
+ mailbox_get_last_internal_error(box, NULL));
+ doveadm_mail_failed_error(ctx, MAIL_ERROR_TEMP);
+ return -1;
+ }
+
if (fts_backend_lookup(backend, box, ctx->search_args->args,
FTS_LOOKUP_FLAG_AND_ARGS,
&result) < 0) {
i_error("fts lookup failed");
John
Patch: safeguard against too large value for uid being sent to solr for single mailbox searches
Hi
I'm reposting a patch for solr integration which I have been applying
locally. It applies against 2.3.17.1.
Dovecot already has a mechanism when doing solr fts searches on multiple
mailboxes that prevents a too large value for maximum rows being sent to
solr.
#define SOLR_MAX_MULTI_ROWS 10
(defined and used in plugins/fts-solr/fts-backend-solr.c)
However on single mailbox searches Dovecot uses the last uid of the
mailbox, which can be too large for solr. My suggested patch is to also
limit maximum rows to return on single mailbox searches.
Here is the patch:
diff -ur dovecot-2.3.17.1-orig/src/plugins/fts-solr/fts-backend-solr.c
dovecot-2.3.17.1-new/src/plugins/fts-solr/fts-backend-solr.c
--- dovecot-2.3.17.1-orig/src/plugins/fts-solr/fts-backend-solr.c
2021-12-03 12:48:47.0 +0100
+++ dovecot-2.3.17.1-new/src/plugins/fts-solr/fts-backend-solr.c
2022-01-09 01:33:23.401341959 +0100
@@ -837,7 +837,7 @@
str = t_str_new(256);
str_printfa(str,
"wt=xml&fl=uid,score&rows=%u&sort=uid+asc&q=%%7b!lucene+q.op%%3dAND%%7d",
- status.uidnext);
+ I_MIN(status.uidnext,SOLR_MAX_MULTI_ROWS));
prefix_len = str_len(str);
if (solr_add_definite_query_args(str, args, and_args)) {
John
Patch: enhancements for solr/tika integration
Hi
here's a patch with some enhancements that I am applying locally for
solr/tika integration. Hopefully this can be considered for inclusion.
I've tested up to 2.3.16 and this patch applies against latest version
2.3.17.1. The contents are:
1. Allow username and password in tika configuration (as is already
supported for solr). For example:
fts_tika = https://user:password@tika_host:tika_port/tika/tika
This means that tika can be more easily located on a different server to
dovecot. Locating Solr and Tika on separate servers can isolate the
Dovecot server from issues that may happen on Solr or Tika (like out of
memory).
2. Introduce an optional configurable limit on the size of emails whose
message bodies will be indexed with solr via a new setting fts_max_size,
to be put in the plugin section. For example
plugin {
...
fts_max_size = 2M
...
}
Message body indexing is skipped for messages that are larger than this
setting.
3. Introduce an optional configurable limit on the size of emails whose
attachments will be sent to tika for keyword extraction, via a new
setting fts_max_size_tika, to be put in the plugin section. For example:
plugin {
...
fts_max_size_tika = 1M
}
I noticed that some very large attachments are being sent to tika and
these often do not have any useful content to index. For simplicity the
check is done on message size not individual or total attachment size.
Here's the patch:
diff -ur dovecot-2.3.17.1-orig/src/plugins/fts/fts-build-mail.c
dovecot-2.3.17.1-new/src/plugins/fts/fts-build-mail.c
--- dovecot-2.3.17.1-orig/src/plugins/fts/fts-build-mail.c 2021-12-03
12:48:47.0 +0100
+++ dovecot-2.3.17.1-new/src/plugins/fts/fts-build-mail.c 2022-01-09
01:33:23.398341998 +0100
@@ -17,6 +17,7 @@
#include "fts-filter.h"
#include "fts-api-private.h"
#include "fts-build-mail.h"
+#include "settings-parser.h"
/* there are other characters as well, but this doesn't have to be
exact */
#define IS_WORD_WHITESPACE(c) \
@@ -25,6 +26,8 @@
wherever */
#define MAX_WORD_SIZE 1024
+static bool debug = FALSE;
+
struct fts_mail_build_context {
struct mail *mail;
struct fts_backend_update_context *update_ctx;
@@ -34,6 +37,7 @@
buffer_t *word_buf, *pending_input;
struct fts_user_language *cur_user_lang;
+ bool oversized_tika;
};
static int fts_build_data(struct fts_mail_build_context *ctx,
@@ -223,7 +227,7 @@
parser_context.user = mail_storage_get_user(storage);
parser_context.content_disposition = ctx->content_disposition;
-
+ parser_context.oversized_tika = ctx->oversized_tika;
if (fts_parser_init(&parser_context, &ctx->body_parser)) {
/* extract text using the the returned parser */
*binary_body_r = TRUE;
@@ -496,7 +500,32 @@
bool binary_body;
const char *error;
int ret;
-
+ uoff_t msg_size;
+ uoff_t fts_max_size = 0;
+ uoff_t fts_max_size_tika = 0;
+ const char * fts_max_size_setting;
+ const char * fts_max_size_tika_setting;
+ bool oversized_msg;
+ bool oversized_tika;
+
+ fts_max_size_setting =
mail_user_plugin_getenv(update_ctx->backend->ns->user, "fts_max_size");
+ if (fts_max_size_setting != NULL) {
+ if (debug) i_debug("fts_max_size %s",fts_max_size_setting);
+ if (settings_get_size(fts_max_size_setting,
&fts_max_size, &error) < 0) {
+ i_error("%s",error);
+ fts_max_size = 0;
+ }
+ if (debug) i_debug("fts_max_size (value)
%"PRIuUOFF_T,fts_max_size);
+ }
+ fts_max_size_tika_setting =
mail_user_plugin_getenv(update_ctx->backend->ns->user,
"fts_max_size_tika");
+ if (fts_max_size_tika_setting != NULL) {
+ if (debug) i_debug("fts_max_size_tika
%s",fts_max_size_tika_setting);
+ if (settings_get_size(fts_max_size_tika_setting,
&fts_max_size_tika, &error) < 0) {
+ i_error("%s",error);
+ fts_max_size_tika = 0;
+ }
+ if (debug) i_debug("fts_max_size_tika (value)
%"PRIuUOFF_T,fts_max_size_tika);
+ }
*may_need_retry_r = FALSE;
if (mail_get_stream_because(mail, NULL, NULL, "fts indexing",
&input) < 0) {
if (mail->expunged)
@@ -505,10 +534,21 @@
mailbox_get_last_internal_error(mail->box, NULL));
return -1;
}
-
+ oversized_msg = FALSE;
+ oversized_tika = FALSE;
+ i_stream_get_size(input,TRUE,&msg_size);
+ if (fts_max_size > 0 && msg_size > fts_max_size) {
+ i_info("Skipping message body indexing because size
%"PRIuUOFF_T" exceeds setting fts_max_size
%s",msg_size,fts_max_size_setting);
+ oversized_msg = TRUE;
+ }
+ if (fts_max_size_tika > 0 && msg_size > fts_max_size_tika) {
+ i_info("Skipping message at
