This is a very difficult problem to solve, as evidenced by the fact that companies who claim to have solved it are worth billions of dollars.
The crux of the issue is that there isn't effective two-way communication between senders and receivers. I work for an ESP. We try to keep bad senders off of our network - sometimes with varying degree of success, but we do our best to keep our mail stream clean. I know other reputable ESPs try to do the same - look at MailChimp's investment into Omnivore, for example. Every time I speak to someone at another ESP, the first question asked is "What are you doing about Microsoft? How are you getting to the inbox at Microsoft?" None of us have any answers. We subscribe to eDataSource and use their data to track our inbox placement as well as competitors. So, I can tell you when Microsoft changed their machine learning algorithms - we saw a huge jump for everyone a few months ago, 30%+ for many senders. However, there was nothing we did that caused that. Our mail stream didn't change. SNDS, JMRP, these tools are simply inadequate to diagnose issues because most of the decision-making happens when the email hits the machine learning model, and those decisions aren't served up in any meaningful way. Obviously we still use them because some data is better than no data, but the fact is that SNDS can report that everything is fine while email doesn't get delivered to the inbox. If you open tickets with tier 1 support, they reply with the same canned responses pre-approved by legal. "Follow best practices, read these help articles, sign up for SNDS and JMRP, etc." It's the same everywhere. I've done consulting for individual clients and it's the same story if I open a support ticket with Oath or anywhere else. None of them know what the machine learning model is doing, none of them can tell you why emails are being filtered, nobody knows. If you ask them to escalate, they don't. They respond with the same pre-approved answers and ignore the fact you asked for an escalation, or they stop responding to you entirely. The simple fact is that there is nowhere to escalate a request of "I think the machine learning got it wrong here" or "All of our data indicates this is a good sender, can you tell us why they're not?" I don't want to send spam. If I had better ways of proving which customers were spammers, they wouldn't be on our network anymore. For every customer our compliance team fires, we have to make a business case to someone at some level within the organization for justification why. That's easy to do when we can prove they're using a non opt-in list, or their bounce rates are high, or some other indicator that they aren't a good sender. But when their content looks legitimate, they're using SPF/DKIM/DMARC, their list is fully opt-in, they have a valid List-Unsubscribe, complaint rates reported by FBLs are low, IP reputation shows as normal, but emails don't make it to the inbox at Microsoft (but they do at Oath, or Gmail) - what am I supposed to say? I have to either say that Microsoft's machine learning knows something that Oath's and Gmail's doesn't, or that their machine learning got it wrong. ESPs are afraid to say this for good reason. If a company like Oath or Microsoft or Google decides to block our entire mail stream, we go out of business. It's as simple as that. If we don't say anything, then these problems continue for months or years and nothing is done about them. Then if you try to bring the issue up you get pushback - "It must be working, you're the only one who's said anything about it, if there were real problems then other people would have complained." I know it's broken when I can set up a brand new outlook.com email address and send an email to myself from inside Microsoft's own webmail client, and the message goes to spam. I know it's broken when clients attempt to hire me as a freelancer to diagnose why they're not getting emails inside Office 365 that they sent to themselves from an external mail system that they also control and whitelisted which only sends transactional emails. I don't know what to tell them except not to use Office 365, because I've never been able to escalate a problem beyond tier 1 support, where it goes nowhere. So I've become jaded into thinking that the only way to get eyes on the problem is if it impacts the bottom line. I'm not sure what the best way to solve the problem is in a reliable, repeatable, scalable issue. I fully understand why these companies have escalation processes in place and why you can't just bypass the normal ticketing process. However, it's exceptionally frustrating when you can never get escalated to someone who knows what they're talking about and who can help you identify or fix the issue. Fixing the issue may simply be giving me enough ammo to go to the CEO and say "We have to fire this customer if you don't want to tank our mail stream and watch your compliance team quit to work for a company that cares about stopping spam." That's OK! That's my problem to deal with! But I can't do it without the data, and the data I'm sitting on is often inadequate. I don't want to make more work for receivers. But, the problem is that users don't want to receive spam, right? It's great that we have spam filters that can try to prevent recipients from receiving spam. It would be even better if the sender didn't send it to begin with. So that's what I'm asking for - give me better ways to ensure that I'm not sending it. We're all human, and my company doesn't have the resources that Google or Microsoft or Oath does. I'm sure there's things that get by that I don't know about. If I knew about them, I would fix them. If I could accurately and reliably identify all of the bad apples on my network, I wouldn't have them, and then you could (largely) trust all of the mail coming out of my network, which should make filtering for it easier. Right? Maybe I'm being naive here and don't understand the full scale of the issue that these huge receivers face. I've never had to deal with the scale of receivers like Google/Microsoft/Oath. Like I said, it's a hard problem to solve. I am optimistic that if senders and receivers could identify a better way to work together, though, it would be mutually beneficial. We could help identify edge cases where machine learning got things wrong, which makes the machine learning better. You can identify bad mail streams we missed, and we can make our compliance/vetting process better. So why aren't we all trying to solve this as a community? Am I crazy? ________________________________ From: mailop <mailop-boun...@mailop.org> on behalf of Michael Peddemors <mich...@linuxmagic.com> Sent: Friday, April 19, 2019 8:33:31 AM To: mailop@mailop.org Subject: Re: [mailop] Our customers e-mail constantly going to outlook.com junkmail (any Microsoft people around?) And to add my two bits in.. Sorry Rich, but for ISP's on the front line, they often will express.. "... as long as it goes to the spam folder, prefer that to customers complaining they didnt' get an email ..." Techies of course might have different 'purist' opinions, but we still have to live in the real world. But to your point about 'blocking' during the SMTP process, life would be so much better if sender's stop obfuscating the MAIL FROM, and make that clearer for recipient servers to identify the senders. While of course the message headers can be inspected, the ability to forge those is much less complicated than forging MAIL FROM. As a 'user' I would expect to allow '@mailop.org', and not say.. (using a well known example ... @in.mailop.org) if it is a mailing list. (Please, no suggestions about work arounds, they are in place of course, accurate MAIL FROM simply makes life easier for everyone) Black/White doesn't fit into how most email providers can provide satisfactory service to their clients, in their eyes.. so handling 'grey' issues needs to involve the decisions of the end user, and only by 'categorization', eg spam folders can that be accomplished. On 2019-04-19 8:12 a.m., Rob McEwen wrote: > On 4/19/2019 8:14 AM, Rich Kulawiec wrote: >> Best practice is to accept/reject messages only >> ... >> no "spam folders" > > > Rich, > > I generally agree with your entire post - except - for your "no spam > folders" part... > > There is the possibility to accept the entire message and then issue the > accept/reject response AFTER the entire message is received (after > "DATA") but still during the original SMTP connection - and THEN put a > copy of the rejected messages into the spam folder. But this does take > more resources. The advantage here is that there is then an "audit > trail" so that the users can have peace of mind if they suspect that the > spam filer blocked a legit message, but they aren't sure if the sender > really sent it - then they can have SOME degree of confidence that it > was never sent when it doesn't show up in either the inbox or the spam > folder. > > But due to the extra resource usage of receiving ALL entire messages, > and saving ALL spams to the spam folder - some mail hosters do a > compromise where the very worst spam - particularly spams that are > blacklisted on multiple high-quality sending-IP blacklists - will get > rejected without receiving the messages - then the OTHER spams that are > rejected by content filtering, or whose sending IPs are more mildly > blacklisted - THOSE then go to the spam folder. Many find that to be a > reasonable compromise. > > In both cases I described - the filter is STILL giving the FINAL > spam/ham decision at connection time, all fed back to the sender - and > that is SUPERIOR spam filtering. (btw - not only does Microsoft not do > this - Google/Gmail/G-Suite ALSO doesn't do this. There are situations > where Google accepts the message, but then puts it in the spam folder > after additional filtering is done after the original SMTP connection.) > > So I do agree that spam filters that provided a reject-accept message at > connection time, as a FINAL filtering decision - are providing a > superior spam filtering service when compared to the ones that accept > the message, but then do after-the-fact filtering, where the sender is > then left with a message SAYING that the message was accepted, but yet > it really wasn't delivered to the inbox. > -- "Catch the Magic of Linux..." ------------------------------------------------------------------------ Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic Linux Products, Services, Development and OpenSource initiatives - LinuxMagic.com<http://www.linuxmagic.com/> www.linuxmagic.com LinuxMagic products, services, development and opensource initiatives. Information about LinuxMagic products and services A Wizard IT Company - For More Info http://www.wizard.ca Wizard IT Services - Web Hosting, Network Solutions, Internet Services<http://www.wizard.ca/> www.wizard.ca Wizard IT provides support and services for the ISP, Telco and Enterprise markets. Specializing in Linux Development, Linux Solutions, and ISP Infrastructure, Wizard Tower TechnoServices is also the parent company for LinuxMagic and City Email, and provides mail server and anti-spam products, as well as hosted and outsourced solutions and custom development. "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd. ------------------------------------------------------------------------ 604-682-0300 Beautiful British Columbia, Canada This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company. _______________________________________________ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
_______________________________________________ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
