Viktor and anyone else,
I'd like your opinion on something I've come up with that seems to work in my
test box. What I've done is set things up so that instead of % thresholds I'm
using a count of sent email. I fully expect the counting to not be 100%
accurate, off by a couple of tens or even hundreds isn't a big deal. What I did
was create the following tables:
DROP TABLE IF EXISTS domains;
CREATE TABLE domains(
domain_id INT GENERATED ALWAYS AS IDENTITY,
domain_name text,
PRIMARY KEY(domain_id)
);
DROP TABLE IF EXISTS transport;
CREATE TABLE transport(
transport_id INT GENERATED ALWAYS AS IDENTITY,
transport_name text,
PRIMARY KEY(transport_id)
);
drop table if exists send_counts;
create table send_counts(
domain_id int unique,
send_count int,
CONSTRAINT fk_domains
FOREIGN KEY(domain_id)
REFERENCES domains(domain_id)
ON DELETE CASCADE
);
drop table if exists thresholds;
create table thresholds(
domain_id int,
threshold int,
CONSTRAINT fk_domains
FOREIGN KEY(domain_id)
REFERENCES domains(domain_id)
ON DELETE CASCADE
);
drop table if exists matrix;
CREATE TABLE matrix(
matrix_id INT GENERATED ALWAYS AS IDENTITY,
domain_id INT,
transport_id INT,
PRIMARY KEY(matrix_id),
CONSTRAINT fk_domains
FOREIGN KEY(domain_id)
REFERENCES domains(domain_id)
ON DELETE CASCADE,
CONSTRAINT fk_transport
FOREIGN KEY(transport_id)
REFERENCES transport(transport_id)
ON DELETE CASCADE
);
So tables w/ the destination domains, transport to use for said domain, count
of emails sent to said domains, thresholds for the domains, and matrix table to
join the domains and transport. I went a bit overboard w/ normalizing the
tables. It probably could have all been put into one, but I'm also using this
an exercise to play w/ postgress. Once all the data has been loaded into the
tables it's ready to roll. What I probably should do is use a user defined
function, but I found that I can put an update and select in the .cfg file and
it works. I'm using this for the query:
query = update send_counts set send_count = case when send_count > threshold
then send_count else send_count + 1 end from domains, thresholds where
send_counts.domain_id=domains.domain_id and
send_counts.domain_id=thresholds.domain_id and domain_name='%s'; select
transport_name as "transport" from matrix m inner join domains d on
m.domain_id=d.domain_id inner join transport t on m.transport_id=t.transport_id
left join send_counts s on m.domain_id=s.domain_id inner join thresholds h on
m.domain_id=h.domain_id where domain_name='%s' and send_count <= threshold;
There are some extra fields and joins in that select query as I was using it
for testing stuff.
Here's what my test data looks like:
select domain_name,transport_name,threshold,send_count from matrix m inner join
domains d on m.domain_id=d.domain_id inner join transport t on
m.transport_id=t.transport_id left join send_counts s on
m.domain_id=s.domain_id inner join thresholds h on m.domain_id=h.domain_id;
domain_name | transport_name | threshold | send_count
-------------+-----------------------+-----------+------------
gmail.com | relay:[send.blah.com] | 15 | 16
yahoo.com | relay:[send.blah.com] | 25 | 20
(2 rows)
The small time test I did worked like a charm, but I'm only doing a single
client sending, not at scale.
I'm interested in anyone's thoughts on what I might have overlooked that could
come back to haunt me if I decide to roll this out anymore.
Thanks in advance.
Sean
-----Original Message-----
From: [email protected] <[email protected]> On
Behalf Of Sean Hennessey
Sent: Wednesday, November 30, 2022 11:38 PM
To: [email protected]
Subject: RE: Is there an easy way to "warm up" a new sending IP w/ Postfix
Viktor,
I want to thank you a million for this. I finally read up on the docs and got
this working. I'm still going to do some more in depth testing, but my quick
little testing seems to be doing exactly what I wanted.
-----Original Message-----
From: [email protected] <[email protected]> On
Behalf Of Viktor Dukhovni
Sent: Tuesday, November 29, 2022 3:44 AM
To: [email protected]
Subject: Re: Is there an easy way to "warm up" a new sending IP w/ Postfix
On Mon, Nov 28, 2022 at 08:57:37PM +0000, Sean Hennessey wrote:
> I searched the list archives and saw the thread of gradual shift of
> traffic from back in February of this year. That gives me some ideas,
> but that seems to be for all traffic, not a subset.
>
> I'd really like a way to send X% of gmail.com traffic to one relay and
> the rest to another relay. Ditto for a couple of other major ESP's
> like Yahoo, MS, etc...
If you're willing to spin up a small Postgres database (modulo typos on my part
that should be easy to correct):
query = SELECT U."transport"
FROM (
SELECT CASE WHEN floor(random()*100) <= T."weight"
THEN T."transport"
END AS "transport"
FROM "transports" AS T
WHERE T."domain" = '%s'
) AS U
WHERE U."transport" IS NOT NULL;
Just populate a table:
CREATE TABLE IF NOT EXISTS "transports" (
"domain" TEXT PRIMARY KEY,
"transport" TEXT NOT NULL,
"weight" INTEGER NOT NULL
);
INSERT INTO "transports" ("domain", "transport", "weight")
VALUES ( "gmail.com", "relay:[gmail-relay.example]", 99),
( "yahoo.com", "relay:[yahoo-relay.example]", 50),
... ;
And gradually lower the weights until, at weight 0, 99% of the traffic is
direct to MX and just 1% of the traffic goes to the bypass relay and after that
the row can be deleted. Initially, at weight 99, all the traffic goes to the
bypass relay.
If you want to specify a custom transport even after removing the relay, add a
fourth (nullable) column and use that value in an ELSE clause of the CASE
statement, in which case that value will be used when the bypass is not
selected.
Keeping the Postgres database local to the MTA will improve performance and
reliability. I'd resist the temptation to centralise it, but that is an option
if you're willing to have Postfix stall when a remote DB server is unreachable
or slow, or can somehow avoid that.
It is not obvious to me, just at the moment, how to do this with the built-in
Postfix randmap, pipemap, uniomap, ...
--
Viktor.
