Thank you very much. I hope it will be helpful.
On Mon, Feb 17, 2025 at 3:55 PM Timo Sirainen via dovecot
wrote:
On 17. Feb 2025, at 13.01, Timo Sirainen via dovecot
wrote:
>
> On 17. Feb 2025, at 12.46, Timo Sirainen via dovecot
wrote:
>> Also I've attached a patc
On 17. Feb 2025, at 12.46, Timo Sirainen via dovecot
wrote:
> Also I've attached a patch that allows using auth-workers with PostgreSQL
> with Dovecot v2.4. Maybe it'll be in v2.4.1.
>
> passdb sql {
> use_worker = yes
> }
Well, attachments don't seem to work on this list, so here's inline pa
On 17. Feb 2025, at 13.01, Timo Sirainen via dovecot
wrote:
>
> On 17. Feb 2025, at 12.46, Timo Sirainen via dovecot
> wrote:
>> Also I've attached a patch that allows using auth-workers with PostgreSQL
>> with Dovecot v2.4. Maybe it'll be in v2.4.1.
>>
>> passdb sql {
>> use_worker = yes
>>
Simple sha512 without multiple rounds is very fast and is not the cause of auth
slowness. But did you mean CRYPT-SHA512 instead?
If the problem isn't due to password hashing, I wonder why it is so slow. Is
the PostgreSQL library really being that slow? Would be interesting to get perf
traces fr
You are right, we are using sha512 and are discussing mass migrating to
sha256 if there is no other way to improve performance of the auth service
itself. but I'm afraid this will increase productivity by a maximum of two
times and the most loaded of the servers will restore the connection pool
aft
What IO do you mean? Service in memory, postgres too, how simple sql query
can be related to IO?
I think it depends on the processor, but seriously, each of our servers
have 10+K users. 48 per second is very low performance in this case. No
packet loss and I talk about the server restart situation
>
> it's definitely not postgres (and "one core" problem for postgres
> related
> with one heavy connect or data modifying, not for "select from")
>
> I see only one process that uses CPU100% with a high amount of logins
> and
> it's auth. And I see only one 100% loaded core on the server.
>
>
I'm guessing the problem is not so much pgsql traffic, than password
veriication. If you are using heavy algorithm like ARGON2 or CRYPT-SHA512
(e.g.) you need to use a worker to distribute the password hashing effort,
otherwise it'll be all done by the main auth process which can become a
bottl
I hope I know how to use top\htop and so on. ;)
And I work with postgres for like 10 years compared to dovecot, which i use
less then year.
it's definitely not postgres (and "one core" problem for postgres related
with one heavy connect or data modifying, not for "select from")
I see only one pro
2.4 docs
"The auth workers are used to execute ___blocking___ passdb and userdb
queries (e.g., MySQL and PAM). They are automatically created and destroyed
as necessary."
postgresql is a non blocking db allowing async queries.
But I'll try to check it on the nearest days.
On Thu, Feb 13, 2025 at
> "Jochen" == Jochen Bern via dovecot writes:
> On 13.02.25 13:55, Anatoliy Zhestov wrote:
> [...]
>> 5 increase the pool in postgres
> [...]
>> However, the performance of the auth service seems limited by one core and
>> cannot rise above 30-40 logins per second for our processor (60-70 wit
You should be able to use https://doc.dovecot.org/2.4.0/core/summaries/
settings.html#passdb_use_worker with pgsql too.
Aki
On 13/02/2025 15:47 EET Anatoliy Zhestov via dovecot
wrote:
Well, I try, but according to documentation - auth-workers are used
for
m
On 13.02.25 13:55, Anatoliy Zhestov wrote:
[...]
5 increase the pool in postgres
[...]
However, the performance of the auth service seems limited by one core and
cannot rise above 30-40 logins per second for our processor (60-70 with
warm cache).
Still, is there any way to parallelize this?
W
Well, I try, but according to documentation - auth-workers are used for
mysql and not for postgres.
Nothing changes. auth process raises cpu% to 100 and performance not
raised anyway.
On Thu, Feb 13, 2025 at 5:08 PM Aki Tuomi
wrote:
> Try setting
>
> auth_cache_verify_password_with_worker=yes
>
I make a big login-pass list, write a python script with threads which make
authentication, shuffle lists for each thread to avoid cache effect and run
it with different numbers of threads printing statistics in separate async
thread.
On Thu, Feb 13, 2025 at 5:10 PM Marc wrote:
> How did you do
How did you do the test?
>
> I set up a test server and started testing it by multiple threads (1-
> 200),
> using all the advice given to me.
> 1 I disabled the postlogin script
> 2 raised maxconns to 200
> 3 set service_count = 400, process_min_avail = 100
> 4 increase auth_cache
> 5 increase
Try setting
auth_cache_verify_password_with_worker=yes
and see if this helps?
Aki
> On 13/02/2025 14:55 EET Anatoliy Zhestov via dovecot
> wrote:
>
>
> I set up a test server and started testing it by multiple threads (1-200),
> using all the advice given to me.
> 1 I disabled the postlogi
I set up a test server and started testing it by multiple threads (1-200),
using all the advice given to me.
1 I disabled the postlogin script
2 raised maxconns to 200
3 set service_count = 400, process_min_avail = 100
4 increase auth_cache
5 increase the pool in postgres
6 enable high-performance
>
> 1) If pgsql is the bottleneck, try multiple pgsql connections: Add
> maxconns=4 (or whatever) to the dovecot-sql.conf.ext's connect line.
Great thanks for this hint. I can't imagine how I could have missed this
parameter.
If I increase maxconns to 200+, does it make sense to use pgbouncer in
>
> I can't really write anything useful. On the client side MUA you can't do
> anything these are all 'unique' connections. You can only optimize to your
> connection to the database. But I am not using this kind of setup. So I
> don't really know. I am using ldap and ldap stuff is being cached by
Good idea, thanks.
As I understand it, this is a one of a standard modboa installation script,
but I think we won't have big problems with updates if we make a trigger
instead of this script.
However, I still don't quite understand what event to assign it to. The
database doesn't know anything abou
On 3. Feb 2025, at 20.18, Anatoliy Zhestov via dovecot
wrote:
>
>>
>> Are you sure the problem is authentication / pgsql? You could test with
>> looping "doveadm auth lookup $user" rapidly. Of course for different users
>> to avoid them coming from cache. Or if you can reproduce it that way, tr
>
>
> Probably not as you only restarted. The limit is when you have create a
> new
> > connection and don't use an existing one.
>
> I don't see a way to reuse an existing connection yet if the number of
> persistent connections after a restart should increase from 0 to 40K
> I miss something o
>
> Do you mean that the contents of this file are not cached? or that there
> is
> some limit on the number of simultaneous requests to read it?
> content of this file not look too heavy
>
> > psql -c "UPDATE core_user SET last_login=now() WHERE username='$USER'"
> >
> > /dev/null
I would put a
> >
> > Long ago in MagicMail we recognized that there was a problem with
> > Postgres direct back addressing a large number of AUTH attempts at
> once,
> > and we went down the road to using a high performace DBFILE
> mechanisms,
> > with real time updates from the Postgres database.. allowed us t
>
> Oh yes? What is this then /usr/local/bin/postlogin.sh
> I don't know you have to look at what is different after the restart.
Do you mean that the contents of this file are not cached? or that there is
some limit on the number of simultaneous requests to read it?
content of this file not loo
>
> Long ago in MagicMail we recognized that there was a problem with
> Postgres direct back addressing a large number of AUTH attempts at once,
> and we went down the road to using a high performace DBFILE mechanisms,
> with real time updates from the Postgres database.. allowed us to scale
> grea
> No, we don't have the noticeable iowait problem as I see it(at least
> until
> the number of connections lower 20-30K). The problem appears when
> thousands
> of clients try to reconnect at the same time and according to the
> documentation the auth service should make a simple request to postgre
>
> Are you sure the problem is authentication / pgsql? You could test with
> looping "doveadm auth lookup $user" rapidly. Of course for different users
> to avoid them coming from cache. Or if you can reproduce it that way, try
> if the same happens for repeating the same user so it does come from
No, we don't have the noticeable iowait problem as I see it(at least until
the number of connections lower 20-30K). The problem appears when thousands
of clients try to reconnect at the same time and according to the
documentation the auth service should make a simple request to postgres for
this.
On 3. Feb 2025, at 7.05, Anatoliy Zhestov via dovecot
wrote:
>
> Hi. We have a performance problem with imap authentication through
> postgresql.
> Our servers(modoboa based) have a big amount of permanent imap
> connections(5000-5).
> Current performance is about 3000 successful authenticat
> Current performance is about 3000 successful authentications per hour.
> No
I don't really get this authentication attempts is limitted by tcp not? So it
does not really matter what you have mariadb, ldap, http, you awalys are
limited to 150-200 r/s. Once you have a connection, you can go easi
32 matches
Mail list logo
