Hi
I can confirm the issue of these iteration operation not completing for
1000's of users is linked to:
MAX_OUTBUF_SIZE defined in auth/auth-master-connection.c
I made it 1024000 and managed to complete quota recalc for 10000 users.
But that's not going to be a solution, since whatever it is set to
provides a hard coded limit to how many users can be iterated for quota
get, quota recalc and index operations. Surprisingly it is not a
completely deterministic limit, since sometimes more users get processed
than other times. I was looking for a time based limit before I thought of
looking at the buffer size.
The strange thing is that the current value hasn't changed in more than 20
years if ever, so that does not explain of itself why this issue arose now
in 2.4.
#define MAX_OUTBUF_SIZE (1024*50)
The likely explanation is that exceeding the buffer size is now being
handled differently to before.
John
On 28/02/2026 15:35, John Fawcett via dovecot wrote:
On 27/02/2026 17:56, Philip Iezzi via dovecot wrote:
...
Yes, I am definitely running into some kind of limit. It's not on
MySQL
level - tuned it a lot for high performance and connection
hammering, and
never found any error in MySQL log.
Above workaround is now running for a full day, pulling all user
quotas
every 4mins, without running into this issue again.
As `doveadm quota get -A` was always running fine on Dovecot 2.3
and I can
now (under Dovecot 2.4.2) retrieve all quotas by piping all users
into
`-F`, it should definitely be possible to fix/optimize the `-A`
iterate_query for mailservers with larger user base.
@Aki did you guys experience this issue and can I provide you any
more
information to help you get this fixed?
I can definitely live with my workaround, just hope others are not
struggling with it and wasting so much time as I did.
Best regards,
Philip
Hi Philip
this is reproducible by filling the usertable with around 2000 users and
running doveadm quota get -A. For me it also fails on
doveadm quota recalc -A
/usr/bin/doveadm index -A '*'
As you already saw the process never completes and gives an error on a
different user each time though roughly somewhere between 2-3 seconds
into the execution for quota recalc.
quota get seems to present slightly more variable behaviour in terms of
how long before it fails but that could be influenced by terminal output
time.
The log entry I saw was "auth: Error: auth-worker: Aborted LIST request
for *: Shutting down"
doveadm user '*' does complete ok, but that only does the iterate query,
whereas the other commands do the iterate and then lookup the single
users too.
I don't see any configurable timeout limits that seem relevant to this
issue, so it looks like a bug when running commands needing to iterate
through 1000s of users with the iterate_query and then doing lookups for
the single users user the userdb query.
John
_______________________________________________
dovecot mailing list -- [1][email protected]
To unsubscribe send an email to [2][email protected]
References
Visible links
1. mailto:[email protected]
2. mailto:[email protected]
_______________________________________________
dovecot mailing list -- [email protected]
To unsubscribe send an email to [email protected]
