-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Tom Collins wrote:
> On Jan 22, 2009, at 1:43 AM, Matt Brookings wrote:
>> Where do you come up with 128bit from? On most systems a 'long long'
>> is going
>> to be a 64bit integer. That's what I'm currently using.
>
> Sorry, you're right on that.
>
> Why isn't it enough to keep the current quota system, and just update
> all related code to use a long long (64 bits) for the byte counter?
> Will quotas ever grow that huge? In the next 10 years?
I'm not replacing the current quota system, and I agree, we should update
all the quota code in vpopmail to be 64bits.
> You could even keep the 32 bit numbers, and just update all
> quota-checking code to round sizes up to the nearest KB -- now you're
> able to have quotas up to 2000 GB (2TB). Since a single email isn't
> going to be 2GB, each entry in maildirsize can still be a 32-bit int and
> you may not have to modify qmail or the POP/IMAP servers. You only need
> to update the code that rebuilds the file and checks for an overquota
> condition.
Here's what I'm doing. I have already created a daemon which answers requests
about user and domain storage. It has a set of rules by which it operates.
1) Always respond quickly, even if the information is not up to date
This is to be sure that processes requesting information on user usage,
like vdelivermail, or an SMTP daemon, never bottleneck on the daemon
if the system is under heavy load, or fail even if the daemon is not
running
or fails.
This is supported by both the server and client code. Neither are allowed
to cause a bottleneck.
2) If a request will take too long to fill and there is no information
available,
it must return a zero usage response so that delivery will not fail. It's
more important to never fail erroneously than to keep quotas enforced in
real-time.
Again, supported by both the server and client.
3) All updates can be deferred into a queue to be handled at reasonable I/O
levels.
If a request will take a very long period of time, or is known to use
large amounts
of disk I/O, for instance, opening a directory with 150k messages in it,
the last
known information on this directory will be used and the directory can be
tagged as
an extreme I/O hog to further defer future updates.
This will have it's own set of rules for dealing with a growing queue of
deferred
updates, etc.
4) Do not enforce, or care, about quotas
The goal of the daemon is not to provide quota enforcement, it merely
provides
information on user usage. Comparing usage to quotas, and enforcing
quotas is entirely
up to the program using the client API.
The client API is very easy to use. Here is an example:
// storage_t is a typedef of a 64bit unsigned integer defined by the
server code
storage_t user_usage = 0, domain_usage = 0;
cHandle = client_connect();
// Error checking here
client_query(cHandle, "u...@domain", strlen("u...@domain"), &user_usage,
&domain_usage);
if (user_usage >= user_quota) fail();
if (domain_usage >= domain_quota) fail();
I've been running this code on a few production servers where vdelivermail and
qmail-smtpd were
patched not to enforce quotas, but simply to run queries on every user it sees.
Most queries
are responded to in 0.010 seconds or less with accuracy to within 15 seconds of
actual disk usage.
There's still lots of work to be done on it, but I hope to release it in a beta
form after
I return from vacation Feb 2nd.
My current TODO:
* Disk updates must be deferrable; maybe with threading
* Statistics gathering like average disk poll time, biggest poll time,
shortest poll time, etc
As you can see, the deferring updates is not yet implemented. The performance
I'm currently
seeing is acceptable, but it would probably not cut it on a *very* large system
with many absent
users and catchalls collecting gigs of email in a single directory.
I hope to release a beta to developers once the deferring code is ready.
Opinions on this? :)
- --
/*
Matt Brookings <m...@inter7.com> GnuPG Key D9414F70
Software developer Systems technician
Inter7 Internet Technologies, Inc. (815)776-9465
*/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFJejRK6QgvSNlBT3ARAnXvAJ4xXnJp4WpSoTR1pHcmXrY1MuRY6ACgqyjM
6fQJQoatA7+31qxgJqB7ViY=
=b74b
-----END PGP SIGNATURE-----
