[Dovecot] performance of maildir on ocfs2
Hi, I would like to run my imap service on a active-active cluster. I wonder how well OCFS2 performs in read and write with millions of smallest files involved. Has anybody got any experience? Thanks John
Re: [Dovecot] performance of maildir on ocfs2
On 26.04.2010 14:37, mailingli...@belfin.ch wrote: Hi, I would like to run my imap service on a active-active cluster. I wonder how well OCFS2 performs in read and write with millions of smallest files involved. Has anybody got any experience? Thanks John I have some experience. I have done some test and benchmarks here for OCFS2 and NFS4. My understanding is that GFS2 limitations and performance are similar to OCFS2's, so I have not included it in the tests. My benckmarks showed that OCFS2 has an order of magnitude better performance for IMAP server (for benchmarks I have used imaptest). Though, there are some concerns: 1. For better performance you would like to stick one account to one server. If one account is checked concurrently from different nodes of OCFS2 cluster they invalidates each-other the direcotory cache for the Maildir. It result in a lot more IO. I use nginx for load balancing incomming pop3/imap connections if front of dovecot servers. 2. OCFS2 has a limit for 32k files per directory. There is directory index support in recent kernels (> 2.6.33) that will remove this limit, but the userland tools are not yet ready for production (not in the master branch of ocfs2-tools) 3. My understanding is that OCFS2 uses a global lock for move/rename. As you know, Maildir format uses a lot of such operations. I think that dbox format (dovecot native) will be better choice, because there are no file moves/renames. I am planning migration to dbox now. If I have to start the service now, I would choose dbox for mail storage. 4. Filesystem quota. OCFS2 has support in recent kernels. It has integrated in ocfs2-tools master branch but there is not yet release of the tools. So I do not hurry to push it in production. I hope it hepls Best regards -- Luben Karavelov Research and development Spectrum Net JSC 36, D-r G. M. Dimitrov Blvd. 1797 Sofia Mobile: +359 884332140 url: www.spnet.net
[Dovecot] zlib plugin dovecot 2.0 - write email
Hello Everyone. There is possible to rename the email writed by a plugin, like zlib ? i explain i need to add 1 line in email header, because of that, i want to rewrite the filename to update the S and W flag. Tks !
[Dovecot] Assertion failed in mail-search-build.c
With Friday's and today's mercurial repo, I'm getting this: Apr 26 12:55:31 master: Info: Dovecot v2.0.beta4 starting up (core dumps disabled) Apr 26 12:58:34 imap-login: Info: Login: user=, method=PLAIN, rip=10.1.1.206, lip=192.168.152.1 Apr 26 12:58:34 imap(m...@box): Panic: file mail-search-build.c: line 59 (mail_search_build_key_int): assertion failed: (sarg->value.subargs != NULL) Apr 26 12:58:34 imap(m...@box): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0xb76a3ad1] -> /usr/lib/dovecot/libdovecot.so.0 [0xb76a3b4f] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0xb76a3e08] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0xb76f859e] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0xb76f860f] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_build+0x82) [0xb76f86f2] -> dovecot/imap [m...@box 10.1.1.206 SEARCH](imap_search_args_build+0x48) [0x805d8a8] -> dovecot/imap [m...@box 10.1.1.206 SEARCH](cmd_search+0xbb) [0x805342b] -> dovecot/imap [m...@box 10.1.1.206 SEARCH] [0x80564dc] -> dovecot/imap [m...@box 10.1.1.206 SEARCH] [0x8056579] -> dovecot/imap [m...@box 10.1.1.206 SEARCH](client_handle_input+0x2d) [0x80566ed] -> dovecot/imap [m...@box 10.1.1.206 SEARCH](client_input+0x5f) [0x805710f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xf5) [0xb76b0115] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x30) [0xb76af160] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x2a) [0xb769ca5a] -> dovecot/imap [m...@box 10.1.1.206 SEARCH](main+0x2b5) [0x805fb95] -> /lib/libc.so.6(__libc_start_main+0xe5) [0xb752e455] -> dovecot/imap [m...@box 10.1.1.206 SEARCH] [0x804e951] Apr 26 12:58:34 master: Error: service(imap): child 13220 killed with signal 6 (core dumps disabled) Login is working fine, just blowing up on that assert by doing a search. The search is pretty basic, in python: server.search( None, "(ALL)" ) Testing with a previous version (from a couple of days earlier -- I don't have the intervening), I don't hit that assertion. The last changeset on mail-search-build.c in the working copy is 11146:3649ad8c9f9a. Let me know if there's anything you'd like me to try.
Re: [Dovecot] performance of maildir on ocfs2
On 26.04.2010 14:51, karavelov wrote: On 26.04.2010 14:37, mailingli...@belfin.ch wrote: Hi, I would like to run my imap service on a active-active cluster. I wonder how well OCFS2 performs in read and write with millions of smallest files involved. Has anybody got any experience? Thanks John I have some experience. I have done some test and benchmarks here for OCFS2 and NFS4. My understanding is that GFS2 limitations and performance are similar to OCFS2's, so I have not included it in the tests. My benckmarks showed that OCFS2 has an order of magnitude better performance for IMAP server (for benchmarks I have used imaptest). Though, there are some concerns: 1. For better performance you would like to stick one account to one server. If one account is checked concurrently from different nodes of OCFS2 cluster they invalidates each-other the direcotory cache for the Maildir. It result in a lot more IO. I use nginx for load balancing incomming pop3/imap connections if front of dovecot servers. > 2. OCFS2 has a limit for 32k files per directory. There is directory index support in recent kernels (> 2.6.33) that will remove this limit, but the userland tools are not yet ready for production (not in the master branch of ocfs2-tools) > 3. My understanding is that OCFS2 uses a global lock for move/rename. As you know, Maildir format uses a lot of such operations. I think that dbox format (dovecot native) will be better choice, because there are no file moves/renames. I am planning migration to dbox now. If I have to start the service now, I would choose dbox for mail storage. 4. Filesystem quota. OCFS2 has support in recent kernels. It has integrated in ocfs2-tools master branch but there is not yet release of the tools. So I do not hurry to push it in production. So bottom line is to use a filesystem such as XFS, distribute and dedicate mailboxes to a number of backend imap servers optimally with direct access to the storage and do imap proxying and loadbalancing in front of those servers. I hope it hepls Thanks, it did! Best wishes
[Dovecot] adding users to passwd-file
Is there a tool equivalent to the system "passwd" command (or maybe "adduser" or "useradd") that can support a passwd-file by setting a password, encrypting it with the salted MD5 scheme? The system "passwd" command doesn't have an option to "do it to this alternate file instead of /etc/shadow".
Re: [Dovecot] adding users to passwd-file
Phil Howard (Mo 26 Apr 2010 22:31:45 CEST): > Is there a tool equivalent to the system "passwd" command (or maybe > "adduser" or "useradd") that can support a passwd-file by setting a > password, encrypting it with the salted MD5 scheme? The system "passwd" > command doesn't have an option to "do it to this alternate file instead of > /etc/shadow". Some people use „htpasswd“, but there not password scheme is coded into the password hash. It's not clear what you're seeking - some tool for generating the hashed string (e.g. „openssl passwd -1 "$cleartext"“) or some tool for managing the entries in the (dovecot) password file. Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de internet & unix support - Heiko Schlittermann HS12-RIPE - gnupg encrypted messages are welcome - key ID: 48D0359B --- gnupg fingerprint: 3061 CFBF 2D88 F034 E8D2 7E92 EE4E AC98 48D0 359B - signature.asc Description: Digital signature
Re: [Dovecot] adding users to passwd-file
On Mon, Apr 26, 2010 at 4:38 PM, Heiko Schlittermann wrote: > Some people use „htpasswd“, but there not password scheme is coded into > the password hash. > > It's not clear what you're seeking - some tool for generating the hashed > string (e.g. „openssl passwd -1 "$cleartext"“) or some tool for managing > the > entries in the (dovecot) password file. > I left it vague to handle some variety. Something to add/delete users is nice. Something to change password in place is nice. Something that given a user and plain text password and everything else Dovecot needs when using it as userdb, and would output to stdout the line contents that would go into the file, would do the job. I want to avoid re-inventing the wheel. If there are no wheels, I guess I'll just look up the salted-MD5 format details and do it.
Re: [Dovecot] adding users to passwd-file
On Mon, April 26, 2010 1:46 pm, Phil Howard wrote:
> On Mon, Apr 26, 2010 at 4:38 PM, Heiko Schlittermann
> wrote:
>
>> Some people use htpasswd, but there not password scheme is coded into
>> the password hash.
>>
>> It's not clear what you're seeking - some tool for generating the hashed
>> string (e.g. openssl passwd -1 "$cleartext") or some tool for managing
>> the
>> entries in the (dovecot) password file.
>>
>
> I left it vague to handle some variety. Something to add/delete users is
> nice. Something to change password in place is nice. Something that
> given
> a user and plain text password and everything else Dovecot needs when
> using
> it as userdb, and would output to stdout the line contents that would go
> into the file, would do the job.
>
> I want to avoid re-inventing the wheel. If there are no wheels, I guess
> I'll just look up the salted-MD5 format details and do it.
Here's a little perl script you can start with for creating the MD5 salted
password (slightly modified from something I found on the net a couple of
years ago):
=
#!/usr/bin/perl -wl
use strict;
use Crypt::PasswdMD5 qw(unix_md5_crypt);
my @salt = ( '.', '/', 0 .. 9, 'A' .. 'Z', 'a' .. 'z' );
# take clear-text password as argument:
my $password = shift || die "usage: $0 password";
my %encrypted;
# generate md5 password
$encrypted{md5} = unix_md5_crypt( $password, gensalt(8) );
print "$_ $encrypted{$_}" for sort keys %encrypted;
# uses global @salt to construct salt string of requested length
sub gensalt {
my $count = shift;
my $salt;
for (1..$count) {
$salt .= (@salt)[rand @salt];
}
return $salt;
}
=
You will need to have the Crypt::PasswdMD5 perl module installed to use this.
Bill
Re: [Dovecot] adding users to passwd-file
> Is there a tool equivalent to the system "passwd" command The dovecotpw command may be a good place to start.
Re: [Dovecot] performance of maildir on ocfs2
On 26.04.2010 21:42, Philipp Snizek wrote: So bottom line is to use a filesystem such as XFS, distribute and dedicate mailboxes to a number of backend imap servers optimally with direct access to the storage and do imap proxying and loadbalancing in front of those servers. Then you should also balance incoming mail and local deliveries and this part is tricky. Also, there should be a hearthbeat/pacemaker for filesystem/IP/services failover. Every choice is a compromise. So you should balance a lot of factors: complexity, administrative overhead, FS limits, performance, stability etc. Best regards luben
