On 5.3.2012, at 14.15, Attila Nagy wrote: >>> On 03/04/12 11:44, Timo Sirainen wrote: >>>> In dovecot-2.1 hg you can now test dsync-based replication. Everything >>>> isn't finished yet, but it appears to work and I've enabled it for my >>>> @dovecot.fi mails. Some issues: >>>> >>> Do you plan to make it more performant in the future? I mean calling >>> doveadm (and ssh) for every change -even when they are aggregated- seems to >>> be very resource intensive, it won't keep up on a machine with a lot of >>> modifications happening every seconds. >> Sure the idea is to improve the performance :) There are two ways: >> >> 1) Use longer running SSH sessions which dsync more than one user at a time. >> >> 2) Use TCP connections instead of SSH. > Don't forget about connection pooling to get concurrency. :)
There's already concurrency. replication_max_conns (default 10) specifies how many dsyncs can be running concurrently. > BTW, despite being somewhat harder to implement, I personally like native > connections better. Native = TCP? It's not difficult, probably a few lines of more code since doveadm server can already listening for TCP connections. It doesn't support SSL though. >>> It would be good to have constantly running daemons on both sides to >>> eliminate the high startup/teardown costs. >> The process startup/teardown costs are pretty low. I'll need to improve >> dsync's performance at some point though. Actually I pretty much redesigned >> the whole dsync already, but I'll probably leave that to v2.2. The current >> design can still be improved. >> > It depends. For a moderately loaded server I get this: > # time ssh root@be02 "echo 1" I meant doveadm/dsync costs, ssh startup is rather slow. > Yes, dsync seems to need some optimizations too. :) > I've tried previously on one pair of our servers with a higher level of > concurrency (8-16 or so, I can't remember), and it couldn't keep up with the > changes. > The method was similar to yours: > - an external library wrote modified user ids to a file > - in an endless loop a script picked up those (moved the file) and started > parallel dsyncs (on ssh) > > The runs were longer and longer... dsync doesn't currently take enough advantage of modseqs and send only the changed data. > BTW, we modify the maildirs externally, so this adds a lot of inefficiency > here... Definitely doesn't help.
