Re: [Dovecot] Replication protocol design

2008-05-02 Thread Ed W
Timo Sirainen wrote: I agree that it would be nice to offer this capability for IMAP clients, but I don't see a way to do this in any reasonable way. If the client wants a continuous replication for all mailboxes the resulting protocol will barely even look like IMAP anymore. And trying to merge

Re: [Dovecot] Replication protocol design

2008-05-02 Thread Timo Sirainen
On Fri, 2008-05-02 at 07:30 +0100, Ed W wrote: > Timo Sirainen wrote: > > Using IMAP protocol for replication has at least two disadvantages: > > > > It occurs to me that there is at least one advantage of putting as much > as possible into the imap protocol: > > mailclients are kind of a spe

Re: [Dovecot] Replication protocol design

2008-05-02 Thread Ed W
Asheesh Laroia wrote: On Thu, 1 May 2008, Ed W wrote: I currently use a small self written proxy app which does some simple analysis of what imap client is talking and does some prefetching via pipelined commands to reduce latency and also sets up a compressed pipe back to the server. Even o

Re: [Dovecot] Replication protocol design

2008-05-02 Thread Asheesh Laroia
On Thu, 1 May 2008, Ed W wrote: I currently use a small self written proxy app which does some simple analysis of what imap client is talking and does some prefetching via pipelined commands to reduce latency and also sets up a compressed pipe back to the server. Even over broadband it gives

Re: [Dovecot] Replication protocol design

2008-05-01 Thread Ed W
Timo Sirainen wrote: Using IMAP protocol for replication has at least two disadvantages: It occurs to me that there is at least one advantage of putting as much as possible into the imap protocol: mailclients are kind of a special of replication client. Really they want to be able to do

Re: [Dovecot] Replication protocol design #2

2008-05-01 Thread Ed W
- 32bit server ID (configurable? based on IPv4 address? 48bit MAC address by reducing PID/timestamp by 16bits? is exposing these to a normal user a security problem?) If in doubt MD5 the information and pick some bits from the MD5 hash

Re: [Dovecot] Replication protocol design #2

2008-05-01 Thread Timo Sirainen
On May 1, 2008, at 7:46 PM, Charles Marcus wrote: I'm also very keen on seeing full support for Shared Folders and client managed ACLs... Has this fallen to the back-burner? I seem to recall this was planned for 1.2, but haven't heard much about it since... At least some kind of a shared

Re: [Dovecot] Replication protocol design #2

2008-05-01 Thread Charles Marcus
On 5/1/2008 7:36 AM, Timo Sirainen wrote: Changes: Hi Timo, The replication work sounds way cool and I look forward to being able to take advantage of it, but... I'm also very keen on seeing full support for Shared Folders and client managed ACLs... Has this fallen to the back-burner? I

Re: [Dovecot] Replication protocol design #2

2008-05-01 Thread Timo Sirainen
Added "current modseq" parameter to all commands and MODSEQ command. Other things missing from the replication protocol are related to mailbox list, i.e. figuring out which mailboxes to replicate, which are newly created, deleted, renamed, etc. There will probably have to be a new "unique global ma

Re: [Dovecot] Replication protocol design

2008-05-01 Thread Ed W
Dovecot stores flag changes as "added flags" and "removed flags" in transaction file, so it doesn't need to do any comparing to figure out what had changed. This makes the flag changes also more reliable. For example if a message originally had flags (\Flagged) and then two servers changed

Re: [Dovecot] Replication protocol design

2008-05-01 Thread Timo Sirainen
On May 1, 2008, at 3:23 PM, Ed W wrote: 2) Sending updated flag/keyword changes can't be done in a standard way, because it only shows the last flag+keyword state, not the changes that were done (e.g. "\Seen" vs. "+\Seen -\Flagged"). Hmm... I guess that can only be done anyway by storing

Re: [Dovecot] Replication protocol design #2

2008-05-01 Thread Ed W
8. Make non-incremental synchronization fast enough to allow replication with users' untrusted Dovecot servers which may connect only rarely. Support also super high latency replication (e.g. using USB sticks). The USB stick sync in particular would be a MASSIVELY powerful feature for an admitt

Re: [Dovecot] Replication protocol design

2008-05-01 Thread Ed W
Hi The updated design #2 should address this. The mailbox synchronization step works pretty much the same as QRESYNC. Thanks will look through that a) Add a FETCH-SMALL command that drops large attachements and somehow remembers this so that they could later be downloaded again when there

Re: [Dovecot] Replication protocol design #2

2008-05-01 Thread Timo Sirainen
On Thu, 2008-05-01 at 14:36 +0300, Timo Sirainen wrote: > Support also super high latency replication (e.g. using USB sticks). If it wasn't clear how this works: "Client" and "server" are both Dovecots, client just has no/slow network connection. 1. Client runs a command to create a replication

Re: [Dovecot] Replication protocol design

2008-05-01 Thread Timo Sirainen
On Tue, 2008-04-29 at 16:28 +0100, Ed W wrote: > I think they imapsync style protocol is the most powerful starting point > for master/master sync. However, that style of sync can be made more > efficient by using the LEMONADE features you previously added, in > particular by offering a limite

[Dovecot] Replication protocol design #2

2008-05-01 Thread Timo Sirainen
Changes: - Added goal 8 and rewrote mailbox synchronization plan. - Added new SELECT command to change active mailbox and removed mailbox ID from command parameters Goals - 1. If a single (or configurable number of) server dies, no mails must be lost that have been reported to IMAP/SMTP cl

Re: [Dovecot] Replication protocol design

2008-04-29 Thread Tommi Virtanen
On Tue, Apr 29, 2008 at 02:59:52AM +0300, Timo Sirainen wrote: > > 1. A mail arrives in server1 > > 2. User reads it (therefore the mail has a UID assigned) > > 3. server2 gets isolated from server1 > > 4. A mail arrives at server2 > > 5. User logs in to server2 and reads it (therefore the mail has

Re: [Dovecot] Replication protocol design

2008-04-29 Thread Ed W
Timo Sirainen wrote: I'll probably be implementing multi-master replication this summer. My previous thoughts about it are here: http://dovecot.org/list/dovecot/2007-December/027284.html Below is a description of how the replication protocol will probably work. It should work just as well for ma

Re: [Dovecot] Replication protocol design

2008-04-28 Thread Timo Sirainen
On Mon, 2008-04-28 at 16:33 -0700, Asheesh Laroia wrote: > Multi-master will be very interesting. I'm really curious what you will > do as far as this scenario: > > 1. A mail arrives in server1 > 2. User reads it (therefore the mail has a UID assigned) I also thought about this kind of a late U

Re: [Dovecot] Replication protocol design

2008-04-28 Thread Asheesh Laroia
On Tue, 29 Apr 2008, Timo Sirainen wrote: I'll probably be implementing multi-master replication this summer. My previous thoughts about it are here: http://dovecot.org/list/dovecot/2007-December/027284.html Below is a description of how the replication protocol will probably work. It should wo

[Dovecot] Replication protocol design

2008-04-28 Thread Timo Sirainen
I'll probably be implementing multi-master replication this summer. My previous thoughts about it are here: http://dovecot.org/list/dovecot/2007-December/027284.html Below is a description of how the replication protocol will probably work. It should work just as well for master-slave and multi-ma