On Aug 11, 2009, at 10:32 AM, Steffen Kaiser wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Mon, 10 Aug 2009, Timo Sirainen wrote:
4. Implement a multi-master filesystem backend for index files. The
idea
would be that all servers accessing the same mailbox must be
talking to
each others via network and every time something is changed, push the
change to other servers. This is actually very similar to my previous
multi-master plan. One of the servers accessing the mailbox would
still
act as a master and handle conflict resolution and writing indexes to
disk more or less often.
What I don't understand here is:
_One_ server is the master, which owns the indexes locally?
Oh, 5. means that this particular server is initiating the write,
right?
Yes, only one would be writing to the shared storage.
You spoke about thousends of servers, if one of them opens a
mailbox, it needs to query all (thousends - 1) servers, which of
them is probably the master of this mailbox. I suppose you need a
"home location" server, which other servers connect to, in order to
get server currently locking (aka acting as master for) this mailbox.
Yeah, keeping track of this information is probably the most difficult
part. But surely it can be done faster than with (thousands-1)
queries :)
There is also another point I'm wondering about:
index files are "really more like memory dumps", you've wrote. so if
you cluster thousends of servers together you'll most probably have
different server architectures, say 32bit vs. 64bit, CISC vs. RISC,
big vs. little endian, ASCII vs. EBCDIC :). To share these memory
dumps without another abstraction layer wouldn't work.
Nah, x86 is all there is ;) Dovecot has been fine so far with this
same design. I think only once I've heard that someone wanted to run
both little and big endian machines with shared NFS storage. 32 vs. 64
bit doesn't matter though, indexes have been bitness-independent since
v1.0.rc9.
I was tried to make the code use the same endianess everywhere, but
the code quickly became so ugly that I decided to just drop it. But
who knows, maybe some day. :)
5. Implement filesystem backend for dbox and permanent index storage
using some scalable distributed database, such as maybe Cassandra.
This
Although I like the "eventually consistent" part, I wonder about the
Java-based stuff of Cassandra.
I'm not yet sure what database exactly to use. I'm not really familiar
with any of them, except the Amazon Dynamo whitepaper that I read, and
that seemed perfect to me. Cassandra still seems to lack some features
that I think are needed.
is the part I've thought the least about, but it's also the part I
hope
to (mostly) outsource to someone else. I'm not going to write a
distributed database from scratch..
I wonder if the index-backend in 4. and 5. shouldn't be the same.
You mean the permanent index storage? Yes, it probably should be the
same in 4 and 5. 4 just has that in-memory layer in the middle.
How many work is it to handle the data in the index files?
What if any server forwards changes to the master and recieves
changes from the master to sync its local read-only cache? So you
needn't handle conflicts (except when network was down) and writes
are consistent originated from this single master server. The actual
mail data is accessed via another API.
When the current master does no longer need to access the mailbox,
it could hand over the "master" stick to another server currently
accessing the mailbox.
http://dovecot.org/tmp/replication-plan.txt explains how I previously
thought about the index replication to work, and I think it'll still
work pretty nicely with the index FS backend too. I guess it could
mostly work like sending everything to master, although for some
changes it wouldn't really be necessary. I'll need to rethink the plan
for this I guess.
