Re: New Cyrus project site and bugzilla

2010-09-15 Thread Matt Selsky
On Mon, 13 Sep 2010, Mark Cave-Ayland wrote: > (On a separate note, if I go to Downloads -> Getting Started and click > on the "AnonymousCVS" wiki link then I get redirected back to the front > page rather than to a page giving information on how to access CVS) Fixed. Cyrus Home Page: http:/

Re: De-duping attachments

2010-09-15 Thread Bron Gondwana
On Wed, Sep 15, 2010 at 05:24:11PM +0100, Gavin McCullagh wrote: > Hi, > > On Wed, 15 Sep 2010, Nik Conwell wrote: > > > Isn't the easy hack for dedup just looking at the above md5 files and > > then doing appropriate hard links? This could be done by a nightly > > trawl of the spool space. A

Re: De-duping attachments

2010-09-15 Thread Patrick Goetz
On 09/14/2010 11:55 PM, Rob Mueller wrote: > > Eg. An architectural firm > might end up sending big blueprint documents back and forth between each > other a lot, so they'd gain a lot from deduplication. > Not to throw a damp towel on this discussion, but isn't this really an administrative probl

Re: sync-server without deletes?

2010-09-15 Thread Henrique de Moraes Holschuh
On Wed, 15 Sep 2010, Shuvam Misra wrote: > > Well, unless you have users delivering mail to each other through IMAP > > on shared folders, one usually configures the MTAs to drop a copy of > > everything into a system mailbox... > > Yes, this is what we do too. We have a milter in Sendmail which a

Re: De-duping attachments

2010-09-15 Thread Gavin McCullagh
Hi, On Wed, 15 Sep 2010, Nik Conwell wrote: > Isn't the easy hack for dedup just looking at the above md5 files and > then doing appropriate hard links? This could be done by a nightly > trawl of the spool space. A bigger win would be to separate the headers > from the messages but that's a

Re: De-duping attachments

2010-09-15 Thread Joseph Brennan
Outside the cyrus box: The Mimedefang milter has a built-in function (optional of course) to remove an attachment, write it to a file, and replace the attachment part with a text part giving a web link to the file. The files could be on a slower type of disk drive than you need for email storage

Re: Using cvt_cyrusdb to convert quota database from skiplist back to quotalegacy.

2010-09-15 Thread Bron Gondwana
On Tue, Sep 14, 2010 at 11:58:03PM +0200, Eric Luyten wrote: > Hello, > > > I am having trouble converting a quota skiplist db back to quotalegacy > format (I know... this is probably not the most common Cyrus operation :-) Yeah, odd! I wonder what's going on there. I'll take a look. > Other

Re: De-duping attachments

2010-09-15 Thread Eric Luyten
On Wed, September 15, 2010 2:12 pm, Simon Matter wrote: > You said ZFS, did you > consider testing its built in deduping? > (If its even there in Solaris 10?) Simon, OpenSolaris does have it (block level dedup) since about one year but it is too recent an addition to the commercial Solaris 10 t

Re: Replication sync-server and Delayed Delete

2010-09-15 Thread Bron Gondwana
On Wed, Sep 15, 2010 at 12:29:18PM +0100, Gavin Gray wrote: > Hi there, > > We have a cyrus murder using replication and we have a few questions > about the behaviour we are seeing on our system. > > 1. cyr_expire on the master doesn't cause any replication to happen. > Is that 'correct'? In othe

Re: Mailbox directory structure

2010-09-15 Thread Artur Kaminski
Date: Wed, 15 Sep 2010 14:02:38 +0200 > From: Michael Menge > Subject: Re: Mailbox directory structure > To: info-cyrus@lists.andrew.cmu.edu > Message-ID: <20100915140238.18471f8lfaaqs...@webmail.uni-tuebingen.de> > Content-Type: text/plain; charset="utf-8" > > Quoting Artur Kaminski : > > > Hey

Re: De-duping attachments

2010-09-15 Thread Simon Matter
> On Wed, September 15, 2010 10:01 am, Simon Matter wrote: > >> I guess much more efficient than a compressing filesystem would be a >> compressing and de-duping filesystem or disk storage in this case. Has >> anyone >> tried this with a Cyrus message store with lots of "corporate message >> data"

Re: Mailbox directory structure

2010-09-15 Thread Michael Menge
Quoting Artur Kaminski : Hey all, I installed imapd server successfully, and moved configuration from old one, but then accidentally loaded another server's configuration from puppet. Now Cyrus looks for user mailboxes in User Mailbox /var/spool/imap/a/ user. /var/spo

Re: De-duping attachments

2010-09-15 Thread Nik Conwell
Great thread. Here as some real world numbers based on our spools here at BU. One of our masters has 4,800 users, 22,000 mailboxes, and is using about 374G of disk. Based on the md5 files for these users there are 6,046,363 messages. If I look at the first md5 value (md5 on the msg if I un

Replication sync-server and Delayed Delete

2010-09-15 Thread Gavin Gray
Hi there, We have a cyrus murder using replication and we have a few questions about the behaviour we are seeing on our system. 1. cyr_expire on the master doesn't cause any replication to happen. Is that 'correct'? In other words if we want to delete folders from the DELETED heirarchy on t

Mailbox directory structure

2010-09-15 Thread Artur Kaminski
Hey all, I installed imapd server successfully, and moved configuration from old one, but then accidentally loaded another server's configuration from puppet. Now Cyrus looks for user mailboxes in User Mailbox /var/spool/imap/a/ user. /var/spool/imap/u/user^ (checke

Re: De-duping attachments

2010-09-15 Thread Shuvam Misra
> Makes sense. There might be some size based logic here too - only > bother applying this on messages over 20k, and where the attachment > is at least 20k in size. Anything smaller than that is pretty > pointless. Yes, absolutely. Left to myself, I'd not have bothered with any attachment less t

Re: Importing/moving an older cyrus message tree into a new system, without IMAP

2010-09-15 Thread Shuvam Misra
> Annotations are defined in RFC 5257. > > They allow an admin to add metadata to a mailbox (or the server). The > cyradm utility sets annotations with its internal info, mboxcfg, and > setinfo commands. Okay, checked. Don't know where these things are used, other than expiry and sieve, but at le

Re: De-duping attachments

2010-09-15 Thread Shuvam Misra
The sparse file idea is brilliant! Never occurred to me. :) We'd have to store the reference-pointer in the message file, so we would omit the actual attachment but eat up perhaps 50 bytes to keep the reference to the file. Shuvam > 1. Completely rewrite the message file removing the attachment

Re: De-duping attachments

2010-09-15 Thread Shuvam Misra
Dear Bron, > So you save, what, 50%. Does that sound about right? Do you have > statistics on how much space you'd save with this theoretical > patch? No, and this is the first thing I want to do. I'm getting some simple utilities developed which will run all week (niced suitably) and extract a

Re: Using cvt_cyrusdb to convert quota database from skiplist back to quotalegacy.

2010-09-15 Thread Eric Luyten
On Wed, September 15, 2010 9:27 am, Simon Matter wrote: >> I am having trouble converting a quota skiplist db back to quotalegacy >> format (I know... this is probably not the most common Cyrus operation :-) >> >> % cvt_cyrusdb /ssd/cyrs/imap/quotas.db skiplist /ssd/cyrs/imap/quota >> quotalegacy

Re: De-duping attachments

2010-09-15 Thread Eric Luyten
On Wed, September 15, 2010 10:01 am, Simon Matter wrote: > I guess much more efficient than a compressing filesystem would be a > compressing and de-duping filesystem or disk storage in this case. Has anyone > tried this with a Cyrus message store with lots of "corporate message data" > stored on

Re: De-duping attachments

2010-09-15 Thread Simon Matter
> On Wed, Sep 15, 2010 at 09:15:13AM +0530, Shuvam Misra wrote: >> Dear Bron, >> >> > http://www.newegg.com/Product/Product.aspx?Item=N82E16822148413 >> > >> > 2TB - US $109. >> >> Don't want to nit-pick here, but the effective price we pay is about >> ten times this. > > Yeah, so? It's going down

Re: Using cvt_cyrusdb to convert quota database from skiplist back to quotalegacy.

2010-09-15 Thread Simon Matter
> Hello, > > > I am having trouble converting a quota skiplist db back to quotalegacy > format (I know... this is probably not the most common Cyrus operation :-) > > % cvt_cyrusdb /ssd/cyrs/imap/quotas.db skiplist /ssd/cyrs/imap/quota > quotalegacy > Converting from /ssd/cyrs/imap/quotas.db (skipl