[Dovecot] Unseen messages question
Hi list, this question is related to the IMAP protocol itself, not really to Dovecot. I'm trying to understand what is the more efficient way to maintain the number of unseen messages of the currently selected mailbox. RFC3501 says a client must not issue a STATUS command to the selected mailbox and that information sent by a SELECT is enough. My current idea follows these steps : * Issue a STATUS before the mailbox is selected => I know how many unseen messages it contains * SELECT the mailbox => I got the eventual first unseen message in this mailbox but I don't understand how this info can be useful * Maintain the unseen counter (on client side) according to what the user do * Send a NOOP command every X minutes and look at the RECENT response to see if there are new messages I think it works pretty well when the mailbox is opened only once. Let's imagine this mailbox is opened twice, by different clients. If one client marks a message as \Seen, how can the second client know about this change? Thanks for your help, Antoine Nguyen http://modoboa.org/
Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?
On 14/04/2012 04:48, Stan Hoeppner wrote: On 4/13/2012 10:31 AM, Ed W wrote: You mean those "answers" like: "you need to read 'those' articles again" Referring to some unknown and hard to find previous emails is not the same as answering? No, referring to this: On 4/12/2012 5:58 AM, Ed W wrote: The claim by ZFS/BTRFS authors and others is that data silently "bit rots" on it's own. Is it not a correct assumption that you read this in articles? If you read this in books, scrolls, or chiseled tablets, my apologies for assuming it was articles. WHAT?!! The original context was that you wanted me to learn some very specific thing that you accused me of misunderstanding, and then it turns out that the thing I'm supposed to learn comes from re-reading every email, every blog post, every video, every slashdot post, every wiki, every ... that mentions ZFS's reason for including end to end checksumming?!! Please stop wasting our time and get specific You have taken my email which contained a specific question, been asked of you multiple times now and yet you insist on only answering irrelevant details with a pointed and personal dig on each answer. The rudeness is unnecessary, and your evasiveness of answers does not fill me with confidence that you actually know the answer... For the benefit of anyone reading this via email archives or whatever, I think the conclusion we have reached is that: modern systems are now a) a complex sum of pieces, any of which can cause an error to be injected, b) the level of error correction which was originally specified as being sufficient is now starting to be reached in real systems, possibly even consumer systems. There is no "solution", however, the first step is to enhance "detection". Various solutions have been proposed, all increase cost, computation or have some disadvantage - however, one of the more promising detection mechanisms is an end to end checksum, which will then have the effect of augmenting ALL the steps in the chain, not just one specific step. As of today, only a few filesystems offer this, roll on more adopting it Regards Ed W
Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?
On Fri, Apr 13, 2012 at 07:33:19AM -0500, Stan Hoeppner wrote: > > > > What I meant wasn't the drive throwing uncorrectable read errors but > > the drives are returning different data that each think is correct or > > both may have sent the correct data but one of the set got corrupted > > on the fly. After reading the articles posted, maybe the correct term > > would be the controller receiving silently corrupted data, say due to > > bad cable on one. > > This simply can't happen. What articles are you referring to? If the > author is stating what you say above, he simply doesn't know what he's > talking about. It has happened to me, with RAID5 not RAID1. It was a firmware bug in the raid controller that caused the RAID array to go silently corrupted. The HW reported everything green -- but the filesystem was reporting lots of strange errors.. This LUN was part of a larger filesystem striped over multiple LUNs, so parts of the fs was OK, while other parts was corrupt. It was this bug: http://delivery04.dhe.ibm.com/sar/CMA/SDA/02igj/7/ibm_fw1_ds4kfc_07605200_anyos_anycpu.chg - Fix 432525 - CR139339 Data corruption found on drive after reconstruct from GHSP (Global Hot Spare) > In closing, I'll simply say this: If hardware, whether a mobo-down SATA > chip, or a $100K SGI SAN RAID controller, allowed silent data corruption > or transmission to occur, there would be no storage industry, and we'll > all still be using pen and paper. The questions you're asking were > solved by hardware and software engineers decades ago. You're fretting > and asking about things that were solved decades ago. Look at the plans are for your favorite fs: http://www.youtube.com/watch?v=FegjLbCnoBw They're planning on doing metadata checksumming to be sure they don't receive corrupted metadata from the backend storage, and say that data validation is a storage subsystem *or* application problem. Hardly a solved problem.. -jf
Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?
On 14/04/2012 04:31, Stan Hoeppner wrote: On 4/13/2012 10:31 AM, Ed W wrote: On 13/04/2012 13:33, Stan Hoeppner wrote: In closing, I'll simply say this: If hardware, whether a mobo-down SATA chip, or a $100K SGI SAN RAID controller, allowed silent data corruption or transmission to occur, there would be no storage industry, and we'll all still be using pen and paper. The questions you're asking were solved by hardware and software engineers decades ago. You're fretting and asking about things that were solved decades ago. So why are so many people getting excited about it now? "So many"? I know of one person "getting excited" about it. You love being vague don't you? Go on, I'll bite again, do you mean yourself? :-) Data densities and overall storage sizes and complexity at the top end of the spectrum are increasing at a faster rate than the consistency/validation mechanisms. That's the entire point of the various academic studies on the issue. Again, you love being vague. By your dismissive "academic studies" phrase, do you mean studies done on a major industrial player, ie NetApp in this case? Or do you mean that it's rubbish because they asked someone with some background in statistics to do the work, rather than asking someone sitting nearby in the office to do it? I don't think the researcher broke into NetApp to do this research, so we have to conclude that the industrial partner was onboard. NetApp seem to do a bunch of engineering of their own (got enough patents..) that I think we can safely assume they very much do their own research on this and it's not just "academic"... I doubt they publish all their own internal research, be thankful you got to see some of the results this way... Note that the one study required a sample set of 1.5 million disk drives. If the phenomenon were a regular occurrence as you would have everyone here believe, they could have used a much smaller sample set. Sigh... You could criticise the study if it had a small number of drives as being under-representive and now you criticise a large study for having too many observations... You cannot have "too many" observations when measuring a small and unpredictable phenomena... Where does it say that they could NOT have reproduced this study with just 10 drives? If you have 1.5 million available, why not use all the results?? Ed, this is an academic exercise. Academia leads industry. Almost always has. Academia blows the whistle and waves hands, prompting industry to take action. Sigh... We are back to the start of the email thread again... Gosh you seem to love arguing and muddying the water for zero reason but to have the last word? It's *trivial* to do a google search and hit *lots* of reports of corruptions in various parts of the system, from corrupting drivers, to hardware which writes incorrectly, to operating system flaws. I just found a bunch more in the Redhat database today while looking for something else. You yourself are very vocal on avoiding certain brands of HD controller which have been rumoured to cause corrupted data... (and thankyou for revealing that kind of thing - it's very helpful) Don't veer off at a tangent now: The *original* email this has spawned is about a VERY specific point. RAID1 appears to offer less protection against a class of error conditions than does RAID6. Nothing more, nothing less. Don't veer off and talk about the minutiae of testing studies at universities, this is a straightforward claim that you have been jumping around and avoiding answering with claims of needing to educate me on SCSI protocols and other fatuous responses. Nor deviate and discuss that RAID6 is inappropriate for many situations - we all get that... There is nothing normal users need to do to address this problem. ...except sit tight and hope they don't loose anything important! :-) Having the prestigious degree that you do, you should already understand the relationship between academic research and industry, and the considerable lead times involved. I'm guessing you haven't attended higher education then? You are confusing graduate and post-graduate systems... Byee Ed W
Re: [Dovecot] [OT] Outlook identities
On Fri, 13 Apr 2012 12:13:30 -0400 Michael Orlitzky articulated: > Exchange... the cure is worse than the disease! This isn't looking > good -- I guess I'll continue to do what I have been: telling people > to switch off of Outlook if they want their mail client to not suck. First of all, there are no existing RFC's that require any MUA to meet the requirements that you desire. So please, stop your wining and crying. It is embarrassing. Second, there are avenues available that can make Outlook behave in a fashion that should be acceptable to you. If you choose not to pursue them, then that is you business. I have had to endure hours of tedious nonsense to get a simple sound card to work under a *.nix environment when I could have simply plugged it into a machine running Microsoft Windows and had it working immediately. Your "the cure is worse than the disease" is just self-serving bull-shit. Outlook + MS Exchange offers features that no other MUA presently comes close to being able to duplicate in an office environment. If these don't fit your needs, then please find an MUA that does. No one is holding a gun to your head. However, your desire to force others to abandon something that works fine for them to simple suit your narrow view of what an MUA should or should not do stinks of fascism. I use Outlook at work and claws-mail at home. Each one fits perfectly into the environment I have chosen to use it in. By the way, after examining your original post, I cannot find a single thing that the proper use of filtering rules and plugins cannot easily accomplish. Instead of your customers using a different MUA, they should consider changing to a new service provider. -- Jerry ♔ Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __
Re: [Dovecot] Dovecot 2.1 with custom OpenSSL fails to build
Timo Sirainen wrote: >> But two libraries are not quite okay. They don't find their SSL libs: >> >> libdovecot-lda.so >> libdovecot-storage.so > > Maybe this fixes it? > > http://hg.dovecot.org/dovecot-2.1/rev/8b91367bc3e1 Works perfectly! Great, now all components find their libraries by themselves. Thanks a lot for fixing this issue which seemed quite complicated. Very good, thank you ... Andreas
[Dovecot] Compressed mbox - patch
Some time ago I complained about very slow access to compressed mboxes.
Unfortunately it looks like that it is very little interest in it, so I
have to investigate some things by myself.
Firstly: some rationale.
Why do I prefer use mbox/maildir over mdbox. Short answer "bus factor"
for support mdbox (not only dovecot)
Longer answer: if something goes wrong withm maildir/mbox i can use
other tools (mutt, or formail or even text editor) and with mdbox ...
I am not ISP, I use dovecot as a "gateway" to my (rather huge) mail
archive. Most of these mails are rather valuable for me, so I prefer use
something "well-known-and-tested".
(I can't do like most ISP's do: write in "Terms of Service" that mail
can be lost or damaged and we give no warranty :) )
So then:
Below my patch.
It contains 2 changes:
1. when buffer is compressed, we try to save last marked offset.
2. Increase temporary buffer for decompression.
without these changes 1.5 GB of bzip compressed mbox with ~20K messages
can't be open in 1.5 day
After applying 1. change it can be open in ~1.5 h
With both changes it was a few minutes.
Maybe it is a good idea to add config parameter to specify size of
decompress buffer?
Patch is against v2.0.18
diff -x '*.o' -x '*.lo' -x '*.la' -u -r ../dovecot-2.0.18/src/lib/istream.c ./src/lib/istream.c
--- ../dovecot-2.0.18/src/lib/istream.c 2011-12-13 12:38:27.0 +0100
+++ ./src/lib/istream.c 2012-04-14 10:27:23.790724625 +0200
@@ -452,6 +452,22 @@
stream->pos -= stream->skip;
stream->skip = 0;
+
+}
+
+void i_stream_compress1(struct istream_private *stream, size_t bytes )
+{
+
+size_t lskip ;
+
+ lskip = (stream->skip > bytes ? bytes : stream->skip );
+
+ memmove(stream->w_buffer, stream->w_buffer + lskip ,
+ stream->pos - lskip);
+ stream->pos -= lskip;
+ stream->skip -= lskip;
+
+
}
void i_stream_grow_buffer(struct istream_private *stream, size_t bytes)
diff -x '*.o' -x '*.lo' -x '*.la' -u -r ../dovecot-2.0.18/src/lib/istream-internal.h ./src/lib/istream-internal.h
--- ../dovecot-2.0.18/src/lib/istream-internal.h 2011-12-13 12:38:27.0 +0100
+++ ./src/lib/istream-internal.h 2012-04-13 00:06:27.700298378 +0200
@@ -51,6 +51,7 @@
i_stream_create(struct istream_private *stream, struct istream *parent, int fd);
void i_stream_compress(struct istream_private *stream);
+void i_stream_compress1(struct istream_private *stream, size_t bytes );
void i_stream_grow_buffer(struct istream_private *stream, size_t bytes);
bool i_stream_get_buffer_space(struct istream_private *stream,
size_t wanted_size, size_t *size_r);
diff -x '*.o' -x '*.lo' -x '*.la' -u -r ../dovecot-2.0.18/src/plugins/zlib/istream-bzlib.c ./src/plugins/zlib/istream-bzlib.c
--- ../dovecot-2.0.18/src/plugins/zlib/istream-bzlib.c 2012-02-09 18:32:48.0 +0100
+++ ./src/plugins/zlib/istream-bzlib.c 2012-04-14 10:35:04.349800777 +0200
@@ -9,12 +9,14 @@
#include
#define CHUNK_SIZE (1024*64)
+#define BUFF_SIZE (1024*1024*16)
struct bzlib_istream {
struct istream_private istream;
-
+
bz_stream zs;
uoff_t eof_offset, stream_size;
+ uoff_t marked_offset;
size_t prev_size, high_pos;
struct stat last_parent_statbuf;
@@ -48,7 +50,6 @@
uoff_t high_offset;
size_t size;
int ret;
-
high_offset = stream->istream.v_offset + (stream->pos - stream->skip);
if (zstream->eof_offset == high_offset) {
i_assert(zstream->high_pos == 0 ||
@@ -87,7 +88,14 @@
if (stream->pos == stream->buffer_size) {
if (stream->skip > 0) {
/* lose our buffer cache */
-i_stream_compress(stream);
+/* try to save our buffer cache as much as possible */
+
+if (zstream->marked && (stream-> skip - (stream->istream.v_offset - zstream->marked_offset)) >0 ){
+
+ i_stream_compress1(stream, stream-> skip - (stream->istream.v_offset - zstream->marked_offset));
+} else {
+ i_stream_compress(stream);
+}
}
if (stream->pos == stream->buffer_size)
@@ -215,8 +223,12 @@
struct bzlib_istream *zstream = (struct bzlib_istream *) stream;
uoff_t start_offset = stream->istream.v_offset - stream->skip;
+ if (mark)
+ zstream->marked_offset = v_offset;
if (v_offset < start_offset) {
/* have to seek backwards */
+
+
i_stream_bzlib_reset(zstream);
start_offset = 0;
} else if (zstream->high_pos != 0) {
@@ -243,6 +255,7 @@
}
i_stream_skip(&stream->istream, avail);
+
} while (i_stream_read(&stream->istream) >= 0);
if (stream->istream.v_offset != v_offset) {
@@ -260,8 +273,11 @@
}
}
- if (mark)
+ if (mark){
zstream->marked = TRUE;
+ zstream->marked_offset = v_offset;
+ }
+
}
static const struct stat *
@@ -329,7 +345,9 @@
i_stream_bzlib_init(zstream);
zstream->istream.iostream.close = i_stream_bzlib_close;
- zstream->istream.max_buffer_size = input->real_stream->max_buffer_size;
+ // zstream->istream.max_buffer_size = (input->real_stream->max_buffer_size);
+ zstream->istream.max_buffer_size = BUFF_SIZE;
+
zstream->istream.re
[Dovecot] Sieve pipe extension - can it retur something?
I have a question about sieve pipe: can it return something to further processing? For example in procmail I can do: --8<---cut here---start->8--- :0 VIRUS=`$CLAMSCAN --mbox --disable-summary --stdout -` --8<---cut here---end--->8--- and then test VIRUS variable. Maybe I missing something, when read http://hg.rename-it.nl/pigeonhole-0.2-sieve-pipe/raw-file/tip/doc/rfc/spec-bosch-sieve-pipe.txt KJ -- http://sporothrix.wordpress.com/2011/01/16/usa-sie-krztusza-kto-nastepny/ Gloffing is a state of mine.
[Dovecot] Dovecot 2.1.4 and client certificates
Version: 2.1.4 OS: Gentoo stable/amd64 OpenSSL version: 1.0.0h I'm having a slight problem with the client certificates in Dovecot 2.1.4. I've set-up the client certificate verification/authentication, and it seems that Dovecot is choking on the trustchain with CRL's that I'm providing to it (attached to this mail). When I enable the client authentication using certificates, and pick the certificate from my client (I've also tried it out with gnutls-cli as well), I get the following errors in Dovecot's log: imap-login: Info: Invalid certificate: Different CRL scope: /CN=Example Root CA/O=Example Inc./C=RS As per the wiki2 configuration page, I've set up the truststore in the following order (everything PEM-encoded): Example Person CA Certificate Example Person CA CRL Example Root CA Certificate Example Root CA CRL Person CA is the one issuing the end-entity certificates, of course. I'm also attaching the certificate I've used for testing. On additional note, the imap-login process also got stuck writing out the error message to the log file, refusing to die when receiving the SIGTERM (had to send SIGKILL). A similar set-up used to work under Dovecot in Debian Squeeze (version 1.2.15). The same file copied over to Dovecot 2.1.4's configuration won't work. I've compiled Dovecot by hand, and I'm not running it in any kind of chroot (this is a developer set-up so I could add support for rfc822Name username extraction I mentioned a week or so ago without messing around as root). Best regards -- Branko Majic Jabber: bra...@majic.rs Please use only Free formats when sending attachments to me. Бранко Мајић Џабер: bra...@majic.rs Молим вас да додатке шаљете искључиво у слободним форматима. trustchain.pem Description: application/x509-ca-cert branko_majic.crt Description: application/x509-ca-cert signature.asc Description: PGP signature
[Dovecot] LMTP auth problem
hey all, im getting the following error:
Apr 14 14:29:44 lmtpdirector1 dovecot: auth: Error: passdb(scorpio,127.0.0.1):
Auth client doesn't have permissions to do a PASS lookup:
/var/run/dovecot/auth-userdb mode=0666, but not owned by UID 112(dovecot)
Apr 14 14:29:44 lmtpdirector1 dovecot: lmtp(18298): Error: user scorpio: Auth
PASS lookup failed
My config. Director servers running both imap and lmtp with a matching set of
real servers accepting imap/lmtp. Imap is working fine, and has been working
fine for a while. Im trying to add lmtp to the director, but i cant seem to get
that working. We're passing passdb on to the real servers. How does this work
with lmtp?
protocols = imap lmtp
protocol lmtp {
auth_socket_path = director-userdb
}
lmtp_proxy = yes
# passdb check on real servers
passdb {
driver = static
args = proxy=y nopassword=y
}
Cor
Re: [Dovecot] LMTP auth problem
Of course the moment I post I seem to have figured it out..
service auth {
unix_listener auth-userdb {
mode = 0777
}
}
Is this safe if your servers are secure?
Cor
Re: [Dovecot] LMTP auth problem
Am 14.04.2012 um 18:24 schrieb Cor Bosman:
> Apr 14 14:29:44 lmtpdirector1 dovecot: auth: Error:
> passdb(scorpio,127.0.0.1): Auth client doesn't have permissions to do a PASS
> lookup: /var/run/dovecot/auth-userdb mode=0666, but not owned by UID
> 112(dovecot)
> Apr 14 14:29:44 lmtpdirector1 dovecot: lmtp(18298): Error: user scorpio: Auth
> PASS lookup failed
I'd just try 'user = dovecot' rather than making it wide open because that's
what the log basically says.
$ doveconf -d | grep 'unix_listener auth-userdb' -A 4
unix_listener auth-userdb {
group =
mode = 0666
user =
}
Regards
Thomas
signature.asc
Description: Message signed with OpenPGP using GPGMail
Re: [Dovecot] LMTP auth problem
> > Apr 14 14:29:44 lmtpdirector1 dovecot: auth: Error:
> > passdb(scorpio,127.0.0.1): Auth client doesn't have permissions to do a
> > PASS lookup: /var/run/dovecot/auth-userdb mode=0666, but not owned by UID
> > 112(dovecot)
> > Apr 14 14:29:44 lmtpdirector1 dovecot: lmtp(18298): Error: user scorpio:
> > Auth PASS lookup failed
>
> I'd just try 'user = dovecot' rather than making it wide open because that's
> what the log basically says.
>
> $ doveconf -d | grep 'unix_listener auth-userdb' -A 4
> unix_listener auth-userdb {
> group =
> mode = 0666
> user =
> }
>
My config was the same as yours. That didnt work for me. But if I add
user = dovecot
mode = 0666
That does work. Of course, the difference between 777 and 666 is
minimal. I think 666 is handled as a special case in the code?
Cor
Re: [Dovecot] Sieve pipe extension - can it retur something?
Op 4/14/2012 2:13 PM, Kamil Jońca schreef: I have a question about sieve pipe: can it return something to further processing? For example in procmail I can do: --8<---cut here---start->8--- :0 VIRUS=`$CLAMSCAN --mbox --disable-summary --stdout -` --8<---cut here---end--->8--- and then test VIRUS variable. Maybe I missing something, when read http://hg.rename-it.nl/pigeonhole-0.2-sieve-pipe/raw-file/tip/doc/rfc/spec-bosch-sieve-pipe.txt For Pigeonhole 0.3/Dovecot 2.1 there is a new plugin called ExtPrograms. Apart from the 'pipe' extension it adds the 'execute' extension that should match just what you want: http://hg.rename-it.nl/pigeonhole-0.3-sieve-extprograms/raw-file/d4683490a878/doc/rfc/spec-bosch-sieve-extprograms.txt Regards, Stephan.
Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?
On 4/14/2012 5:04 AM, Jan-Frode Myklebust wrote: > On Fri, Apr 13, 2012 at 07:33:19AM -0500, Stan Hoeppner wrote: >>> >>> What I meant wasn't the drive throwing uncorrectable read errors but >>> the drives are returning different data that each think is correct or >>> both may have sent the correct data but one of the set got corrupted >>> on the fly. After reading the articles posted, maybe the correct term >>> would be the controller receiving silently corrupted data, say due to >>> bad cable on one. >> >> This simply can't happen. What articles are you referring to? If the >> author is stating what you say above, he simply doesn't know what he's >> talking about. > > It has happened to me, with RAID5 not RAID1. It was a firmware bug > in the raid controller that caused the RAID array to go silently > corrupted. The HW reported everything green -- but the filesystem was > reporting lots of strange errors.. This LUN was part of a larger > filesystem striped over multiple LUNs, so parts of the fs was OK, while > other parts was corrupt. > > It was this bug: > > > http://delivery04.dhe.ibm.com/sar/CMA/SDA/02igj/7/ibm_fw1_ds4kfc_07605200_anyos_anycpu.chg >- Fix 432525 - CR139339 Data corruption found on drive after > reconstruct from GHSP (Global Hot Spare) Note my comments were specific to the RAID1 case, or a concatenated set of RAID1 devices. And note the discussion was framed around silent corruption in the absence of bugs and hardware failure, or should I say, where no bugs or hardware failures can be identified. > > >> In closing, I'll simply say this: If hardware, whether a mobo-down SATA >> chip, or a $100K SGI SAN RAID controller, allowed silent data corruption >> or transmission to occur, there would be no storage industry, and we'll >> all still be using pen and paper. The questions you're asking were >> solved by hardware and software engineers decades ago. You're fretting >> and asking about things that were solved decades ago. > > Look at the plans are for your favorite fs: > > http://www.youtube.com/watch?v=FegjLbCnoBw > > They're planning on doing metadata checksumming to be sure they don't > receive corrupted metadata from the backend storage, and say that data > validation is a storage subsystem *or* application problem. You can't made sure you don't receive corrupted data. You take steps to mitigate the negative effects of it if and when it happens. The XFS devs are planning this for the future. If the problem was here now, this work would have already been done. > Hardly a solved problem.. It has been up to this point. The issue going forward is that current devices don't employ sufficient consistency checking to meet future needs. And the disk drive makers apparently don't want to consume the additional bits required to properly do this in the drives. If they'd dedicate far more bits to ECC we may not have this issue. But since it appears this isn't going to change, kernel, filesystem and application developers are taking steps to mitigate it. Again, this "silent corruption" issue as described in the various academic papers is a future problem for most, not a current problem. It's only a current problem for those are the bleeding edge of large scale storage. Note that firmware bugs in individual products aren't part of this issue. Those will be with us forever in various products because humans make mistakes. No amount of filesystem or application code can mitigate those. The solution to that is standard best practices: snapshots, backups, or even mirroring all your storage across different vendor hardware. -- Stan
Re: [Dovecot] Better to use a single large storage server or multiple smaller for mdbox?
On 4/14/2012 5:00 AM, Ed W wrote: > On 14/04/2012 04:48, Stan Hoeppner wrote: >> On 4/13/2012 10:31 AM, Ed W wrote: >> >>> You mean those "answers" like: >>> "you need to read 'those' articles again" >>> >>> Referring to some unknown and hard to find previous emails is not the >>> same as answering? >> No, referring to this: >> >> On 4/12/2012 5:58 AM, Ed W wrote: >> >>> The claim by ZFS/BTRFS authors and others is that data silently "bit >>> rots" on it's own. >> Is it not a correct assumption that you read this in articles? If you >> read this in books, scrolls, or chiseled tablets, my apologies for >> assuming it was articles. >> > > WHAT?!! The original context was that you wanted me to learn some very > specific thing that you accused me of misunderstanding, and then it > turns out that the thing I'm supposed to learn comes from re-reading > every email, every blog post, every video, every slashdot post, every > wiki, every ... that mentions ZFS's reason for including end to end > checksumming?!! No, the original context was your town crier statement that the sky is falling due to silent data corruption. I pointed out that this is not the case, currently, that most wouldn't see this until quite a few years down the road. I provided facts to back my statement, which you didn't seem to grasp or comprehend. I pointed this out and your top popped with a cloud of steam. > Please stop wasting our time and get specific Whose time am I wasting Ed? You're the primary person one on this list who wastes everyone's time with these drawn out threads, usually unrelated to Dovecot. I have been plenty specific. The problem is you lack the knowledge and understanding of hardware communication. You're upset because I'm not pointing out the knowledge you seem to lack? Is that not a waste of everyone's time? Is that not be even "more insulting"? Causing even more excited/heated emails from you? > You have taken my email which contained a specific question, been asked > of you multiple times now and yet you insist on only answering > irrelevant details with a pointed and personal dig on each answer. The > rudeness is unnecessary, and your evasiveness of answers does not fill > me with confidence that you actually know the answer... Ed, I have not been rude. I've been attempting to prevent you dragging us into the mud, which you've done, as you often do. How specific would you like me to get? This is what you seem to be missing: Drives perform per sector CRC before transmitting data to the HBA. ATA, SATA, SCSI, SAS, fiber channel devices and HBAs all perform CRC on wire data. The PCI/PCI-X/PCIe buses/channels and Southbridge all perform CRC on wire data. HyperTransport, and Intel's proprietary links also perform CRC on wire transmissions. Server memory is protected by ECC, some by ChipKill which can tolerate double bit errors. With today's systems and storage densities, with error correcting code on all data paths within the system, and on the drives themselves, "silent data corruption" is not an issue--in absence of defective hardware or a bug, which are not relevant to the discussion. > For the benefit of anyone reading this via email archives or whatever, I > think the conclusion we have reached is that: modern systems are now a) > a complex sum of pieces, any of which can cause an error to be injected, Errors occur all the time. And they're corrected nearly all of the time, on modern complex systems. Silent errors do not occur frequently, usually not at all, on most modern systems. > b) the level of error correction which was originally specified as being > sufficient is now starting to be reached in real systems, FSVO 'real systems'. The few occurrences of "silent data corruption" I'm aware of have been documented in academic papers published by researches working at taxpayer funded institutions. In the case of CERN, the problem was a firmware bug in the Western Digital drives that caused an issue with the 3Ware controllers. This kind of thing happens when using COTS DIY hardware in the absence of proper load validation testing. So this case doesn't really fit the Henny-penny silent data corruption scenario as a firmware bug caused it. One that should have been caught and corrected during testing. In the other cases I'm aware of, all were HPC systems which generated SDC under extended high loads, and these SDCs nearly all occurred somewhere other than the storage systems--CPUs, RAM, interconnect, etc. HPC apps tend to run the CPUs, interconnects, storage, etc, at full bandwidth for hours at a time, across tens of thousands of nodes, so the probability of SDC is much higher simply due to scale. > possibly even > consumer systems. Possibly? If you're going to post pure conjecture why not say "possibly even iPhones or Androids"? There's no data to back either claim. Stick to the facts. > There is no "solution", however, the first step is to > enhance "detection". Va
