In following this discussion, I get the feeling that you and Richard are somewhat talking past each other. He asked you about the hardware you are currently running on, whereas you seem to be interested in a model for the impact of scrubbing on I/O throughput that you can apply to some not-yet-acquired hardware.
It should be clear by now that the model you are looking for does not exist given how new ZFS is, and Richard has been focusing his comments on your existing (home) configuration since that is what you provided specs for. Since you haven't provided specs for this larger system you may be purchasing in the future, I don't think anyone can give you specific guidance on what the I/O impact of scrubs on your configuration will be. Richard seems to be giving more design guidelines and hints, and just generally good to know information to keep in mind while designing your solution. Of course, he's been giving it in the context of your 11 disk wide RAIDZ2 and not the 200 TB monster you only described in the last e-mail. Stepping back, it may be worthwhile to examine the advice Richard has given, in the context of the larger configuration. First, you won't be using commodity hardware for your enterprise-class storage system, will you? Second, I would imagine that as a matter of practice, most people schedule their pools to scrub as far away from prime hours as possible. Maybe it's possible, and maybe it's not. The question to the larger community should be "who is running a 100+ TB pool and how have you configured your scrubs?" Or even "for those running 100+ TB pools, do your scrubs interfere with your production traffic/throughput? If so, how do you compensate for this?" Third, as for ZFS scrub prioritization, Richard answered your question about that. He said it is low priority and can be tuned lower. However, he was answering within the context of an 11 disk RAIDZ2 with slow disks His exact words were: "This could be tuned lower, but your storage is slow and *any* I/O activity will be noticed." If you had asked about a 200 TB enterprise-class pool, he may have had a different response. I don't know if ZFS will make different decisisons regarding I/O priority on commodity hardware as opposed to enterprise hardware, but I imagine it does *not*. If I'm mistaken, someone should correct me. Richard also said "In b133, the priority scheduler will work better than on older releases." That may not be an issue since you haven't acquired your hardware YET, but again, Richard didn't know that you were talking about a 200 TB behemoth because you never said that. Fourth, Richard mentioned a wide RAIDZ2 set. Hopefully, if nothing else, we've seen that designing larger ZFS storage systems with pools composed of smaller top level VDEVs works better, and preferably mirrored top level VDEVs in the case of lots of small, random reads. You didn't indicate the profile of the data to be stored on your system, so no one can realistically speak to that. I think the general guidance is sound. Multiple top level VDEVs, preferably mirrors. If you're creating RAIDZ2 top level VDEVs, then they should be smaller (narrower) in terms of the number of disks in the set. 11 would be too many, based on what I have seen and heard on this list cross referenced with the (little) information you have provided. RAIDZ2 would appear to require more CPU power that RAIDZ, based on the report you gave and thus may have less negative impact on the performance of your storage system. I'll cop to that. However, you never mentioned how your 200 TB behemoth system will be used, besides an off-hand remark about CIFS. Will it be serving CIFS? NFS? Raw ZVOLs over iSCSI? You never mentioned any of that. Asking about CIFS if you're not going to serve CIFS doesn't make much sense. That would appear to be another question for the ZFS gurus here -- WHY does RAIDZ2 cause so much negative performance impact on your CIFS service while RAIDZ does not? Your experience is that a scrub of a RAIDZ2 maxed CPU while a RAIDZ scrub did not, right? Fifth, the pool scrub should probably be as far away from peak usage times as possible. That may or may not be feasible, but I don't think anyone would disagree with that advice. Again, I know there are people running large pools who perform scrubs. It might be worthwhile to directly ask what these people have experienced in terms of scrub performance on RAIDZ vs. RAIDZ2, or in general. Finally, I would also note that Richard has been very responsive to your questions (in his own way) but you increasingly seem to be hostile and even disrespectful toward him. (I've noticed this in more then one of your e-mails; they sound progressively more self-centered and selfish. That's just my opinion.) If this is a community, that's not a helpful way to treat a senior member of the community, even if he's not answering the question you want answered. Keep in mind that asking the wrong questions is the leading cause of wrong answers, as a former boss of mine likes to say. Maybe you would have gotten responses you found more useful and less insulting had you phrased your questions in an different way? And no, Richard doesn't need me to defend him, especially since I don't know him from a can of paint. Your attacks (for lack of a better word) on him seem unwarranted and I can't stay silent about it any longer. Anyway, hopefully that helps in some way. And hopefully you'll get how you're appearing to others who are reading your words. Right now, in MY opinion alone, you look like a n00b who isn't respectful enough to deserve the help of anyone here. On Tue, Mar 16, 2010 at 07:35, Tonmaus <sequoiamo...@gmx.net> wrote: > Hi Richard, > > > > - scrubbing the same pool, configured as raidz1 > > didn't max out CPU which is no surprise (haha, slow > > storage...) the notable part is that it didn't slow > > down payload that much either. > > > > raidz creates more, smaller writes than a mirror or > > simple stripe. If the disks are slow, > > then the IOPS will be lower and the scrub takes > > longer, but the I/O scheduler can > > manage the queue better (disks are slower). > > This wasn't mirror vs. raidz but raidz1 vs. raidz2, whereas the latter > maxes out CPU and the former maxes out physical disc I/O. Concurrent payload > degradation isn't that extreme on raidz1 pools, as it seems. Hence, the CPU > theory that you still seem to be reluctant to follow. > > > > There are several > > bugs/RFEs along these lines, something like: > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bu > > g_id=6743992 > > Thanks to pointing at this. As it seems, it's a problem for a couple of > years already. Obviously, the opinion is being shared that this a management > problem, not a HW issue. > > As a Project Manager I will soon have to take a purchase decision for an > archival storage system (A/V media), and one of the options we are looking > into is SAMFS/QFS solution including tiers on disk with ZFS. I will have to > make up my mind if the pool sizes we are looking into (typically we will > need 150-200 TB) are really manageable under the current circumstances when > we think about including zfs scrub in the picture. From what I have learned > here it rather looks as if there will be an extra challenge, if not even a > problem for the system integrator. That's unfortunate. > > Regards, > > Tonmaus > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > -- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it's a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
