Ciro Iriarte wrote: > 2010/6/19 Miles Fidelman<[email protected]>: > >> Hi Folks, >> >> I'm experiencing a very odd, daily, high-load situation - that seems to >> localize in my disk stack. I direct this to the xen-users, linux-raid >> and linux-ha lists as I expect there's a pretty high degree of >> experience on these lists with complicated disk driver stacks. >> >> <snip> >> >> All of which leads to two questions: >> - if it's a disk going bad, why does this manifest nightly, at roughly >> the same time, and effect only one DomU? >> - if it's something in the DomU, by what mechanism is that rippling all >> they way down to a component of a raid array, hidden below several >> layers of stuff that's supposed to isolate virtual volumes from hardware? >> >> The only thought that occurs to me is that perhaps there's a bad record >> or block on that one drive, that only gets exercised when on particular >> process runs. >> - is that a possibility? >> - if yes, why isn't drbd or md or something catching it and fixing it >> (or adding the block to the bad block table)? >> - any suggestions on diagnostic or drive rebuilding steps to take next? >> (includings that I can do before staying up until 2am tomorrow!) >> > Did you run a long S.M.A.R.T test?. RAID6 on software is not a good > idea, even worse with that hardware. Also, install collectl, it will > save disk, cpu and memory stats as Sar, but will also collect info on > all the processes for later analysis that way you can tell if it's a > particular user space process that triggers the high load. > First off - thanks for the suggestions!
I just ran smartctl -t long /dev/sdb -- and it's reporting nothing indicative of a bad disk. All the attributes well above the failure threshold. But.... for a short period during the test, I saw the same symptom that's occuring nightly - atop started reporting 98% disk busy, which rippled through to push iowait to 100% on my DomU. So clearly this is something disk related. But after a few minutes, everything dropped to normal levels. So this seems to suggest that either: - a) something very localized is wrong with the disk (e.g., a bad block), or, - b) something weird is going on going on at the software level (not sure what this would be), or, - c) both of the above - say, one of the drivers trying read/write onto a not-quite-completely bad block - and doing huge numbers of reads until it retrieves something The last comes to mind as a result of learning, the hard way, that a failing disk can manifest by slowing a server to a crawl, as the drivers keep retrying operations until they succeed, rather than declaring an error and dropping a drive from a RAID array. (Anybody know if this is tunable - I've yet to find anything about this around the net.) Re. your other comments: - beyond the complexity and load, can you say a little more about why "RAID6 on software is not a good idea, even worse with that hardware." - top, atop, sar, etc. haven't been particularly helpful - thanks for your pointer to collectl, starting to play with it now, as well as collectd - though it's starting to look like the problem is either hardware or software below userspace, so not clear what's going to help, other than maybe turning on /proc/sys/vm/block_dump and watching the data go by. > This deserves a deeper analysis with collected data, but if you are > sure the drive is bad, replace it and check if it still happens (it > will hit you back when adding the new disk again though) > > I'm just more confused at this point. Any other suggestions as to what to look at, and how, would be very much appreciated. I'm also wondering if it would make sense to drop the disk from the array, doing a complete diagnostic and reformat, then bringing it backup. Thoughts? Suggestions as to procedure to minimize impact on the the array and rebuild (yes I know RAID6 degrades below 4 disks - not quite sure of the implications though for running with 3 drives for a few hours). Thanks again, Miles > -- In theory, there is no difference between theory and practice. In<fnord> practice, there is. .... Yogi Berra _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
