[CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
S 6.5 -> 7.2/Kafka 0.8 -> 0.9 upgrade on those clusters, and there has been no impact to their performance. We've been banging our heads against the wall for a few weeks now, really hoping someone from the community can point us in the right direction. Thanks, Kelly Lesperance ___

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
Thanks, Kelly On 2016-05-25, 1:21 PM, "centos-boun...@centos.org on behalf of m.r...@5-cent.us" wrote: >Kelly Lesperance wrote: >> I’ve posted this on the forums at >> https://www.centos.org/forums/viewtopic.php?f=47&t=57926&p=244614#p244614 >> - post

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
[merging] The HBA the drives are attached to has no configuration that I’m aware of. We would have had to accidentally change 23 of them ☺ Thanks, Kelly On 2016-05-25, 1:25 PM, "Kelly Lesperance" wrote: >They are: > >[root@r1k1 ~] # hdparm -I /dev/sda > >/dev/sda

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
gt; >On 25.05.2016 19:26, Kelly Lesperance wrote: >> [merging] >> >> The HBA the drives are attached to has no configuration that I’m aware of. >> We would have had to accidentally change 23 of them ☺ >> >> Thanks, >> >> Kelly >> >&g

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
Hdparm didn’t get far: [root@r1k1 ~] # hdparm -tT /dev/sda /dev/sda: Timing cached reads: Alarm clock [root@r1k1 ~] # On 2016-05-25, 2:44 PM, "Kelly Lesperance" wrote: >The HBA is an HP H220. > >We haven’t really benchmarked individual drives – all 12 drives are utiliz

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
) On 2016-05-25, 3:01 PM, "centos-boun...@centos.org on behalf of John R Pierce" wrote: >On 5/25/2016 11:44 AM, Kelly Lesperance wrote: >> The HBA is an HP H220. > > >OH.its a very good idea to verify the driver is at the same revision >level as the firmwa

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
sys] # cat module/mpt2sas/version 20.100.00.00 On 2016-05-25, 3:20 PM, "centos-boun...@centos.org on behalf of m.r...@5-cent.us" wrote: >John R Pierce wrote: >> On 5/25/2016 11:44 AM, Kelly Lesperance wrote: >>> The HBA is an HP H220. >> >> OH.its

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
Already done – they’re not being very helpful, as we don’t have a support contract, just standard warranty. On 2016-05-25, 4:27 PM, "centos-boun...@centos.org on behalf of m.r...@5-cent.us" wrote: >Kelly Lesperance wrote: >> LSI/Avago’s web pages don’t have any downloads

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-25 Thread Kelly Lesperance
I should rephrase that – some parts of HP are helping us, but the team I opened the case with isn’t being very helpful. On 2016-05-25, 4:29 PM, "Kelly Lesperance" wrote: >Already done – they’re not being very helpful, as we don’t have a support >contract, just standard warrant

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-26 Thread Kelly Lesperance
ntos.org on behalf of cpol...@surewest.net" wrote: >On 2016-05-25 19:13, Kelly Lesperance wrote: >> Hdparm didn’t get far: >> >> [root@r1k1 ~] # hdparm -tT /dev/sda >> >> /dev/sda: >> Timing cached reads: Alarm clock >> [root@r1k1 ~] # >

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-05-27 Thread Kelly Lesperance
2016 09:54 AM, Kelly Lesperance wrote: >> What we're seeing is that when the weekly raid-check script executes, >> performance nose dives, and I/O wait skyrockets. The raid check starts out >> fairly fast (2K/sec - the limit that's been set), but then

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-06-01 Thread Kelly Lesperance
off a raid check on one of our CentOS 6 hosts as well, the window seems to be 15 - 20K/Sec on that host). Kelly On 2016-05-27, 9:21 AM, "Kelly Lesperance" wrote: >All of our Kafka clusters are fairly write-heavy. The cluster in question is >our second-heaviest – w

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

2016-06-01 Thread Kelly Lesperance
Software RAID 10. Servers are HP DL380 Gen 8s, with 12x4 TB 7200 RPM drives. On 2016-06-01, 3:52 PM, "centos-boun...@centos.org on behalf of m.r...@5-cent.us" wrote: >Kelly Lesperance wrote: >> I did some additional testing - I stopped Kafka on the host, and kicked >&