On Tue, Jan 19, 2016 at 04:52:38PM +1000, Piers Rowan wrote: > >1. what kind of disks are these? > > HP Hardware Array with default LVM on it. This was just to take live > snapshots of MySQL to be able to restart replication without issue.
so, how are the HP Array controller and disks configured? JBOD? one big RAID-5 arrayŝ a RAID-1 array for mysql and the rest as RAID5 for the OS and /home etc? if it's one big RAID-5 array, with LVM on top of that then there is no separation of disk I/O between mysql and mail - they're all using the same drives all the time. performance-wise, this is about the worst possible way to set it up....good for maximising storage capacity, but bad for disk I/O performance. BTW, what kind of server is it? brand/model. and specs (CPU, RÆM, etc). I guess it's some model of HP server. and what's the exact model of the HP Array Controller? 'lspci | grep HP' or 'lspci | grep -i array' should show it. HP provide a command-line tool for linux called hpacucli - if you haven't installed it already, you should do so ASAP. It will let you query and configure your HP Array from linux. You'll have to search on the HP web site for download and install instructions - or maybe it came on a CD with your server. Rather than wade through the inevitably over-verbose official corporate documentation (read that too, but as a reference later), you can find quick guides on how to use its basic functions with google, e.g.: http://koo.fi/blog/2008/06/08/hp-array-configuration-and-diagnostic-utilities-on-linux/ http://www.thegeekstuff.com/2014/07/hpacucli-examples/ BTW, hpacucli is also useful for server monitoring purposes - tools like nagios have plugins to use it for array status monitoring and alerting. if you're wondering why Russell and I are asking all these questions it's because you haven't provided anywhere near enough details to answer your performance question. > > 2. is /home RAID-5? i'm guessing it is since RAID-10 with 3 drives > > and a hot-spare doesn't make any sense. RAID-5 can be dreadfully > > slow, especially on random writes. > > RAID 5 + hot spare > > what kind of virtual disks are you using for the VMś? cow2 image > > files? raw or lvm partitions? partitions are much faster than qcow2 > > files. > > Not sure - you can check the VM's config to see if it uses a file image (e.g. if using libvirt, maybe something like /var/lib/libvert/images/mailserver.img) or an LVM partition (e.g. /dev/mapper/vg-mail or /dev/dm-?) what kind of virtualisation are you using? KVM? Xen? with libvirt? if you're using libvirt, you can use 'virsh dumpxml' and sed to query the vm's disk config: virsh dumpxml $vmname | sed -n -e '/<disk/,/<\/disk/ p' if you don't know the VM's name you can get it with 'virsh list'. what does 'ls -l /dev/mapper' show? and that list_disks one-liner i asked for yesterday? and 'lvdisplay'? > can't break up the HP array to give it a dedicated disk now tho. Too > much risk of downtime. do you have enough drive bays to add an SSD or pair of SSDs? if so, you can begin to migrate from your current setup to an improved performance setup like i described in my initial reply. you can even do it without spare drive bays if you have a spare sata port or two, just connect the drives to the ports and power and hang them loosely in the case while you're transferring the files the transfer can be done with rsync - a good method which minimises downtime is to run rsync while the mail server is still running normally to transfer the files to the new disk(s). you can run this rsync operation repeatedly as often as you like to keep it in sync (i would advise putting the exact rsync command with all its options in a shell script so you can just run the script without having to remember/retype the options every time). when you're ready to cut over to the new setup, stop the mail daemons and run a final rsync. then unmount and remount the new drives into their correct location and start up the mail daemons again. the bulk of the transfer is done while the service is still up and running, and if you plan each step/command in advance (i.e. give yourself a step-by-step "script" to follow on the command line), downtime will be not much more than the minimum time needed to rsync any changes between the last rsync run and the service being stopped. and you can do the same later, with your mysql disks. > >>iostat -x 10 > >>Linux 2.6.32-573.7.1.el6.x86_64 19/01/16 _x86_64_ (12 CPU) just noticed this - that's quite an old kernel, what version of Centos are you running? > >>avg-cpu: %user %nice %system %iowait %steal %idle > >> 5.27 0.00 1.74 3.98 0.00 89.01 > >> > >>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz > >>avgqu-sz await svctm %util > >>sda 144.40 398.34 92.54 95.71 5790.73 4493.15 54.63 > >>0.32 1.71 1.38 25.92 > > > so you're running a backup at the moment? is that when the slowdown > > occurs, or does it happen any old time? > > No it was used for a project a couple of years ago and never unplugged there's a lot of read and write activity showing on it, so something's using it. > >>dm-5 0.00 0.00 73.20 404.46 4478.78 3774.52 17.28 > >>0.24 0.27 0.46 21.75 > > > it's odd that most of the I/O is on just one of these /home drives. > > > I guessed that was how the RAID card presented itself to the OS nope. lvm doesn't work like that. each of those dm-? drives is a separate LVM volume. but there's no point in guessing how your disks are set up, you need to provide some more details, especially the hp array setup and the output of lvdisplay. craig -- craig sanders <[email protected]> BOFH excuse #250: Program load too heavy for processor to lift. _______________________________________________ luv-main mailing list [email protected] http://lists.luv.asn.au/listinfo/luv-main
