Juergen, Thanks for the tip! I'll do some experimentation the next chance I get.
I agree that the root cause seems like a performance bug from my perspective. -Chris On Tue, Feb 5, 2019 at 4:27 AM Juergen Sauer <[email protected]> wrote: > Am 05.02.19 um 07:49 schrieb Chris Hartman: > > Greetings! > > > > I actually messaged about this several months(?) ago, though you > > articulated it better than I. > > > > I run a 2-node HA VM Cluster with KVM/pacemaker on top of DRBD 8.4 very > > comparable to your hardware.and have experienced similar symptoms during > > backup procedures. When it's really bad, one node will fence the other > > because the remote disk becomes unresponsive past the DRBD timeout > > threshold (auto calculated around 42 seconds). > > > > My only work around has been to keep all VMs on a single node at a time > and > > manually move all nodes periodically- this setup tolerates the I/O spike > > much better. However, we don't get the performance benefit of having both > > nodes active, not to mention the added administrative overhead. > > Hi Chris, > I found, that thotteling, limiting the I/O for virtual machine drives > helps: > > I.E. > # virsh blkdeviotune Virt-Name vda --total-iops-sec 1000 > --total-bytes-sec 52428800 > > See also: > > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-blockio-techniques > > But this is only a workaround, not the solution that drbd8 fails on > heavy I/O, due self generated Load escalation. > > mit freundlichen Grüßen > Jürgen Sauer > -- > Jürgen Sauer - automatiX GmbH, > +49-4209-4699, [email protected] > Geschäftsführer: Jürgen Sauer, > Gerichtstand: Amtsgericht Walsrode • HRB 120986 > Ust-Id: DE191468481 • St.Nr.: 36/211/08000 > GPG Public Key zur Signaturprüfung: > http://www.automatix.de/juergen_sauer_publickey.gpg >
_______________________________________________ drbd-user mailing list [email protected] http://lists.linbit.com/mailman/listinfo/drbd-user
