On Fri, Jun 08, 2018 at 11:35:45AM +0200, Kashyap Chamarthy wrote: > On Thu, Jun 07, 2018 at 01:07:48PM -0500, Matt Riedemann wrote: > > On 6/7/2018 12:56 PM, melanie witt wrote: > > > Recently, we've received interest about increasing the maximum number of > > > allowed volumes to attach to a single instance > 26. The limit of 26 is > > > because of a historical limitation in libvirt (if I remember correctly) > > > and is no longer limited at the libvirt level in the present day. So, > > > we're looking at providing a way to attach more than 26 volumes to a > > > single instance and we want your feedback. > > > > The 26 volumes thing is a libvirt driver restriction. > > The original limitation of 26 disks was because at that time there was > no 'virtio-scsi'. > > (With 'virtio-scsi', each of its controller allows upto 256 targets, and > each target can use any LUN (Logical Unit Number) from 0 to 16383 > (inclusive). Therefore, the maxium allowable disks on a single > 'virtio-scsi' controller is 256 * 16384 == 4194304.) Source[1].
Not totally true for Nova. Nova handles one virtio-scsi controller per guest and plug all the volumes in one target so in theory that would be 16384 LUN (only). But you made a good point the 26 volumes thing is not a libvirt driver restriction. For example the QEMU SCSI native implementation handles 256 disks. About the virtio-blk limitation I made the same finding but Tsuyoshi Nagata shared an interesting point saying that virtio-blk is not longer limited by the number of PCI slot available. That in recent kernel and QEMU version [0]. I could join what you are suggesting at the bottom and fix the limit to 256 disks. [0] https://review.openstack.org/#/c/567472/16/nova/virt/libvirt/blockinfo.py@162 > [...] > > > > Some ideas that have been discussed so far include: > > > > > > A) Selecting a new, higher maximum that still yields reasonable > > > performance on a single compute host (64 or 128, for example). Pros: > > > helps prevent the potential for poor performance on a compute host from > > > attaching too many volumes. Cons: doesn't let anyone opt-in to a higher > > > maximum if their environment can handle it. > > Option (A) can still be considered: We can limit it to 256 disks. Why? > > FWIW, I did some digging here: > > The upstream libguestfs project after some thorough testing, arrived at > a limit of 256 disks, and suggest the same for Nova. And if anyone > wants to increase that limit, the proposer should come up with a fully > worked through test plan. :-) (Try doing any meaningful I/O to so many > disks at once, and see how well that works out.) > > What more, the libguestfs upstream tests 256 disks, and even _that_ > fails sometimes: > > https://bugzilla.redhat.com/show_bug.cgi?id=1478201 -- "kernel runs > out of memory with 256 virtio-scsi disks" > > The above bug is fixed now in kernel-4.17.0-0.rc3.git1.2. (And also > required a corresponding fix in QEMU[2], which is available from version > v2.11.0 onwards.) > > [...] > > > [1] https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg02823.html > -- virtio-scsi limits > [2] https://git.qemu.org/?p=qemu.git;a=commit;h=5c0919d > > -- > /kashyap > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
