On 2014-10-30 16:48, Vadim Rozenfeld wrote: > On Thu, 2014-10-30 at 14:54 +0800, Wangting (Kathy) wrote: >>> On Tue, 2014-02-18 at 13:11 -0800, Nicholas A. Bellinger wrote: >>>> On Tue, 2014-02-18 at 13:00 -0800, Nicholas A. Bellinger wrote: >>>>> On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: >>>>> >>>>> <SNIP> >>>>> >>>>>>>>> Hi Yan, >>>>>>>>> >>>>>>>>> So recently I've been doing some KVM guest performance comparisons >>>>>>>>> between the scsi-mq prototype using virtio-scsi + vhost-scsi, and >>>>>>>>> Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + >>>>>>>>> vhost-scsi using PCIe flash backend devices. >>>>>>>>> >>>>>>>>> I've noticed that small block random performance for the MSFT guest >>>>>>>>> is >>>>>>>>> at around ~80K IOPs with multiple vioscsi LUNs + adapters, which >>>>>>>>> ends up >>>>>>>>> being well below what the Linux guest with scsi-mq + virtio-scsi is >>>>>>>>> capable of (~500K). >>>>>>>>> >>>>>>>>> After searching through the various vioscsi registry settings, it >>>>>>>>> appears that MSIEnabled is being explicitly disabled (0x00000000), >>>>>>>>> that >>>>>>>>> is different from what vioscsi.inx is currently defining: >>>>>>>>> >>>>>>>>> [pnpsafe_pci_addreg_msix] >>>>>>>>> HKR, "Interrupt Management",, 0x00000010 >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties",, >>>>>>>>> 0x00000010 >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", >>>>>>>>> MSISupported, 0x00010001, 0 >>>>>>>>> HKR, "Interrupt Management\MessageSignaledInterruptProperties", >>>>>>>>> MessageNumberLimit, 0x00010001, 4 >>>>>>>>> >>>>>>>>> Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 >>>>>>>>> is >>>>>>>>> explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as >>>>>>>>> well >>>>>>>>> as VioScsiFindAdapter() code always ends setting msix_enabled = >>>>>>>>> FALSE >>>>>>>>> here, regardless of MSI_SUPPORTED: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 >>>>>>>>> >>>>>>>>> Also looking at virtio_stor.c for the raw block driver, >>>>>>>>> MSI_SUPPORTED=1 >>>>>>>>> appears to be the default setting for the driver included in the >>>>>>>>> offical >>>>>>>>> virtio-win iso builds, right..? >>>>>>>>> >>>>>>>>> Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys >>>>>>>>> build of my own, but before going down the WDK development rabbit >>>>>>>>> whole, >>>>>>>>> I'd like to better understand why you've explicitly disabled this >>>>>>>>> logic >>>>>>>>> within vioscsi.c code to start..? >>>>>>>>> >>>>>>>>> Is there anything that needs to be addressed / carried over from >>>>>>>>> virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c >>>>>>>>> miniport code..? >>>>>>> >>>>>>> Hi Nicholas, >>>>>>> >>>>>>> I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some >>>>>>> reasons decided to keep it disabled until adding mq support. >>>>>>> >>>>>>> >>>>>>> You definitely should be able to turn on MSI_SUPPORTED, rebuild the >>>>>>> driver, and switch MSISupported to 1 to make vioscsi driver working in >>>>>>> MSI mode. >>>>>>> >>>>>> >>>>>> Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over >>>>>> the next days with a test build on Server 2012 / Server 2008 R2 and see >>>>>> how things go.. >>>>>> >>>>> >>>>> Just a quick update on progress. >>>>> >>>>> I've been able to successfully build + load a unsigned vioscsi.sys >>>>> driver on Server 2012 with WDK 8.0. >>>>> >>>>> Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant >>>>> performance and efficiency gain, on the order of 100K to 225K IOPs for >>>>> 4K block random I/O workload, depending on read/write mix. >>>>> >>>> >>>> One other performance related question.. >>>> >>>> In vioscsi.c:VioScsiFindAdapter() code, the default setting for >>>> adaptExt->queue_depth ends up getting set to 32 (pageNum / 4) when >>>> indirect mode is enabled in the following bits: >>>> >>>> if(adaptExt->indirect) { >>>> adaptExt->queue_depth = max(2, (pageNum / 4)); >>>> } else { >>>> adaptExt->queue_depth = pageNum / >>>> ConfigInfo->NumberOfPhysicalBreaks >>>> - 1; >>>> } >>>> >>>> Looking at viostor/virtio_stor.c:VirtIoFindAdapter() code, the default >>>> setting for ->queue_depth appears to be 128 (pageNum): >>>> >>>> #if (INDIRECT_SUPPORTED) >>>> if(!adaptExt->dump_mode) { >>>> adaptExt->indirect = CHECKBIT(adaptExt->features, >>>> VIRTIO_RING_F_INDIRECT_DESC); >>>> } >>>> if(adaptExt->indirect) { >>>> adaptExt->queue_depth = pageNum; >>>> } >>>> #else >>>> adaptExt->indirect = 0; >>>> #endif >>>> >>>> Is there a reason for the lower queue_depth for vioscsi vs. viostor..? >>> >>> It's a horrible work around for the following bug: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1013443 >>> >>> I'm going to remove it as soon as found better solution for it. >>> >>> Best regards, >>> Vadim. >>> >>> >> Hi Vadim, >> >> I have found that Bug 1013443 has been closed with a >> resolution of ERRATA. >> >> The windows device queue must be between 20 and 254 >> for StorPortSetDeviceQueueDepth to succeed. >> >> So I have the question that why queue_depth can not be >> set to pageNum(128)? > > It will create a problem on multi disk setup, when several > disks are attached to the same virtio-scsi pci controller. > Adding some sort of manually managed SRBs queue for storing and > resubmitting pending requests can solve this problem. > > Cheers, > Vadim. >
Is there a patch for it? >> >> Best wishes, >> Ting Wang >> >>>> >>>> How about using min(adaptExt->scsi_config.cmd_per_lun, pageNum) instead..? >>>> >>>> Thanks! >>>> >>>> -nab >>>> >>>> >> >> > > > > . >
