On 05.02.15 12:30, David Gibson wrote:
> On Thu, Feb 05, 2015 at 11:22:13AM +0100, Alexander Graf wrote:
>>
>>
>>
>>> Am 05.02.2015 um 03:55 schrieb David Gibson <da...@gibson.dropbear.id.au>:
>>>
>>>> On Thu, Feb 05, 2015 at 01:54:39AM +0100, Alexander Graf wrote:
>>>>
>>>>
>>>>> On 05.02.15 01:48, David Gibson wrote:
>>>>>> On Wed, Feb 04, 2015 at 04:19:14PM +0100, Alexander Graf wrote:
>>>>>>
>>>>>>
>>>>>>> On 04.02.15 02:32, David Gibson wrote:
>>>>>>>> On Wed, Feb 04, 2015 at 08:19:06AM +1100, Paul Mackerras wrote:
>>>>>>>>> On Tue, Feb 03, 2015 at 05:10:51PM +1100, David Gibson wrote:
>>>>>>>>> qemu currently implements the hypercalls H_LOGICAL_CI_LOAD and
>>>>>>>>> H_LOGICAL_CI_STORE as PAPR extensions. These are used by the SLOF
>>>>>>>>> firmware
>>>>>>>>> for IO, because performing cache inhibited MMIO accesses with the MMU
>>>>>>>>> off
>>>>>>>>> (real mode) is very awkward on POWER.
>>>>>>>>>
>>>>>>>>> This approach breaks when SLOF needs to access IO devices implemented
>>>>>>>>> within KVM instead of in qemu. The simplest example would be
>>>>>>>>> virtio-blk
>>>>>>>>> using an iothread, because the iothread / dataplane mechanism relies
>>>>>>>>> on
>>>>>>>>> an in-kernel implementation of the virtio queue notification MMIO.
>>>>>>>>>
>>>>>>>>> To fix this, an in-kernel implementation of these hypercalls has been
>>>>>>>>> made,
>>>>>>>>> however, the hypercalls still need to be enabled from qemu. This
>>>>>>>>> performs
>>>>>>>>> the necessary calls to do so.
>>>>>>>>>
>>>>>>>>> Signed-off-by: David Gibson <da...@gibson.dropbear.id.au>
>>>>>>>>
>>>>>>>> [snip]
>>>>>>>>
>>>>>>>>> + ret1 = kvmppc_enable_hcall(kvm_state, H_LOGICAL_CI_LOAD);
>>>>>>>>> + if (ret1 != 0) {
>>>>>>>>> + fprintf(stderr, "Warning: error enabling H_LOGICAL_CI_LOAD
>>>>>>>>> in KVM:"
>>>>>>>>> + " %s\n", strerror(errno));
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + ret2 = kvmppc_enable_hcall(kvm_state, H_LOGICAL_CI_STORE);
>>>>>>>>> + if (ret2 != 0) {
>>>>>>>>> + fprintf(stderr, "Warning: error enabling H_LOGICAL_CI_STORE
>>>>>>>>> in KVM:"
>>>>>>>>> + " %s\n", strerror(errno));
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + if ((ret1 != 0) || (ret2 != 0)) {
>>>>>>>>> + fprintf(stderr, "Warning: Couldn't enable H_LOGICAL_CI_* in
>>>>>>>>> KVM, SLOF"
>>>>>>>>> + " may be unable to operate devices with in-kernel
>>>>>>>>> emulation\n");
>>>>>>>>> + }
>>>>>>>>
>>>>>>>> You'll always get these warnings if you're running on an old (meaning
>>>>>>>> current upstream) kernel, which could be annoying.
>>>>>>>
>>>>>>> True.
>>>>>>>
>>>>>>>> Is there any way
>>>>>>>> to tell whether you have configured any devices which need the
>>>>>>>> in-kernel MMIO emulation and only warn if you have?
>>>>>>>
>>>>>>> In theory, I guess so. In practice I can't see how you'd enumerate
>>>>>>> all devices that might require kernel intervention without something
>>>>>>> horribly invasive.
>>>>>>
>>>>>> We could WARN_ONCE in QEMU if we emulate such a hypercall, but its
>>>>>> handler is io_mem_unassigned (or we add another minimum priority huge
>>>>>> memory region on all 64bits of address space that reports the breakage).
>>>>>
>>>>> Would that work for the virtio+iothread case? I had the impression
>>>>> the kernel handled notification region was layered over the qemu
>>>>> emulated region in that case.
>>>>
>>>> IIRC we don't have a way to call back into kvm saying "please write to
>>>> this in-kernel device". But we could at least defer the warning to a
>>>> point where we know that we actually hit it.
>>>
>>> Right, but I'm saying we might miss the warning in cases where we want
>>> it, because the KVM device is shadowed by a qemu device, so qemu won't
>>> see the IO as unassigned or unhandled.
>>>
>>> In particular, I think that will happen in the case of virtio-blk with
>>> iothread, which is the simplest case in which to observe the problem.
>>> The virtio-blk device exists in qemu and is functional, but we rely on
>>> KVM catching the queue notification MMIO before it reaches the qemu
>>> implementation of the rest of the device's IO space.
>>
>> But in that case the VM stays functional and will merely see a
>> performance hit when using virtio in SLOF, no? I don't think that's
>> a problem worth worrying users about.
>
> Alas, no. The iothread stuff *relies* on the in-kernel notification,
> so it will not work if the IO gets punted to qemu. This is the whole
> reason for the in-kernel hcall implementation.
So at least with vhost-net the in-kernel trapping is optional. If we
happen to get MMIO into QEMU, we'll just handle it there.
Enlighten me why the iothread stuff can't handle it that way too.
Alex
