On 22/07/2020 15:39, Oliver O'Halloran wrote:
> On Wed, Jul 15, 2020 at 6:00 PM Alexey Kardashevskiy <a...@ozlabs.ru> wrote:
>>
>>>>> *
>>>>> - * Generally, one M64 BAR maps one IOV BAR. To avoid
>>>>> conflict
>>>>> - * with other devices, IOV BAR size is expanded to be
>>>>> - * (total_pe * VF_BAR_size). When VF_BAR_size is half of
>>>>> M64
>>>>> - * segment size , the expanded size would equal to half of
>>>>> the
>>>>> - * whole M64 space size, which will exhaust the M64 Space
>>>>> and
>>>>> - * limit the system flexibility. This is a design decision
>>>>> to
>>>>> - * set the boundary to quarter of the M64 segment size.
>>>>> + * The 1/4 limit is arbitrary and can be tweaked.
>>>>> */
>>>>> - if (total_vf_bar_sz > gate) {
>>>>> - mul = roundup_pow_of_two(total_vfs);
>>>>> - dev_info(&pdev->dev,
>>>>> - "VF BAR Total IOV size %llx > %llx, roundup
>>>>> to %d VFs\n",
>>>>> - total_vf_bar_sz, gate, mul);
>>>>> - iov->m64_single_mode = true;
>>>>> - break;
>>>>> - }
>>>>> - }
>>>>> + if (vf_bar_sz > (phb->ioda.m64_segsize >> 2)) {
>>>>> + /*
>>>>> + * On PHB3, the minimum size alignment of M64 BAR in
>>>>> + * single mode is 32MB. If this VF BAR is smaller
>>>>> than
>>>>> + * 32MB, but still too large for a segmented window
>>>>> + * then we can't map it and need to disable SR-IOV
>>>>> for
>>>>> + * this device.
>>>>
>>>>
>>>> Why not use single PE mode for such BAR? Better than nothing.
>>>
>>> Suppose you could, but I figured VFs were mainly interesting since you
>>> could give each VF to a separate guest. If there's multiple VFs under
>>> the same single PE BAR then they'd have to be assigned to the same
>>
>> True. But with one PE per VF we can still have 15 (or 14?) isolated VFs
>> which is not hundreds but better than 0.
>
> We can only use single PE BARs if the per-VF size is >= 32MB due to
> the alignment requirements on P8. If the per-VF size is smaller then
> we're stuck with multiple VFs inside the same BAR which is bad due to
> the PAPR requirements mentioned below. Sure we could look at doing
> something else, but considering this matches the current behaviour
> it's a bit hard to care...
>
>>> guest in order to retain the freeze/unfreeze behaviour that PAPR
>>> requires. I guess that's how it used to work, but it seems better just
>>> to disable them rather than having VFs which sort of work.
>>
>> Well, realistically the segment size should be 8MB to make this matter
>> (or the whole window 2GB) which does not seem to happen so it does not
>> matter.
>
> I'm not sure what you mean.
I mean how can we possibly hit this case, what m64_segsize would the
platform have to trigger this. The whole check seems useless but whatever.
--
Alexey
