> On Mar 18, 2021, at 8:59 PM, Phil Auld <pa...@redhat.com> wrote:
>
> On Thu, Mar 18, 2021 at 09:26:58AM +0800 changhuaixin wrote:
>>
>>
>>> On Mar 17, 2021, at 4:06 PM, Peter Zijlstra <pet...@infradead.org> wrote:
>>>
>>> On Wed, Mar 17, 2021 at 03:16:18PM +0800, changhuaixin wrote:
>>>
>>>>> Why do you allow such a large burst? I would expect something like:
>>>>>
>>>>> if (burst > quote)
>>>>> return -EINVAL;
>>>>>
>>>>> That limits the variance in the system. Allowing super long bursts seems
>>>>> to defeat the entire purpose of bandwidth control.
>>>>
>>>> I understand your concern. Surely large burst value might allow super
>>>> long bursts thus preventing bandwidth control entirely for a long
>>>> time.
>>>>
>>>> However, I am afraid it is hard to decide what the maximum burst
>>>> should be from the bandwidth control mechanism itself. Allowing some
>>>> burst to the maximum of quota is helpful, but not enough. There are
>>>> cases where workloads are bursty that they need many times more than
>>>> quota in a single period. In such cases, limiting burst to the maximum
>>>> of quota fails to meet the needs.
>>>>
>>>> Thus, I wonder whether is it acceptable to leave the maximum burst to
>>>> users. If the desired behavior is to allow some burst, configure burst
>>>> accordingly. If that is causing variance, use share or other fairness
>>>> mechanism. And if fairness mechanism still fails to coordinate, do not
>>>> use burst maybe.
>>>
>>> It's not fairness, bandwidth control is about isolation, and burst
>>> introduces interference.
>>>
>>>> In this way, cfs_b->buffer can be removed while cfs_b->max_overrun is
>>>> still needed maybe.
>>>
>>> So what is the typical avg,stdev,max and mode for the workloads where you
>>> find
>>> you need this?
>>>
>>> I would really like to put a limit on the burst. IMO a workload that has
>>> a burst many times longer than the quota is plain broken.
>>
>> I see. Then the problem comes down to how large the limit on burst shall be.
>>
>> I have sampled the CPU usage of a bursty container in 100ms periods. The
>> statistics are:
>> average : 42.2%
>> stddev : 81.5%
>> max : 844.5%
>> P95 : 183.3%
>> P99 : 437.0%
>>
>> If quota is 100000ms, burst buffer needs to be 8 times more in order for
>> this workload not to be throttled.
>> I can't say this is typical, but these workloads exist. On a machine running
>> Kubernetes containers,
>> where there is often room for such burst and the interference is hard to
>> notice, users would prefer
>> allowing such burst to being throttled occasionally.
>>
>
> I admit to not having followed all the history of this patch set. That said,
> when I see the above I just
> think your quota is too low for your workload.
>
Yeah, more quota is helpful for this workload. But that usually prevents us
from improving the total CPU
usage by putting more work onto a single machine.
> The burst (mis?)feature seems to be a way to bypass the quota. And it sort
> of assumes cooperative
> containers that will only burst when they need it and then go back to normal.
>
>> In this sense, I suggest limit burst buffer to 16 times of quota or around.
>> That should be enough for users to
>> improve tail latency caused by throttling. And users might choose a smaller
>> one or even none, if the interference
>> is unacceptable. What do you think?
>>
>
> Having quotas that can regularly be exceeded by 16 times seems to make the
> concept of a quota
> meaningless. I'd have thought a burst would be some small percentage.
>
> What if several such containers burst at the same time? Can't that lead to
> overcommit that can effect
> other well-behaved containers?
>
I see. Maybe there should be some calculation on the probabilities of that, as
Peter has replied.
>
> Cheers,
> Phil
>
> --
