- Test environment: Intel Xeon Server platform CPU(s): 192 On-line CPU(s) list: 0-191 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 NUMA node(s): 4
- Kernel under test: Core scheduling v5 base https://github.com/digitalocean/linux-coresched/tree/coresched/v5-v5.5.y - Test set based on sysbench 1.1.0-bd4b418: A: sysbench cpu in cgroup cpu 1 + sysbench mysql in cgroup mysql 1 (192 workload tasks for each cgroup) B: sysbench cpu in cgroup cpu 1 + sysbench cpu in cgroup cpu 2 + sysbench mysql in cgroup mysql 1 + sysbench mysql in cgroup mysql 2 (192 workload tasks for each cgroup) - Test results briefing: 1 Good results: 1.1 For test set A, coresched could achieve same or better performance compared to smt_off, for both cpu workload and sysbench workload 1.2 For test set B, cpu workload, coresched could achieve better performance compared to smt_off 2 Bad results: 2.1 For test set B, mysql workload, coresched performance is lower than smt_off, potential fairness issue between cpu workloads and mysql workloads 2.2 For test set B, cpu workload, potential fairness issue between 2 cgroups cpu workloads - Test results: Note: test results in following tables are Tput normalized to default baseline -- Test set A Tput normalized results: +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+ | | **** | default | coresched | smt_off | *** | default | coresched | smt_off | +====================+========+===========+=============+===========+=======+=============+===============+=============+ | cgroups | **** | cg cpu 1 | cg cpu 1 | cg cpu 1 | *** | cg mysql 1 | cg mysql 1 | cg mysql 1 | +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+ | sysbench workload | **** | cpu | cpu | cpu | *** | mysql | mysql | mysql | +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+ | 192 tasks / cgroup | **** | 1 | 0.95 | 0.54 | *** | 1 | 0.92 | 0.97 | +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+ -- Test set B Tput normalized results: +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+------+-------------+---------------+-------------+-----+-------------+---------------+-------------+ | | **** | default | coresched | smt_off | *** | default | coresched | smt_off | ** | default | coresched | smt_off | * | default | coresched | smt_off | +====================+========+===========+=============+===========+=======+=============+===============+=============+======+=============+===============+=============+=====+=============+===============+=============+ | cgroups | **** | cg cpu 1 | cg cpu 1 | cg cpu 1 | *** | cg cpu 2 | cg cpu 2 | cg cpu 2 | ** | cg mysql 1 | cg mysql 1 | cg mysql 1 | * | cg mysql 2 | cg mysql 2 | cg mysql 2 | +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+------+-------------+---------------+-------------+-----+-------------+---------------+-------------+ | sysbench workload | **** | cpu | cpu | cpu | *** | cpu | cpu | cpu | ** | mysql | mysql | mysql | * | mysql | mysql | mysql | +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+------+-------------+---------------+-------------+-----+-------------+---------------+-------------+ | 192 tasks / cgroup | **** | 1 | 0.9 | 0.47 | *** | 1 | 1.32 | 0.66 | ** | 1 | 0.42 | 0.89 | * | 1 | 0.42 | 0.89 | +--------------------+--------+-----------+-------------+-----------+-------+-------------+---------------+-------------+------+-------------+---------------+-------------+-----+-------------+---------------+-------------+ > On Date: Wed, 4 Mar 2020 16:59:50 +0000, vpillai <vpil...@digitalocean.com> > wrote: > To: Nishanth Aravamudan <naravamu...@digitalocean.com>, Julien Desfossez > <jdesfos...@digitalocean.com>, Peter Zijlstra <pet...@infradead.org>, Tim > Chen <tim.c.c...@linux.intel.com>, mi...@kernel.org, t...@linutronix.de, > p...@google.com, torva...@linux-foundation.org > CC: vpillai <vpil...@digitalocean.com>, linux-kernel@vger.kernel.org, > fweis...@gmail.com, keesc...@chromium.org, kerr...@google.com, Phil Auld > <pa...@redhat.com>, Aaron Lu <aaron....@gmail.com>, Aubrey Li > <aubrey.in...@gmail.com>, aubrey...@linux.intel.com, Valentin Schneider > <valentin.schnei...@arm.com>, Mel Gorman <mgor...@techsingularity.net>, Pawan > Gupta <pawan.kumar.gu...@linux.intel.com>, Paolo Bonzini > <pbonz...@redhat.com>, Joel Fernandes <joe...@google.com>, > j...@joelfernandes.org > > > Fifth iteration of the Core-Scheduling feature. > > Core scheduling is a feature that only allows trusted tasks to run > concurrently on cpus sharing compute resources(eg: hyperthreads on a > core). The goal is to mitigate the core-level side-channel attacks > without requiring to disable SMT (which has a significant impact on > performance in some situations). So far, the feature mitigates user-space > to user-space attacks but not user-space to kernel attack, when one of > the hardware thread enters the kernel (syscall, interrupt etc). > > By default, the feature doesn't change any of the current scheduler > behavior. The user decides which tasks can run simultaneously on the > same core (for now by having them in the same tagged cgroup). When > a tag is enabled in a cgroup and a task from that cgroup is running > on a hardware thread, the scheduler ensures that only idle or trusted > tasks run on the other sibling(s). Besides security concerns, this > feature can also be beneficial for RT and performance applications > where we want to control how tasks make use of SMT dynamically. > > This version was focusing on performance and stability. Couple of > crashes related to task tagging and cpu hotplug path were fixed. > This version also improves the performance considerably by making > task migration and load balancing coresched aware. > > In terms of performance, the major difference since the last iteration > is that now even IO-heavy and mixed-resources workloads are less > impacted by core-scheduling than by disabling SMT. Both host-level and > VM-level benchmarks were performed. Details in: > https://lkml.org/lkml/2020/2/12/1194 > https://lkml.org/lkml/2019/11/1/269 > > v5 is rebased on top of 5.5.5(449718782a46) > https://github.com/digitalocean/linux-coresched/tree/coresched/v5-v5.5.y >
