Thanks for the explanation. I still think it's weird to hardcode an exception for core 0.
If no pmd-cpu-mask is specified other cores might be used, depending on the numa affinity. Perhaps we can call set_priority only if pmd-cpu-mask is specified? That seems more consistent. Thanks, Daniele 2016-07-15 7:52 GMT-07:00 Bodireddy, Bhanuprakash < bhanuprakash.bodire...@intel.com>: > >-----Original Message----- > >From: Daniele Di Proietto [mailto:daniele.di.proie...@gmail.com] > >Sent: Friday, July 15, 2016 2:19 AM > >To: Bodireddy, Bhanuprakash <bhanuprakash.bodire...@intel.com> > >Cc: dev@openvswitch.org > >Subject: Re: [ovs-dev] [PATCH v2] netdev-dpdk: Set pmd thread priority > > > >Thanks for the patch. > Hello Daniele, > Thanks for looking in to this patch. > > >Is there any reason why core 0 is treated specially? > it's very uncommon to see 'core0' isolated and HPC threads pinned to the > core0. On multicore systems to improve application performance and > mitigate Interrupts, IRQs get explicitly pinned to Core 0. In few more > cases, core 0 is treated more like a management/control core that is used > to launch applications on other cores. For this reasons I treat Core 0 > special. > > >I think we should put pmd_thread_setpriority in lib/ovs-numa.c (adding > >a ovs_numa prefix), and do nothing if dummy_numa is false. > > Agree. > > Or perhaps > >integrate it the pthread_setschedparam in > >ovs_numa_thread_setaffinity_core(). > >I've noticed that processes with the same affinity as a PMD thread will > >become > >totally unresponsive after this patch. Is this expected? Will this have a > >negative > >impact on the overall stability of the system? > > There are 2 sides to this problem. > (i) Out of Box Deployment (Not specifying dpdk-lcore-mask, pmd-cpu-mask): > As it is now, when OVS DPDK is run out of box, one pmd thread shall > be created and gets pinned to core 0. In this case the pmd thread shall run > with default scheduling policy and priority with no impact to the stability > of the system. > > (ii) High performance Deployment with SA (Explicitly specify > dpdk-lcore-mask, pmd-cpu-mask): > In this case user wants optimum Fastpath performance + SA and is > explicitly pinning the control thread and pmd threads to cores. Only in > this case the Real time scheduling policy shall be applied to the pmd > threads as any disruption to the threads would impact the fastpath > performance. > > I have come across cases where in multi VM deployments with HT enabled, > due to wrong pinning of Qemu threads to the pmd cores, > the pmd thread starvation was observed which eventually destabilizing the > system. > > Regards, > Bhanuprakash. > > > > >2016-07-05 13:05 GMT-07:00 Bhanuprakash Bodireddy > ><bhanuprakash.bodire...@intel.com>: > >Set the DPDK pmd thread scheduling policy to SCHED_RR and static > >priority to highest priority value of the policy. This is to deal with > >pmd thread starvation case where another cpu hogging process can get > >scheduled/affinitized on to the same core the pmd thread is running there > >by significantly impacting the datapath performance. > > > >Setting the realtime scheduling policy to the pmd threads is one step > >towards Fastpath Service Assurance in OVS DPDK. > > > >The realtime scheduling policy is applied only when CPU mask is passed > >to 'pmd-cpu-mask'. The exception to this is 'pmd-cpu-mask=1', where the > >policy and priority shall not be applied to pmd thread spawned on core0. > >For example: > > > > * In the absence of pmd-cpu-mask or if pmd-cpu-mask=1, one pmd > > thread shall be created and affinitized to 'core 0' with default > > scheduling policy and priority applied. > > > > * If pmd-cpu-mask is specified with CPU mask > 1, one or more pmd > > threads shall be spawned on the corresponding core(s) in the mask > > and real time scheduling policy SCHED_RR and highest static > > priority is applied to the pmd thread(s). > > > >To reproduce the issue use following commands: > > > >ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=6 > >taskset 0x2 cat /dev/zero > /dev/null & > > > >Also OVS control threads should not be affinitized to the pmd cores. > >For example 'dpdk-lcore-mask' and 'pmd-cpu-mask' should be exclusive. > > > >v1->v2: > >* Removed #ifdef and introduced dummy function "pmd_thread_setpriority" > > in netdev-dpdk.h > >* Rebase > > > >Signed-off-by: Bhanuprakash Bodireddy > ><bhanuprakash.bodire...@intel.com> > >--- > > lib/dpif-netdev.c | 8 ++++++++ > > lib/netdev-dpdk.c | 14 ++++++++++++++ > > lib/netdev-dpdk.h | 7 +++++++ > > 3 files changed, 29 insertions(+) > > > >diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c > >index 37c2631..6ff81d6 100644 > >--- a/lib/dpif-netdev.c > >+++ b/lib/dpif-netdev.c > >@@ -2849,6 +2849,14 @@ pmd_thread_main(void *f_) > > ovs_numa_thread_setaffinity_core(pmd->core_id); > > dpdk_set_lcore_id(pmd->core_id); > > poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list); > >+ > >+ /* Set pmd thread's scheduling policy to SCHED_RR and priority to > >+ * highest priority of SCHED_RR policy, In absence of pmd-cpu-mask > (or) > >+ * pmd-cpu-mask=1, default scheduling policy and priority shall > >+ * apply to pmd thread */ > >+ if (pmd->core_id) { > >+ pmd_thread_setpriority(SCHED_RR); > >+ } > > reload: > > emc_cache_init(&pmd->flow_cache); > > > >diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c > >index 02e2c58..ce1683b 100644 > >--- a/lib/netdev-dpdk.c > >+++ b/lib/netdev-dpdk.c > >@@ -3541,3 +3541,17 @@ dpdk_thread_is_pmd(void) > > { > > return rte_lcore_id() != NON_PMD_CORE_ID; > > } > >+ > >+void > >+pmd_thread_setpriority(int policy) > >+{ > >+ struct sched_param threadparam; > >+ int err; > >+ > >+ memset(&threadparam, 0, sizeof(threadparam)); > >+ threadparam.sched_priority = sched_get_priority_max(policy); > >+ err = pthread_setschedparam(pthread_self(), policy, &threadparam); > >+ if (err) { > >+ VLOG_WARN("Thread priority error %d",err); > >+ } > >+} > >diff --git a/lib/netdev-dpdk.h b/lib/netdev-dpdk.h > >index 80bb834..1890ae4 100644 > >--- a/lib/netdev-dpdk.h > >+++ b/lib/netdev-dpdk.h > >@@ -26,6 +26,7 @@ struct smap; > > void netdev_dpdk_register(void); > > void free_dpdk_buf(struct dp_packet *); > > void dpdk_set_lcore_id(unsigned cpu); > >+void pmd_thread_setpriority(int policy); > > > > #else > > > >@@ -51,6 +52,12 @@ dpdk_set_lcore_id(unsigned cpu OVS_UNUSED) > > /* Nothing */ > > } > > > >+static inline void > >+pmd_thread_setpriority(int policy OVS_UNUSED) > >+{ > >+ /* Nothing */ > >+} > >+ > > #endif /* DPDK_NETDEV */ > > > > void dpdk_init(const struct smap *ovs_other_config); > >-- > >2.4.11 > > > >_______________________________________________ > >dev mailing list > >dev@openvswitch.org > >http://openvswitch.org/mailman/listinfo/dev > > _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
