>Thanks for the explanation. > >I still think it's weird to hardcode an exception for core 0. > >If no pmd-cpu-mask is specified other cores might be used, depending on the >numa affinity. >Perhaps we can call set_priority only if pmd-cpu-mask is specified? That >seems more consistent.
I agree to this and looks good to me. I will send out v3 as discussed here. Regards, Bhanu Prakash. >Thanks, >Daniele > >2016-07-15 7:52 GMT-07:00 Bodireddy, Bhanuprakash ><bhanuprakash.bodire...@intel.com>: >>-----Original Message----- >>From: Daniele Di Proietto [mailto:daniele.di.proie...@gmail.com] >>Sent: Friday, July 15, 2016 2:19 AM >>To: Bodireddy, Bhanuprakash <bhanuprakash.bodire...@intel.com> >>Cc: dev@openvswitch.org >>Subject: Re: [ovs-dev] [PATCH v2] netdev-dpdk: Set pmd thread priority >> >>Thanks for the patch. >Hello Daniele, >Thanks for looking in to this patch. > >>Is there any reason why core 0 is treated specially? >it's very uncommon to see 'core0' isolated and HPC threads pinned to the >core0. On multicore systems to improve application performance and >mitigate Interrupts, IRQs get explicitly pinned to Core 0. In few more cases, >core 0 is treated more like a management/control core that is used to launch >applications on other cores. For this reasons I treat Core 0 special. > >>I think we should put pmd_thread_setpriority in lib/ovs-numa.c (adding >>a ovs_numa prefix), and do nothing if dummy_numa is false. > >Agree. > > Or perhaps >>integrate it the pthread_setschedparam in >>ovs_numa_thread_setaffinity_core(). >>I've noticed that processes with the same affinity as a PMD thread will >>become >>totally unresponsive after this patch. Is this expected? Will this have a >>negative >>impact on the overall stability of the system? > >There are 2 sides to this problem. >(i) Out of Box Deployment (Not specifying dpdk-lcore-mask, pmd-cpu-mask): > As it is now, when OVS DPDK is run out of box, one pmd thread shall be >created and gets pinned to core 0. In this case the pmd thread shall run with >default scheduling policy and priority with no impact to the stability of the >system. > >(ii) High performance Deployment with SA (Explicitly specify dpdk-lcore-mask, >pmd-cpu-mask): > In this case user wants optimum Fastpath performance + SA and is >explicitly pinning the control thread and pmd threads to cores. Only in this >case the Real time scheduling policy shall be applied to the pmd threads as any >disruption to the threads would impact the fastpath performance. > >I have come across cases where in multi VM deployments with HT enabled, >due to wrong pinning of Qemu threads to the pmd cores, >the pmd thread starvation was observed which eventually destabilizing the >system. > >Regards, >Bhanuprakash. > >> >>2016-07-05 13:05 GMT-07:00 Bhanuprakash Bodireddy >><bhanuprakash.bodire...@intel.com>: >>Set the DPDK pmd thread scheduling policy to SCHED_RR and static >>priority to highest priority value of the policy. This is to deal with >>pmd thread starvation case where another cpu hogging process can get >>scheduled/affinitized on to the same core the pmd thread is running there >>by significantly impacting the datapath performance. >> >>Setting the realtime scheduling policy to the pmd threads is one step >>towards Fastpath Service Assurance in OVS DPDK. >> >>The realtime scheduling policy is applied only when CPU mask is passed >>to 'pmd-cpu-mask'. The exception to this is 'pmd-cpu-mask=1', where the >>policy and priority shall not be applied to pmd thread spawned on core0. >>For example: >> >> * In the absence of pmd-cpu-mask or if pmd-cpu-mask=1, one pmd >> thread shall be created and affinitized to 'core 0' with default >> scheduling policy and priority applied. >> >> * If pmd-cpu-mask is specified with CPU mask > 1, one or more pmd >> threads shall be spawned on the corresponding core(s) in the mask >> and real time scheduling policy SCHED_RR and highest static >> priority is applied to the pmd thread(s). >> >>To reproduce the issue use following commands: >> >>ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=6 >>taskset 0x2 cat /dev/zero > /dev/null & >> >>Also OVS control threads should not be affinitized to the pmd cores. >>For example 'dpdk-lcore-mask' and 'pmd-cpu-mask' should be exclusive. >> >>v1->v2: >>* Removed #ifdef and introduced dummy function >"pmd_thread_setpriority" >> in netdev-dpdk.h >>* Rebase >> >>Signed-off-by: Bhanuprakash Bodireddy >><bhanuprakash.bodire...@intel.com> >>--- >> lib/dpif-netdev.c | 8 ++++++++ >> lib/netdev-dpdk.c | 14 ++++++++++++++ >> lib/netdev-dpdk.h | 7 +++++++ >> 3 files changed, 29 insertions(+) >> >>diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c >>index 37c2631..6ff81d6 100644 >>--- a/lib/dpif-netdev.c >>+++ b/lib/dpif-netdev.c >>@@ -2849,6 +2849,14 @@ pmd_thread_main(void *f_) >> ovs_numa_thread_setaffinity_core(pmd->core_id); >> dpdk_set_lcore_id(pmd->core_id); >> poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list); >>+ >>+ /* Set pmd thread's scheduling policy to SCHED_RR and priority to >>+ * highest priority of SCHED_RR policy, In absence of pmd-cpu-mask (or) >>+ * pmd-cpu-mask=1, default scheduling policy and priority shall >>+ * apply to pmd thread */ >>+ if (pmd->core_id) { >>+ pmd_thread_setpriority(SCHED_RR); >>+ } >> reload: >> emc_cache_init(&pmd->flow_cache); >> >>diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c >>index 02e2c58..ce1683b 100644 >>--- a/lib/netdev-dpdk.c >>+++ b/lib/netdev-dpdk.c >>@@ -3541,3 +3541,17 @@ dpdk_thread_is_pmd(void) >> { >> return rte_lcore_id() != NON_PMD_CORE_ID; >> } >>+ >>+void >>+pmd_thread_setpriority(int policy) >>+{ >>+ struct sched_param threadparam; >>+ int err; >>+ >>+ memset(&threadparam, 0, sizeof(threadparam)); >>+ threadparam.sched_priority = sched_get_priority_max(policy); >>+ err = pthread_setschedparam(pthread_self(), policy, &threadparam); >>+ if (err) { >>+ VLOG_WARN("Thread priority error %d",err); >>+ } >>+} >>diff --git a/lib/netdev-dpdk.h b/lib/netdev-dpdk.h >>index 80bb834..1890ae4 100644 >>--- a/lib/netdev-dpdk.h >>+++ b/lib/netdev-dpdk.h >>@@ -26,6 +26,7 @@ struct smap; >> void netdev_dpdk_register(void); >> void free_dpdk_buf(struct dp_packet *); >> void dpdk_set_lcore_id(unsigned cpu); >>+void pmd_thread_setpriority(int policy); >> >> #else >> >>@@ -51,6 +52,12 @@ dpdk_set_lcore_id(unsigned cpu OVS_UNUSED) >> /* Nothing */ >> } >> >>+static inline void >>+pmd_thread_setpriority(int policy OVS_UNUSED) >>+{ >>+ /* Nothing */ >>+} >>+ >> #endif /* DPDK_NETDEV */ >> >> void dpdk_init(const struct smap *ovs_other_config); >>-- >>2.4.11 >> >>_______________________________________________ >>dev mailing list >>dev@openvswitch.org >>http://openvswitch.org/mailman/listinfo/dev _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
