On (11/09/23 17:32), Bruce Richardson wrote: > Date: Thu, 9 Nov 2023 17:32:31 +0000 > From: Bruce Richardson <bruce.richard...@intel.com> > To: Rahul Gupta <rahulg...@linux.microsoft.com> > CC: Thomas Monjalon <tho...@monjalon.net>, rahul gupta > <rahulrgupt...@gmail.com>, Dmitry Kozlyuk <dmitry.kozl...@gmail.com>, > Stephen Hemminger <step...@networkplumber.org>, dev@dpdk.org, > sovar...@linux.microsoft.com, ok...@kernel.org, > sujithsan...@microsoft.com, sowmini.varad...@microsoft.com > Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > On Thu, Nov 09, 2023 at 09:26:27AM -0800, Rahul Gupta wrote: > > On (11/08/23 16:40), Thomas Monjalon wrote: > > > Date: Wed, 08 Nov 2023 16:40:07 +0100 > > > From: Thomas Monjalon <tho...@monjalon.net> > > > To: rahul gupta <rahulrgupt...@gmail.com>, Dmitry Kozlyuk > > > <dmitry.kozl...@gmail.com> > > > Cc: Stephen Hemminger <step...@networkplumber.org>, Rahul Gupta > > > <rahulg...@linux.microsoft.com>, dev@dpdk.org, > > > sovar...@linux.microsoft.com, ok...@kernel.org, > > > sujithsan...@microsoft.com, sowmini.varad...@microsoft.com, Rahul Gupta > > > <rahulg...@microsoft.com> > > > Subject: Re: [RFC] eal: RFC to refactor rte_eal_init into sub-functions > > > > > > 08/11/2023 14:53, Dmitry Kozlyuk: > > > > 2023-11-07 23:03 (UTC+0530), rahul gupta: > > > > > > > From: Rahul Gupta <rahulg...@linux.microsoft.com> > > > > > > > To: dev@dpdk.org, tho...@monjalon.net > > > > > > > Cc: sovar...@linux.microsoft.com, ok...@kernel.org, > > > > > > sujithsan...@microsoft.com, sowmini.varad...@microsoft.com, > > > > > > rahulrgupt...@gmail.com, Rahul Gupta <rahulg...@microsoft.com>, > > > > > > Rahul > > > > > > Gupta <rahulg...@linux.microsoft.com> > > > > > > > Subject: [RFC] eal: RFC to refactor rte_eal_init into > > > > > > > sub-functions > > > > > > > Date: Thu, 2 Nov 2023 11:19:24 -0700 > > > > > > > X-Mailer: git-send-email 1.8.3.1 > > > > > > > > > > > > > > From: Rahul Gupta <rahulg...@microsoft.com> > > > > > > > > > > > > > > Initialization often requires rte_eal_init + > > > > > > > rte_pktmbuf_pool_create > > > > > > > which can consume a total time of 500-600 ms: > > > > > > > a) For many devices FLR may take a significant chunk of time > > > > > > > (200-250 ms in our use-case), this FLR is triggered during > > > > > > > device > > > > > > > probe in rte_eal_init(). > > > > > > > b) rte_pktmbuf_pool_create() can consume upto 300-350 ms for > > > > > > > applications that require huge memory. > > > > > > > > > > > > > > This cost is incurred on each restart (which happens in our > > > > > > > use-case > > > > > > > during binary updates for servicing). > > > > > > > This patch provides an optimization using pthreads that > > > > > > > appplications > > > > > > > can use and which can save 200-230ms. > > > > > > > > > > > > > > In this patch, rte_eal_init() is refactored into two parts- > > > > > > > a) 1st part is dependent code ie- it’s a perquisite of the FLR and > > > > > > > mempool creation. So this code needs to be executed before any > > > > > > > pthreads. Its named as rte_eal_init_setup() > > > > > > > b) 2nd part of code is independent code ie- it can execute in > > > > > > > parallel > > > > > > > to mempool creation in a pthread. Its named as > > > > > > > rte_probe_and_ioctl(). > > > > > > > > > > > > > > Existing applications require no changes unless they wish to > > > > > > > leverage > > > > > > > the optimization. > > > > > > > > > > > > > > If the application wants to use pthread functionality, it should > > > > > > > call- > > > > > > > a) rte_eal_init_setup() then create two or more pthreads- > > > > > > > b) in one pthread call- rte_probe_and_ioctl(), > > > > > > > c) second pthread call- rte_pktmbuf_pool_create() > > > > > > > d) (optional) Other pthreads for any other independent function. > > > > > > > > > > > > > > Signed-off-by: Rahul Gupta <rahulg...@linux.microsoft.com> > > > > > > > > I doubt that the new API is required. > > > > It is already possible to block all devices from automatic probing > > > > with EAL options and then probe explicitly in any threads desired. > > > > At the same time, this RFC shows a valuable optimization pattern, > > > > so maybe it is worth having in DPDK as an example. > > > > There are DPDK use cases when probing is completely unnecessary. > > > > > > It seems here we want to do the device probing, > > > but start it in parallel of other tasks. > > > > > > > Exposing the initialization process stages makes it harder to refactor > > > > and requires precise documentation of when and what is initialized > > > > (for example, in this RFC rte_eal_init_setup() > > > > does not make service core API usable yet). > > > > > > Yes the init order is sensitive, that's why we have a big init function. > > > But in general I would agree to try splitting it with necessary warnings > > > and explanations. > > > > > > > P. S. You may be also interested in using `--huge-unlink=never` > > > > to speed rte_pktmbuf_pool_create() during restarts: > > > > > > > > https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3 > > > > > > Yes good tip :) > > > > > > > > Thank you for the comments. I will send a patch shortly. > > eal_init_async(); //Internally forks a thread to do FLR. > > /* Application can do other stuff, including mempool_create, possibly in > > multiple threads. If threads are forked, then application has to do any > > needed thread-joins */ > > eal_init_async_done(); //To sync with FLR thread. > > Just to note, the documentation on rte_eal_init_async() needs to call out > very explicitly what DPDK APIs, if any, can be called before the call to > async_done(). > > /Bruce
Yes, I will document it in commit log and near code. Regards, Rahul.
