On Mon, 26 Apr 2021 15:14:59 +0530 Jerin Jacob <jerinjac...@gmail.com> wrote:
> On Sat, Apr 24, 2021 at 5:56 AM Honnappa Nagarahalli > <honnappa.nagaraha...@arm.com> wrote: > > > > <snip> > > > > > > > > > > > > On Thu, Mar 11, 2021 at 12:01 AM Honnappa Nagarahalli > > > > > > > > > > <honnappa.nagaraha...@arm.com> wrote: > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > Performance of L3fwd example application is one > > > > > > > > > > > of the key > > > > > > > > > > benchmarks in DPDK. However, the application does not have > > > > > > > > > > many debugging statistics to understand the performance > > > > > > > > > > issues. We have added L3fwd as another mode/stream to > > > > > > > > > > testpmd which provides > > > > > > > > enough > > > > > > > > > > statistics at various levels. This has allowed us to debug > > > > > > > > > > the performance issues effectively. > > > > > > > > > > > > > > > > > > > > > > There is more work to be done to get it to upstreamable > > > > > > > > > > > state. I am > > > > > > > > > > wondering if such a patch is helpful for others and if the > > > > > > > > > > community would be interested in taking a look. Please let > > > > > > > > > > me know > > > > > > > what you think. > > > > > > > > > > > > > > > > > > > > We are using app/proc-info/ to attach and analyze the > > > performance. > > > > > > > > > > That helps to analyze the unmodified application. I think, > > > > > > > > > > if something is missing in proc-info app, in my opinion it > > > > > > > > > > is better to enhance proc-info so that it can help other > > > > > > > > > > third-party > > > > > applications. > > > > > > > > > > > > > > > > > > > > Just my 2c. > > > > > > > > > Thanks Jerin. We will explore that. > > > > > > > > > > > > > > > > I agree it is dangerous to rely too much on testpmd for > > > > > > > > everything. > > > > > > > > Please tell us what in testpmd could be useful out of it. > > > > > > > > > > > > > > > Things that are very helpful in testpmd are: 1) HW statistics > > > > > > > from the NIC 2) Forwarding stats 3) Burst stats (indication of > > > > > > > headroom > > > > > > > availability) 4) Easy to set parameters like RX and TX queue > > > > > > > depths (among others) without having to recompile. > > > > > > > > > > > > [Kathleen Capella] > > > > > > Thank you for the suggestion of app/proc-info. I've tried it out > > > > > > with l3fwd and see that it does have the HW stats from the NIC and > > > > > > the forwarding > > > > > stats. > > > > > > However, it does not have the burst stats testpmd offers, nor the > > > > > > > > > > One option to see such level of debugging would be to have > > > > > - Create a memzone in the primary process > > > > > - Application under test can update the stats in memzone based on > > > > > the code flow > > > > > - proc-info can read the counters updated by application under test > > > > > using the memzone object got through rte_memzone_lookup() > > > > Agreed. Currently, using app/proc-info does not provide this ability. > > > > We > > > cannot add this capability to app/proc-info as these stats would be > > > specific to > > > L3fwd application. > > > > > > I meant creating generic counter-read/write infra via memzone to not make > > > it > > > as l3fwd specific. > > Currently, app/proc-info is able to print the stats as they are > > standardized via the API. But for statistics that are generated in the > > application, they are very specific to that application. For ex: burst > > stats in testpmd are very specific to it and another application might > > implement the same in a very different manner. > > > > In needs to be something like the app/proc-info just needs to be a dumb > > displaying utility and the application has to do all the heavy lifting of > > copying the exact display strings to the memory. > > Yes. > > > > > > > > > > > > > Another approach will be using rte_trace()[1] for debugging/tracing > > > > > by adding tracepoints in l3fwd for such events. > > > > > It has a timestamp and the trace format is opensource trace > > > > > format(CTF(Common trace format)), so that we can use post posting > > > > > tools to analyze. > > > > > [1] > > > > > https://doc.dpdk.org/guides/prog_guide/trace_lib.html > > > > This is good for analyzing an incident. I think it is an overhead for > > > development purposes. > > > > > > Consider if one wants to add burst stats, one can add stats increment > > > under > > > RTE_TRACE_POINT_FP, it will be emitted whenever code flow through that > > > path. Set of events of can be viewed in trace viewer[1]. Would that be > > > enough? > > > Adding traces to l3fwd can be upstreamed as it is useful for others for > > > debugging. > > > > > > [1] > > > https://github.com/jerinjacobk/share/blob/master/dpdk_trace.JPG > > This needs post processing of the trace info to derive the information, is > > it correct? For ex: for burst stats, there will be several traces generated > > collecting the number of packets returned by rte_eth_rx_burst which needs > > to be post processed. > > Or You can have an additional variable to acculate it. > > > Also, adding traces is equivalent to adding statistics in L3fwd. > > Yes. > > If the sole purpose only stats then it is better to add status in > l3fwd without performance impact. I thought some thing else. > > > > > > > > > > > > > > ability to easily change parameters without having to recompile, > > > > > > which helps reduce debugging time significantly. > > We will not be able to fix this above issue. > > It depends on what you want to debug. Trace can be disabled at runtime. DPDK has existing API's for application metrics but they are rarely used. Why not implement rte_metrics in l3fwd and proc-info?
