On Fri, Feb 21, 2020 at 1:28 PM Van Haaren, Harry <harry.van.haa...@intel.com> wrote: > > > -----Original Message----- > > From: David Marchand <david.march...@redhat.com> > > Sent: Thursday, February 20, 2020 1:25 PM > > To: Van Haaren, Harry <harry.van.haa...@intel.com> > > Cc: Aaron Conole <acon...@redhat.com>; dev <dev@dpdk.org> > > Subject: Re: [RFC] service: stop lcore threads before 'finalize' > > > > On Mon, Feb 10, 2020 at 3:16 PM Van Haaren, Harry > > <harry.van.haa...@intel.com> wrote: > > > > > We need a fix for this issue. > > > > > > > > +1 > > > > > > > > Interestingly, Stephen patch that joins all pthreads at > > > > > rte_eal_cleanup [1] makes this issue disappear. > > > > > So my understanding is that we are missing a api (well, I could not > > > > > find a way) to synchronously stop service lcores. > > > > > > > > Maybe we can take that patch as a fix. I hate to see this segfault > > > > in the field. I need to figure out what I missed in my cleanup > > > > (probably missed a synchronization point). > > > > > > I haven't easily reproduced this yet - so I'll investigate a way to > > > reproduce with close to 100% rate, then we can identify the root cause > > > and actually get a clean fix. If you have pointers to reproduce easily, > > > please let me know. > > > > > > > ping. > > I want a fix in 20.05, or I will start considering how to drop this thing. > > Hi David, > > I have been attempting to reproduce, unfortunately without success. > > Attempted you suggested meson test approach (thanks for suggesting!), but > I haven't had a segfault with that approach (yet, and its done a lot of > iterations..)
I reproduced it on the first try, just now. Travis catches it every once in a while (look at the ovsrobot). For the reproduction, this is on my laptop (core i7-8650U), baremetal, no fancy stuff. FWIW, the cores are ruled by the "powersave" governor. I can see the frequency oscillates between 3.5GHz and 3.7Ghz while the max frequency is 4.2GHz. Travis runs virtual machines with 2 cores, and there must be quite some overprovisioning on those servers. We can expect some cycles being stolen or at least something happening on the various cores. > > I've made the service-cores unit tests delay before exit, in an attempt > to have them access previously rte_free()-ed memory, no luck to reproduce. Ok, let's forget about the segfault, what do you think of the backtrace I caught? A service lcore thread is still in the service loop. The master thread of the application is in the libc exiting code. This is what I get in all crashes. > > Thinking perhaps we need it on exit, I've also POCed a unit test that leaves > service cores active on exit on purpose, to try have them poll after exit, > still no luck. > > Simplifying the problem, and using hello-world sample app with a > rte_eal_cleaup() > call at the end also doesn't easily aggravate the problem. > > From code inspection, I agree there is an issue. It seems like a call to > rte_service_lcore_reset_all() from rte_service_finalize() is enough... > But without reproducing it is hard to have good confidence in a fix. You promised a doc update on the services API. Thanks. -- David Marchand
