We saw some crashes in earlier versions when native handles in RocksDB (even for config option objects) were manually and too eagerly released.
Maybe you have a similar issue here? On Thu, Aug 25, 2016 at 6:27 PM, Gyula Fóra <gyula.f...@gmail.com> wrote: > Hi, > This seems to be a sneaky concurrency issue in our custom statebackend > implementation. > > I made some changes, will keep you posted. > > Cheers, > Gyula > > On Thu, Aug 25, 2016, 10:54 Gyula Fóra <gyula.f...@gmail.com> wrote: > > > Hi, > > > > Sure I am sending the TM logs in priv. > > > > Currently what I did was to bump the Rocks version to 4.9.0 let's see if > > that helps. > > > > Cheers, > > Gyula > > > > Till Rohrmann <trohrm...@apache.org> ezt írta (időpont: 2016. aug. 25., > > Cs, 10:35): > > > >> Hi Gyula, > >> > >> I haven't seen this problem before. Do you have the logs of the failed > TMs > >> so that we have some more context what was going on? > >> > >> Cheers, > >> Till > >> > >> On Thu, Aug 25, 2016 at 9:40 AM, Gyula Fóra <gyf...@apache.org> wrote: > >> > >> > Hi guys, > >> > > >> > For quite some time now we fairly frequently experience a task manager > >> > crashes around the time new streaming jobs are deployed. We use > RocksDB > >> > backend so this might be related. > >> > > >> > We tried changing the GC from G1 to CMS that didnt help. > >> > > >> > Yesterday for instance 6 task managers crashed one ofter the other > with > >> > similar errors: > >> > > >> > *** Error in `java': double free or corruption (!prev): > >> 0x00007fac0414d760 > >> > *** > >> > *** Error in `java': free(): invalid pointer: 0x00007f8dcc0026c0 *** > >> > *** Error in `java': double free or corruption (!prev): > >> 0x00007f15247f9a90 > >> > *** > >> > ... > >> > > >> > Does anyone have any clue what might cause this or how to debug? > >> > This is very a critical issue :( > >> > > >> > Cheers, > >> > Gyula > >> > > >> > > >
