Here is the gist with the two gif
https://gist.github.com/PierreCoquentin/d2df46e5e1c0d3506f6311b343e6f775
On Fri, Feb 10, 2017 at 7:45 AM, Guozhang Wang <wangg...@gmail.com> wrote:
> Pierre,
>
> Apache mailing list has some restricts to attach large attachments and I
> think that is why your gif files are not shown up. Could you try using a
> gist link?
>
> Guozhang
>
> On Wed, Feb 8, 2017 at 9:49 AM, Pierre Coquentin <
> pierre.coquen...@gmail.com
> > wrote:
>
> > Well, I am a little perplexed now... I have already recompiled the branch
> > 0.10.1 with rocksdb 4.11.2 and it doesn't seem better.
> > So I have modified the launcher of our jvm to use jemalloc with the
> > profile enabled and from the first result I have, it seems that the
> problem
> > comes from the method all()...
> > But I don't understand how the iterator could not be closed correctly
> with
> > a try-with-resources... Tomorrow, I will try by calling explicitly the
> > method close...
> > Attached a gif taken after a reboot of the jvm with the profiling
> enabled,
> > the second, 12 hours later, we clearly see a problem with the iterator
> and
> > the call of DBIter.SeekToFirst and DBIter.Next which both are called in
> the
> > method RocksDBStore.all().
> >
> >
> > On Wed, Feb 8, 2017 at 6:30 PM, Guozhang Wang <wangg...@gmail.com>
> wrote:
> >
> >> Hello Pierre,
> >>
> >> As Damian said your code looks fine and I cannot think of a direct
> reason
> >> for the rocksdb memory leak on top of my head.
> >>
> >> Could you build and try out the latest Kafka trunk (will be released as
> >> 0.10.2 in a few days) which contains a newer version of RocksDB and see
> if
> >> this issue still exists?
> >>
> >>
> >> Guozhang
> >>
> >> On Sat, Feb 4, 2017 at 11:54 AM, Damian Guy <damian....@gmail.com>
> wrote:
> >>
> >> > Looks fine
> >> >
> >> > On Sat, 4 Feb 2017 at 19:27, Pierre Coquentin <
> >> pierre.coquen...@gmail.com>
> >> > wrote:
> >> >
> >> > Oh ok, this is a snippet of the code we use :
> >> >
> >> > List<KeyValue<K, V>> keyValues = new ArrayList<>();
> >> > try (KeyValueIterator<K, V> iterator = kvStore.all()) {
> >> > iterator.forEachRemaining(keyValues::add);
> >> > }
> >> > // process all entries at once
> >> > try {
> >> > // business stuff
> >> > } catch (Exception e) {
> >> > // handling problem
> >> > } finally {
> >> > // delete them
> >> > keyValues.stream().map(item ->
> >> > item.key).forEach(kvStore::delete);
> >> > }
> >> >
> >> > Something missing ?
> >> >
> >> > On Sat, Feb 4, 2017 at 7:59 PM, Damian Guy <damian....@gmail.com>
> >> wrote:
> >> >
> >> > > Keeping the rocksdb iterator wouldn't cause a memory leak in the
> heap.
> >> > That
> >> > > is why i asked.
> >> > >
> >> > > On Sat, 4 Feb 2017 at 16:36 Pierre Coquentin <
> >> pierre.coquen...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > The iterator is inside a try-with-resources. And if the memory
> leak
> >> was
> >> > > > inside our code, we will see it using visualvm or jmap, and that's
> >> not
> >> > > the
> >> > > > case. This is not a memory leak in the heap. That's why my guess
> >> goes
> >> > > > directly to rocksdb.
> >> > > >
> >> > > > On Sat, Feb 4, 2017 at 5:31 PM, Damian Guy <damian....@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > > Hi Pierre,
> >> > > > >
> >> > > > > When you are iterating over the entries do you close the
> iterator
> >> > once
> >> > > > you
> >> > > > > are finished? If you don't then that will cause a memory leak.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Damian
> >> > > > >
> >> > > > > On Sat, 4 Feb 2017 at 16:18 Pierre Coquentin <
> >> > > pierre.coquen...@gmail.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > We ran a few tests with apache kafka 0.10.1.1.
> >> > > > > > We use a Topology with only one processor and a KVStore
> >> configured
> >> > as
> >> > > > > > persistent backed by rocksdb 4.9.0. Each events received are
> >> stored
> >> > > > using
> >> > > > > > the method put(key, value) and in the punctuate method, we
> >> iterate
> >> > > over
> >> > > > > all
> >> > > > > > entries with all(), processing them and deleting them with
> >> > > delete(key).
> >> > > > > > Then after a few days, the jvm crashed due to lack of memory.
> >> > > Visualvm
> >> > > > or
> >> > > > > > jmap doesn't show anything, so my guesses were there was a
> >> memory
> >> > > leak
> >> > > > in
> >> > > > > > rocksdb. We configured the KVStore to be in memory and as you
> >> can
> >> > see
> >> > > > in
> >> > > > > > picture 2, the memory is stable.
> >> > > > > > I didn't run the same test with a newer version of rocksdb
> yet.
> >> > > > > > Any thoughts ?
> >> > > > > > Regards,
> >> > > > > >
> >> > > > > > Pierre
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> >
>
>
> --
> -- Guozhang
>
