Hi Miguel, I checked your code and it seems fine to me, so I would not suspect anything with your logic. The next thing I'd suggest to check if you have many cases where the same key gets deleted and then re-inserted (you can add some logging at the `put` and `delete` calls`).
Guozhang On Tue, Jan 4, 2022 at 6:31 PM Miguel González <miguel.gonza...@klar.mx> wrote: > Hello > > Here's my store definition and my transformer with the whole logic (I > included the transformer with issues and another version (V2) of the > transformer with some improvements that I believe are necessary) > > https://gist.github.com/magg/576bf3381c9c0501b9761b54e9d86375 > > Thanks > - Miguel > > > > > > On Tue, Jan 4, 2022 at 5:21 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Miguel, > > > > How is your kvStore being constructed? Could you paste the snippet of the > > related construction code, as well as the related iterating / deletion > code > > here? > > > > On Tue, Jan 4, 2022 at 2:25 PM Matthias J. Sax <mj...@apache.org> wrote: > > > > > Not 100% sure. From what you describe it should work as expected. > > > > > > It seems `delete()` does not delete the key from the store (ie, > RocksDB) > > > itself (for unknown reasons)? > > > > > > Are you closing all your iterators correctly? (More or less a wild > guess > > > at the moment.) > > > > > > Did you enable caching for the store? (Just to double check if it could > > > be caching related or not.) > > > > > > > > > -Matthias > > > > > > > > > On 12/24/21 11:08 AM, Miguel González wrote: > > > > Hello > > > > > > > > I'm using Kafka Streams and I have a transformer that uses > > > > a TimestampedKeyValueStore, I have a punctuator that is in charge of > > > > cleaning the store, > > > > > > > > Basically I'm iterating the store using kvStore.all() and deleting > the > > > keys > > > > based on some logic with kvStore.delete(key); > > > > > > > > I'm seeing the changelog topic for the store grow unbounded, I'm > seeing > > > > many values with null for the same keys... I think those are called > > > > tombstones right? but the punctuator is constantly doing the same > > thing > > > > trying to delete the same keys.. I see more tombstones being > inserted. > > > > > > > > Is this the expected behavior? If so, how can I correctly clean that > > > store? > > > > > > > > thanks > > > > - Miguel > > > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
