Chris, consider using log compaction. Ryanne
On Tue, Nov 13, 2018, 3:17 PM Chris Toomey <ctoo...@gmail.com> wrote: > We're considering using GlobalKTables / KeyValueStores for locally caching > topic content in services. The topics would be compacted such that only the > latest key/value pair would exist for a given key. > > One question that's come up is how to determine, when bootstrapping the > app, when the cache has been populated with the latest content from the > topic (so we start with a "warm" cache). ReadOnlyKeyValueStore has > an approximateNumEntries() method that we could use to see how much we've > got, but trying to figure out how much there is in the topic looks much > more difficult -- the only way I can see via the APIs / code is to use an > AdminClient to get the topic partitions and then the KafkaConsumer to get > the end offsets for those. > > Does anyone have experience doing this kind of caching? How did you handle > the bootstrapping issue? > > Any thoughts on easier or better ways to determine when the cache is warm? > > thx, > Chris >
