> If all topics share the same cache space, and we won't recycle space by topic. > So when some topics are rebalanced to other brokers, the previous broker > keeps its cache > and results in cache space wasting. > The topic shedding would be quite frequent under some cases, I am not sure > how badly it > will go, maybe we can add some metrics like valid cache space rate for > monitoring ?
My thought here is that, even if we make the load-shedding a frequent event, we need to optimize for steady state scenario, where the topic is running uninterruptely. Right now, we are the effectively "wasting" 99% of the cache space because we are forced to evict entries from the cache in order to not stress the JVM GC. I believe, temporarily wasting some cache space when a topic is closed in the broker will not be a problem and it will anyway get corrected soon when the cache rotates, and it will rotate faster depending on the write rate. Regarding the metrics for the "valid cache space", these will be very expensive to calculate and it would undermine any usefullness we can get out of it. Matteo
