We did some more analysis on why the disk utilisation is continuously increasing. Turns out it's the RocksDB WAL that's utilising most of the disk space. The LOG.old WAL files are not getting deleted. Ideally they should have been. RocksDB provides certain configuration for purging WAL files <https://github.com/facebook/rocksdb/wiki/basic-operations#purging-wal-files>. But I am not sure how to set these configs. Any help would be really appreciated. Just for reference, our Kafka brokers are on v0.10.0.1 and RocksDB version is 4.8.0.
On Mon, Mar 20, 2017 at 12:29 PM, Mahendra Kariya < mahendra.kar...@go-jek.com> wrote: > Hey Guozhang, > > Thanks a lot for these insights. We are facing the exact same problem as > Tianji. Our commit frequency is also quite high. We flush almost around 16K > messages per minute to Kafka at the end of the topology. > > Another issue that we are facing is that rocksdb is not deleting old data. > We have set the time window retention duration to 1 hour, but the disk size > is constantly increasing. Ideally, the disk utilisation graph should > plateau after some time. > > We would like to apply the config change suggestions that you have given. > But we are on Kafka 0.10.0.1. And from the docs, it seems > rocksdb.config.setter is not available for this version. Is there any > other way for us to configure rocksdb? > > Does Kafka 0.10.0.1 emit any rocksdb related metrics that we can monitor > and set up alerts on? > > > Thanks! > > >
