Hi Yun, Thank you for pointing that out. In our production landscapes with live customers, we have 10 second checkpoint interval and 7MB of average checkpoint size. We do incremental checkpoints. If we keep the checkpoint interval longer (i.e. 1 minute) then the kafka consumer lag starts increasing. The reason is that over the period of 1 minute, the checkpoint size grows and the job takes long time to do the checkpoint and as a result kafka consumer lag for our live traffic goes high. In order to keep checkpoint size small, we tried 10 second option which is working out well and our kafka lag never exceeds beyond 20 messages on average. But i agree with you that 10 second option does not feel right and is too frequent in my opinion.
Do you have any recommendations for checkpointing interval please ? Best Regards, On Tue, 28 Jan 2020 at 07:46, Yun Tang <myas...@live.com> wrote: > Hi Ahmad > > Apart from setting the logger level of RocksDB, I also wonder why you > would meet rocksdb checkpoint IO logs were filling up disk space very very > quickly. How larger the local checkpoint state is and how long the > checkpoint interval is? I think you might give a too short interval of > checkpoint, even you could avoid to record too many logs, and I don't think > current checkpoint configuration is appropriate. > > Best > Yun Tang > ------------------------------ > *From:* Ahmad Hassan <ahmad.has...@gmail.com> > *Sent:* Monday, January 27, 2020 20:22 > *To:* user <user@flink.apache.org> > *Subject:* Re: Flink RocksDB logs filling up disk space > > > Thanks Chesnay! > > On Mon, 27 Jan 2020 at 11:29, Chesnay Schepler <ches...@apache.org> wrote: > > Please see https://issues.apache.org/jira/browse/FLINK-15068 > > On 27/01/2020 12:22, Ahmad Hassan wrote: > > Hello, > > In our production systems, we see that flink rocksdb checkpoint IO logs > are filling up disk space very very quickly in the order of GB's as the > logging is very verbose. How do we disable or suppress these logs please ? > The rocksdb file checkpoint.cc is dumping huge amount of checkpoint logs > like > > Log(db_options.info_log, "Hard Linking %s", src_fname.c_str()); > > > Best Regards, > > >
