The CPU/IO required to complete a compaction phase will grow as the log grows but you can manage this via the cleaner's various configs. Check out properties starting log.cleaner in the docs ( https://kafka.apache.org/documentation). All databases that implement LSM storage have a similar overhead (Cassandra, HBase, RocksDB etc). Note the first (active) segment is never compacted.
On Thu, Nov 30, 2017 at 6:59 AM Kane Kim <kane.ist...@gmail.com> wrote: > I want to confirm if kafka has to re-compact all log segments, as log grows > doesn't it become slower as well? > > On Tue, Nov 28, 2017 at 11:33 PM, Jakub Scholz <ja...@scholz.cz> wrote: > > > There is quite a nice section on this in the documentation - > > http://kafka.apache.org/documentation/#compaction ... I think it should > > answer your questions. > > > > On Wed, Nov 29, 2017 at 7:19 AM, Kane Kim <kane.ist...@gmail.com> wrote: > > > > > How does kafka log compaction work? > > > Does it compact all of the log files periodically against new changes? > > > > > >
