Minor clarification (since new segment appeared twice) : bq. before a new one is deleted.
The 'new one' (in the last sentence) would become old when another segment is created. Cheers On Tue, Jun 12, 2018 at 6:42 PM, Gwen Shapira <g...@confluent.io> wrote: > See below: > > On Mon, Jun 11, 2018 at 3:36 AM, Simon Cooper < > simon.coo...@featurespace.co.uk> wrote: > > > Hi, > > > > I've ben trying to work out the details of when exactly kafka log > segments > > get deleted for to the retention period, so it would be helpful if > someone > > could clarify the behaviour: > > > > > > * Is a segment only deleted when all messages in that segment have > > 'timed out', or are messages deleted within each segment? > > > > Kafka only deletes entire segments (except for compacted topics, which are > a different story) > > > > > * Does the server artificially limit the messages returned to clients > > to those within the retention period, even if they still exist in the > > segment file? > > > > Older messages can be read if the segment wasn't deleted yet. You can check > the "beginning of log" offset JMX metric to see what is the oldest offset > available to consumers on each partition. > > > > * Does the segment deletion happen when a new segment is created, or > > is it done as a separate operation by the log cleaner? > > > > Separate operation by log cleaner, but note that active segment is never > deleted so sometimes you are waiting for new segment to get created before > a new one is deleted. > > > > > > Thanks for the help! > > Simon Cooper > > > > > > -- > *Gwen Shapira* > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > <http://www.confluent.io/blog> >
