[
https://issues.apache.org/jira/browse/KAFKA-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma resolved KAFKA-2871.
--------------------------------
Resolution: Duplicate
This was changed as part of KIP-33.
> Newly replicated brokers don't expire log segments properly
> -----------------------------------------------------------
>
> Key: KAFKA-2871
> URL: https://issues.apache.org/jira/browse/KAFKA-2871
> Project: Kafka
> Issue Type: Bug
> Components: replication
> Affects Versions: 0.8.2.1
> Reporter: Evan Huus
> Assignee: Neha Narkhede
> Priority: Minor
>
> We recently brought up a few brokers to replace some existing nodes, and used
> the provided script to reassign partitions from the retired nodes to the new
> ones, one at a time.
> A little while after the fact, we noticed extreme disk usage on the new
> nodes. Tracked this down to the fact that the replicated segments are all
> timestamped from the moment of replication rather than using whatever
> timestamp was set on the original node. Since this is the timestamp the log
> roller uses, it takes a full week (rollover time) before any data is purged
> from the new brokers.
> In the short term, what is the safest workaround? Can we just `rm` these old
> segments, or should we be messing with the filesystem metadata so kafka
> removes them itself?
> In the longer term, the partition mover should be setting timestamps
> appropriately on the segments it moves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
