[
https://issues.apache.org/jira/browse/KAFKA-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106431#comment-14106431
]
Jay Kreps commented on KAFKA-1489:
----------------------------------
Yeah I get what you want here, I think you basically want to say "keep as much
data as you can, only throw away when you are low on space". As I mentioned
this is a little weird in that retention among replicas for a partition may be
somewhat different from machine to machine depending on the distribution of
data on each replica, but this may be okay.
So maybe what you are looking for is something like
max.total.disk.space.bytes=12345
disk.full.discard.policy={oldest, largest, none}
(we can improve the names).
Probably the total disk space config should be per-data directory (since you
may have multiple volumes and you don't want any of them to get full).
Let me know if that sounds right.
> Global threshold on data retention size
> ---------------------------------------
>
> Key: KAFKA-1489
> URL: https://issues.apache.org/jira/browse/KAFKA-1489
> Project: Kafka
> Issue Type: New Feature
> Components: log
> Affects Versions: 0.8.1.1
> Reporter: Andras Sereny
> Assignee: Jay Kreps
> Labels: newbie
>
> Currently, Kafka has per topic settings to control the size of one single log
> (log.retention.bytes). With lots of topics of different volume and as they
> grow in number, it could become tedious to maintain topic level settings
> applying to a single log.
> Often, a chunk of disk space is dedicated to Kafka that hosts all logs
> stored, so it'd make sense to have a configurable threshold to control how
> much space *all* data in Kafka can take up.
> See also:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201406.mbox/browser
> http://mail-archives.apache.org/mod_mbox/kafka-users/201311.mbox/%3c20131107015125.gc9...@jkoshy-ld.linkedin.biz%3E
--
This message was sent by Atlassian JIRA
(v6.2#6252)
