Guozhang, Thanks for the reply. I figured it out after a while. Indeed, the global default time based retention was tripping me. I was using older data for testing and publishing messages with explicit timestamps. It took me a while to figure out what was happening because kafka-topics.sh does not display the parameters of a topic that have a default value. Thus, I was under the impression that the topic's data was being kept indefinitely.
It would be best if kaka-topic.sh displayed all configuration for a topic, or at least something as important as the retention period, even if the value comes from a global default. On Sun, Jan 22, 2017 at 3:29 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Note that for log retention, Kafka brokers have a global config that could > be applied to any topics, and topics themselves have a per-topic config > that can override the broker-level global config, you may want to check > both the broker configs as well as the topic configs (e.g. with the > kafka-topics command tool) to make sure that time-based retention is > properly set on both levels. >
