You need to consider your total partition count as you do this. After 30 days, assuming 1 partition per topic, you have 7200 partitions. Depending on how many brokers you have, this can start to be a problem. We just found an issue on one of our clusters that has over 70k partitions that there¹s now a problem with doing actions like a preferred replica election for all topics because the JSON object that gets written to the zookeeper node to trigger it is too large for Zookeeper¹s default 1 MB data size.
You also need to think about the number of open file handles. Even with no data, there will be open files for each topic. -Todd On 8/11/14, 2:19 PM, "Chen Wang" <chen.apache.s...@gmail.com> wrote: >Folks, >Is there any potential issue with creating 240 topics every day? Although >the retention of each topic is set to be 2 days, I am a little concerned >that since right now there is no delete topic api, the zookeepers might be >overloaded. >Thanks, >Chen
