In order to delete topics, you need to shut down the entire cluster (all brokers), delete the topics from Zookeeper, and delete the log files and partition directory from the disk on the brokers. Then you can restart the cluster. Assuming that you can take a periodic outage on your cluster, you can do it this way.
Reading what you’re intending to do in other parts of this thread, have you considered setting up 1 week’s worth of topics with 3 day retention, and having your producer and consumer rotate between them. That is, on Sunday at 12:00 AM, you start with topic1, then proceed to topic2 at 12:06, and so on. The next week, you loop around over exactly the same topics, knowing that the retention settings have cleared out the old data. -Todd On 8/11/14, 4:45 PM, "Chen Wang" <chen.apache.s...@gmail.com> wrote: >Todd, >I actually only intend to keep each topic valid for 3 days most. Each of >our topic has 3 partitions, so its around 3*240*3 =2160 partitions. Since >there is no api for deleting topic, i guess i could set up a cron job >deleting the out dated topics(folders) from zookeeper.. >do you know when the delete topic api will be available in kafka? >Chen > > >On Mon, Aug 11, 2014 at 3:47 PM, Todd Palino ><tpal...@linkedin.com.invalid> >wrote: > >> You need to consider your total partition count as you do this. After 30 >> days, assuming 1 partition per topic, you have 7200 partitions. >>Depending >> on how many brokers you have, this can start to be a problem. We just >> found an issue on one of our clusters that has over 70k partitions that >> there¹s now a problem with doing actions like a preferred replica >>election >> for all topics because the JSON object that gets written to the >>zookeeper >> node to trigger it is too large for Zookeeper¹s default 1 MB data size. >> >> You also need to think about the number of open file handles. Even with >>no >> data, there will be open files for each topic. >> >> -Todd >> >> >> On 8/11/14, 2:19 PM, "Chen Wang" <chen.apache.s...@gmail.com> wrote: >> >> >Folks, >> >Is there any potential issue with creating 240 topics every day? >>Although >> >the retention of each topic is set to be 2 days, I am a little >>concerned >> >that since right now there is no delete topic api, the zookeepers >>might be >> >overloaded. >> >Thanks, >> >Chen >> >>
