Todd, Yes I actually thought about that. My concern is that even a weeks topic partition(240*7*3 = 5040) is too many. Does linkedin have a good experience in using this many topics in your system?:-) Thanks, Chen
On Mon, Aug 11, 2014 at 9:02 PM, Todd Palino <tpal...@linkedin.com.invalid> wrote: > In order to delete topics, you need to shut down the entire cluster (all > brokers), delete the topics from Zookeeper, and delete the log files and > partition directory from the disk on the brokers. Then you can restart the > cluster. Assuming that you can take a periodic outage on your cluster, you > can do it this way. > > Reading what you’re intending to do in other parts of this thread, have > you considered setting up 1 week’s worth of topics with 3 day retention, > and having your producer and consumer rotate between them. That is, on > Sunday at 12:00 AM, you start with topic1, then proceed to topic2 at > 12:06, and so on. The next week, you loop around over exactly the same > topics, knowing that the retention settings have cleared out the old data. > > -Todd > > On 8/11/14, 4:45 PM, "Chen Wang" <chen.apache.s...@gmail.com> wrote: > > >Todd, > >I actually only intend to keep each topic valid for 3 days most. Each of > >our topic has 3 partitions, so its around 3*240*3 =2160 partitions. Since > >there is no api for deleting topic, i guess i could set up a cron job > >deleting the out dated topics(folders) from zookeeper.. > >do you know when the delete topic api will be available in kafka? > >Chen > > > > > >On Mon, Aug 11, 2014 at 3:47 PM, Todd Palino > ><tpal...@linkedin.com.invalid> > >wrote: > > > >> You need to consider your total partition count as you do this. After 30 > >> days, assuming 1 partition per topic, you have 7200 partitions. > >>Depending > >> on how many brokers you have, this can start to be a problem. We just > >> found an issue on one of our clusters that has over 70k partitions that > >> there¹s now a problem with doing actions like a preferred replica > >>election > >> for all topics because the JSON object that gets written to the > >>zookeeper > >> node to trigger it is too large for Zookeeper¹s default 1 MB data size. > >> > >> You also need to think about the number of open file handles. Even with > >>no > >> data, there will be open files for each topic. > >> > >> -Todd > >> > >> > >> On 8/11/14, 2:19 PM, "Chen Wang" <chen.apache.s...@gmail.com> wrote: > >> > >> >Folks, > >> >Is there any potential issue with creating 240 topics every day? > >>Although > >> >the retention of each topic is set to be 2 days, I am a little > >>concerned > >> >that since right now there is no delete topic api, the zookeepers > >>might be > >> >overloaded. > >> >Thanks, > >> >Chen > >> > >> > >
