Yes, IMHO, that is going to be way too many topics. Use a smaller number of topics, and embedded attributes like "tag" and "user" in the messages written to Kafka.
Phiilp ----------------------------------------- http://www.philipotoole.com On Friday, September 5, 2014 4:21 AM, Sharninder <sharnin...@gmail.com> wrote: I'm not really sure about your exact use-case but I don't think having a topic per user is very efficient. Deleting topics in kafka, at the moment, isn't really straightforward. You should rethink your date pipeline a bit. Also, just because kafka has the ability to store messages for a certain time, don't think of it as a data store. Kafka is a streaming system, think of it as a fast queue that gives you the ability to move your pointer back. -- Sharninder On Fri, Sep 5, 2014 at 4:27 PM, Aris Alexis <aris.alexis....@gmail.com> wrote: > Thanks for the reply. If I use it only for activity streams like twitter: > > I would want a topic for each #tag and a topic for each user and maybe > foreach city. Would that be too many topics or it doesn't matter since most > of them will be deleted in a specified interval. > > > > Best Regards, > Aris Giachnis > > > On Fri, Sep 5, 2014 at 6:57 AM, Sharninder <sharnin...@gmail.com> wrote: > > > Since you want all chats and mail history persisted all the time, I > > personally wouldn't recommend kafka for your requirement. Kafka is more > > suitable as a streaming system where events expire after a certain time. > > Look at something more general purpose like hbase for persisting data > > indefinitely. > > > > So, for example all activity streams can go into kafka from where > consumers > > will pick up messages to parse and put them to hbase or other clients. > > > > -- > > Sharninder > > > > > > > > > > > > On Fri, Sep 5, 2014 at 12:05 AM, Aris Alexis <snowboard...@gmail.com> > > wrote: > > > > > Hello, > > > > > > I am building a big web application that I want to be massively > scalable > > (I > > > am using cassandra and titan as a general db). > > > > > > I want to implement the following: > > > > > > real time web chat that is persisted so that user a in the future can > > > recall his chat with user b,c,d much like facebook. > > > mail like messages in the web application (not sure about this as it is > > > somewhat covered by the first one) > > > user activity streams > > > users subscribing to topics for example florida/musicevents > > > > > > Could i use kafka for this? can you recommend another technology maybe? > > > > > >
