Especially since the same code will need to process all these messages. Topics are typically used to separate messages between apps, modules, etc. If all your apps consume all the topics, there's something wrong in the design.
Gwen On Fri, Sep 5, 2014 at 9:31 AM, Philip O'Toole <philip.oto...@yahoo.com.invalid> wrote: > Yes, IMHO, that is going to be way too many topics. Use a smaller number of > topics, and embedded attributes like "tag" and "user" in the messages written > to Kafka. > > Phiilp > > > ----------------------------------------- > http://www.philipotoole.com > > > On Friday, September 5, 2014 4:21 AM, Sharninder <sharnin...@gmail.com> wrote: > > > > I'm not really sure about your exact use-case but I don't think having a > topic per user is very efficient. Deleting topics in kafka, at the moment, > isn't really straightforward. You should rethink your date pipeline a bit. > > Also, just because kafka has the ability to store messages for a certain > time, don't think of it as a data store. Kafka is a streaming system, think > of it as a fast queue that gives you the ability to move your pointer back. > > -- > Sharninder > > > > > On Fri, Sep 5, 2014 at 4:27 PM, Aris Alexis <aris.alexis....@gmail.com> > wrote: > >> Thanks for the reply. If I use it only for activity streams like twitter: >> >> I would want a topic for each #tag and a topic for each user and maybe >> foreach city. Would that be too many topics or it doesn't matter since most >> of them will be deleted in a specified interval. >> >> >> >> Best Regards, >> Aris Giachnis >> >> >> On Fri, Sep 5, 2014 at 6:57 AM, Sharninder <sharnin...@gmail.com> wrote: >> >> > Since you want all chats and mail history persisted all the time, I >> > personally wouldn't recommend kafka for your requirement. Kafka is more >> > suitable as a streaming system where events expire after a certain time. >> > Look at something more general purpose like hbase for persisting data >> > indefinitely. >> > >> > So, for example all activity streams can go into kafka from where >> consumers >> > will pick up messages to parse and put them to hbase or other clients. >> > >> > -- >> > Sharninder >> > >> > >> > >> > >> > >> > On Fri, Sep 5, 2014 at 12:05 AM, Aris Alexis <snowboard...@gmail.com> >> > wrote: >> > >> > > Hello, >> > > >> > > I am building a big web application that I want to be massively >> scalable >> > (I >> > > am using cassandra and titan as a general db). >> > > >> > > I want to implement the following: >> > > >> > > real time web chat that is persisted so that user a in the future can >> > > recall his chat with user b,c,d much like facebook. >> > > mail like messages in the web application (not sure about this as it is >> > > somewhat covered by the first one) >> > > user activity streams >> > > users subscribing to topics for example florida/musicevents >> > > >> > > Could i use kafka for this? can you recommend another technology maybe? >> > > >> > >>
