Thanks, Kenny for confirming. Message updates I mean to say that for same document/message there will be updates coming in (for e.g. person details may change). As you mentioned using the proper key should make that happen so good on that.
On Thu, Dec 15, 2016 at 1:16 PM, Kenny Gorman <ke...@eventador.io> wrote: > A couple thoughts.. > > - If you plan on fetching old messages in a non-contiguous manner then > this may not be the best design. For instance, “give me messages from > mondays for the last 3 quarters” is better served with a database. But if > you want to say “give me messages from the last month until now” that works > great. > > - I am not sure what you mean by updating messages. You would need to have > some sort of key and push in new messages with that key. Then when you read > by key, the application should understand that the latest is the version it > should use. > > - Alternatively, you can consume to something like a DB and use SQL to > select what you want using regular SQL. We see this pattern a lot. > > - For storing messages indefinitely it’s mostly making sure the config > options are set appropriately and you have enough storage space. Set > replication to something that makes you comfortable, maybe take backups as > was mentioned. > > Hope this helps some > > Kenny Gorman > Founder > www.eventador.io > > > > On Dec 15, 2016, at 12:00 PM, Susheel Kumar <susheel2...@gmail.com> > wrote: > > > > Hello Folks, > > > > I am going thru an existing design where Kafka is planned to be utilised > in > > below manner > > > > > > 1. Messages will pushed to Kafka by producers > > 2. There will be updates to existing messages on ongoing basis. The > > expectation is that all the updates are consolidated in Kafka and the > > latest and greatest version/copy is kept > > 3. Consumers will read the messages from Kafka and push to Solr for > > ingestion purposes > > 4. There will be no purging/removal of messages since it is expected to > > replay the messages in the future and perform full-re-ingestion. So > > messages will be kept in Kafka for indefinite period similar to > database > > where data once stored remains there and can be used later in teh > future. > > > > > > Do you see any pitfalls / any issue with this design especially wrt to > > storing the messages indefinitely. > > > > > > Thanks, > > Susheel > >
