Hello Folks, I am going thru an existing design where Kafka is planned to be utilised in below manner
1. Messages will pushed to Kafka by producers 2. There will be updates to existing messages on ongoing basis. The expectation is that all the updates are consolidated in Kafka and the latest and greatest version/copy is kept 3. Consumers will read the messages from Kafka and push to Solr for ingestion purposes 4. There will be no purging/removal of messages since it is expected to replay the messages in the future and perform full-re-ingestion. So messages will be kept in Kafka for indefinite period similar to database where data once stored remains there and can be used later in teh future. Do you see any pitfalls / any issue with this design especially wrt to storing the messages indefinitely. Thanks, Susheel
