Hi guys, I'm still seeking for a solution, I hope many face this problem in production. Any working solutions will help us.
On Sat, Feb 17, 2024 at 2:16 PM Karthick <ibmkarthickma...@gmail.com> wrote: > Dear Kafka Community, > > I am reaching out to seek your valuable feedback and insights on a > proposed solution we are considering for managing Kafka outages using > Cassandra. > > At our organization, we heavily rely on Kafka for real-time data > processing and messaging. However, like any technology, Kafka is > susceptible to occasional outages which can disrupt our operations and > impact our services. To mitigate the impact of such outages and ensure > continuity, we are exploring the possibility of leveraging Cassandra as a > backup solution. > > Our proposed approach involves storing messages in Cassandra during Kafka > outages. Subsequently, we plan to implement a scheduler that will read from > Cassandra and attempt to write these messages back into Kafka once it is > operational again. > > We believe that by adopting this strategy, we can achieve the following > benefits: > > 1. > > Improved Fault Tolerance: By having a backup mechanism in place, we > can reduce the risk of data loss and ensure continuity of operations during > Kafka outages. > 2. > > Enhanced Reliability: Cassandra's distributed architecture and > built-in replication features make it well-suited for storing data > reliably, even in the face of failures. > 3. > > Scalability: Both Cassandra and Kafka are designed to scale > horizontally, allowing us to handle increased loads seamlessly. > > Before proceeding further with this approach, we would greatly appreciate > any feedback, suggestions, or concerns from the community. Specifically, we > are interested in hearing about: > > - Potential challenges or drawbacks of using Cassandra as a backup > solution for Kafka outages. > - Best practices or recommendations for implementing such a backup > mechanism effectively. > - Any alternative approaches or technologies that we should consider? > > Your expertise and insights are invaluable to us, and we are eager to > learn from your experiences and perspectives. Please feel free to share > your thoughts or reach out to us with any questions or clarifications. > > Thank you for taking the time to consider our proposal, and we look > forward to hearing from you soon. > Thanks and regards, > Karthick >
