Hi Karthick, To share a few thoughts on this - I guess a lot depends on your actual setup / requirements, such as:
- The sort of potential outages you really need to tackle with this - The volume of data you have to continuously process - Constraints you may have in terms of messages that could be(come) duplicated or out-of-order in cases of outages. At least that will likely happen at the boundaries where you switch to / back from a secondary backup storage, and in the process of replaying data from that backup storage back to Kafka. For certain kinds of network outages for example, your backup solution could fail the same way as the access to Kafka - so you might not be able to connect to Cassandra either. At a previous job, we used a local on-disk storage buffer for specific (critical) producers that couldn't afford even short outages in the connection to Kafka. Producers would then replay buffered data from disk back to Kafka once it came back up. That is of course just 'local' handling (and limited by available disk-space), but it doesn't introduce a dependency on another external system as backup. If you are however certain to need another (external) system to serve as backup, why not just use another (fully separate) Kafka cluster as your backup, too? Using Kafka for this again seems only natural to me. Introducing a fully different technology seems like overkill when it's really just another message buffer you need - plus a different tech-stack can have its own requirements and challenges to maintain. And as added benefit, any tooling you may have to observe your actual Kafka cluster could be used in similar ways also for data in the backup cluster (even if that's ideally just empty). Best, Ralph On Fri, Mar 8, 2024 at 12:54 PM Karthick <ibmkarthickma...@gmail.com> wrote: > Hi guys, I'm still seeking for a solution, I hope many face this problem in > production. Any working solutions will help us. > > On Sat, Feb 17, 2024 at 2:16 PM Karthick <ibmkarthickma...@gmail.com> > wrote: > > > Dear Kafka Community, > > > > I am reaching out to seek your valuable feedback and insights on a > > proposed solution we are considering for managing Kafka outages using > > Cassandra. > > > > At our organization, we heavily rely on Kafka for real-time data > > processing and messaging. However, like any technology, Kafka is > > susceptible to occasional outages which can disrupt our operations and > > impact our services. To mitigate the impact of such outages and ensure > > continuity, we are exploring the possibility of leveraging Cassandra as a > > backup solution. > > > > Our proposed approach involves storing messages in Cassandra during Kafka > > outages. Subsequently, we plan to implement a scheduler that will read > from > > Cassandra and attempt to write these messages back into Kafka once it is > > operational again. > > > > We believe that by adopting this strategy, we can achieve the following > > benefits: > > > > 1. > > > > Improved Fault Tolerance: By having a backup mechanism in place, we > > can reduce the risk of data loss and ensure continuity of operations > during > > Kafka outages. > > 2. > > > > Enhanced Reliability: Cassandra's distributed architecture and > > built-in replication features make it well-suited for storing data > > reliably, even in the face of failures. > > 3. > > > > Scalability: Both Cassandra and Kafka are designed to scale > > horizontally, allowing us to handle increased loads seamlessly. > > > > Before proceeding further with this approach, we would greatly appreciate > > any feedback, suggestions, or concerns from the community. Specifically, > we > > are interested in hearing about: > > > > - Potential challenges or drawbacks of using Cassandra as a backup > > solution for Kafka outages. > > - Best practices or recommendations for implementing such a backup > > mechanism effectively. > > - Any alternative approaches or technologies that we should consider? > > > > Your expertise and insights are invaluable to us, and we are eager to > > learn from your experiences and perspectives. Please feel free to share > > your thoughts or reach out to us with any questions or clarifications. > > > > Thank you for taking the time to consider our proposal, and we look > > forward to hearing from you soon. > > Thanks and regards, > > Karthick > > >
