Hi guys, I'm still seeking for a solution, I hope many face this problem in
production. Any working solutions will help us.
On Sat, Feb 17, 2024 at 2:16 PM Karthick wrote:
> Dear Kafka Community,
>
> I am reaching out to seek your valuable feedback and insights on a
> proposed solution we are considering for managing Kafka outages using
> Cassandra.
>
> At our organization, we heavily rely on Kafka for real-time data
> processing and messaging. However, like any technology, Kafka is
> susceptible to occasional outages which can disrupt our operations and
> impact our services. To mitigate the impact of such outages and ensure
> continuity, we are exploring the possibility of leveraging Cassandra as a
> backup solution.
>
> Our proposed approach involves storing messages in Cassandra during Kafka
> outages. Subsequently, we plan to implement a scheduler that will read from
> Cassandra and attempt to write these messages back into Kafka once it is
> operational again.
>
> We believe that by adopting this strategy, we can achieve the following
> benefits:
>
>1.
>
>Improved Fault Tolerance: By having a backup mechanism in place, we
>can reduce the risk of data loss and ensure continuity of operations during
>Kafka outages.
>2.
>
>Enhanced Reliability: Cassandra's distributed architecture and
>built-in replication features make it well-suited for storing data
>reliably, even in the face of failures.
>3.
>
>Scalability: Both Cassandra and Kafka are designed to scale
>horizontally, allowing us to handle increased loads seamlessly.
>
> Before proceeding further with this approach, we would greatly appreciate
> any feedback, suggestions, or concerns from the community. Specifically, we
> are interested in hearing about:
>
>- Potential challenges or drawbacks of using Cassandra as a backup
>solution for Kafka outages.
>- Best practices or recommendations for implementing such a backup
>mechanism effectively.
>- Any alternative approaches or technologies that we should consider?
>
> Your expertise and insights are invaluable to us, and we are eager to
> learn from your experiences and perspectives. Please feel free to share
> your thoughts or reach out to us with any questions or clarifications.
>
> Thank you for taking the time to consider our proposal, and we look
> forward to hearing from you soon.
> Thanks and regards,
> Karthick
>
