> I'm not sure what HDFS backups would bring
I’m not sure how realistic the threat is, but I was thinking a case in which
bug in Kafka corrupts the log files. I would personally sleep better knowing
there is a Kafka-independent backup of all data.
> how would you recover from e.g. all Kafka n
Good points.
I would backup all partitions to HDFS (etc.), as fast as the data arrives. In
case the Kafka becomes corrupted, the topics can be repopulated from the
backup. In my case, all clients track their own offsets, so they should in
theory be able to continue as if nothing had happened.
ion to terabytes-scale. But don’t
take my word for it, as I have not tried it.
Cheers, Giidox
> On 09 Mar 2016, at 15:10, Daniel Schierbeck wrote:
>
> I'm considering an architecture where Kafka acts as the primary datastore,
> with infinite retention of messages. The messag
available offset?
Best regards,
Giidox
way to get a callback or exception for seeking to an
invalid offset? Is there a way to discover what the valid offset range is, with
the new consumer?
Is the old consumer API deprecated (or is it planned to be deprecated)?
Any help is much appreciated.
- Giidox
