You are looking for a K-V store here, so the general answer is no. But Kafka does have an internal K-V store but only for consumer offsets. So there are some tricks we can play:
If your processing node is consuming from Kafka and the offset of a snapshot is the offset of a Kafka partition, you can simply commit the offsets to Kafka. Later on when the node restarts, you can fetch the committed offset and that will be what you want. If you snapshot offsets has nothing to do with Kafka, you may do the following: 1. produce your snapshot to the snapshot checkpoint topic and get back the offset of the produced message. 2. use a consumer to commit the offset returned by broker in step 1 using the node name as the group name for the checkpoint topic partition. 3. When restart, fetch the offset of using node name as group name and that offset is what you need. Thanks, Jiangjie (Becket) Qin On Tue, Aug 4, 2015 at 3:16 AM, Aki <jack-ak...@marmelandia.com> wrote: > I'd like to save a snapshot of a processing node's state in a compacted > kafka topic. A large number of nodes would save their snapshots in the same > partition. > > What is an efficient way for a (restarted) node to find the offset of its > latest snapshot? Using just Kafka (no database, local file, etc.), is there > a more efficient way than to consume the partition from the earliest > available offset (potentially reading a lot of snapshots of other nodes). > > Thanks!
