Hi all, I would like to start a discussion for FIP-53: Log Scanner Offset
Management

https://cwiki.apache.org/confluence/spaces/FLUSS/pages/449282458/FIP-53+Log+Scanner+Offset+Management


Today a LogScanner has no server-side persistent consumption offset: if the
client crashes without checkpointing, the offset is lost and you can only
restart from earliest, latest or timestamp. FIP-53 gives the LogScanner
Kafka-style consumer-group semantics so that offsets are managed by the
server.

Scanners sharing a group id form a logical group with a single server-side
committed offset per group, so a new member resumes where a crashed one
left off. Two subscription modes are supported: a manual-assignment mode,
where the user subscribes to specific buckets and no rebalance happens; and
a group-managed mode, where the cluster assigns buckets to members and
member join/leave triggers an automatic rebalance, with each bucket
consumed by exactly one member. This is handled by a GroupCoordinator
running in-process on each TabletServer, with offsets and group metadata
persisted in an internal KV system table.

Looking forward to your feedback.

Reply via email to