navina commented on a change in pull request #8047:
URL: https://github.com/apache/pinot/pull/8047#discussion_r789190783
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionGroupConsumer.java
##########
@@ -39,4 +58,19 @@
*/
MessageBatch fetchMessages(StreamPartitionMsgOffset startOffset,
StreamPartitionMsgOffset endOffset, int timeoutMs)
throws TimeoutException;
+
+ /**
+ * Checkpoints the consumption state of the stream partition group in the
source
+ *
+ * This is useful in systems that require preserving consumption state on
the source in order to resume or replay
+ * consumption of data.
+ * The offset returned will be used for offset comparisons and persisted to
the ZK segment metadata. Hence, the
+ * returned value should be same or equivalent (in comparison) to the
lastOffset provided in the input.
+ *
+ * @param lastOffset checkpoint the stream at this offset (exclusive)
+ * @return Returns the offset that should be used as the next upcoming
offset for the stream partition group
+ */
+ default StreamPartitionMsgOffset checkpoint(StreamPartitionMsgOffset
lastOffset) {
Review comment:
What do you do if the source system returns a reference for this
checkpoint that should be used for resuming consumption at that point ? It has
to be persisted in ZK as a part of the end offset. This reference can be part
of the system-specific `StreamPartitionMsgOffset` representation. It should not
affect the ability to compare `StreamPartitionMsgOffset`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
