mcvsubbu commented on a change in pull request #8047:
URL: https://github.com/apache/pinot/pull/8047#discussion_r789159074
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionGroupConsumer.java
##########
@@ -39,4 +58,19 @@
*/
MessageBatch fetchMessages(StreamPartitionMsgOffset startOffset,
StreamPartitionMsgOffset endOffset, int timeoutMs)
throws TimeoutException;
+
+ /**
+ * Checkpoints the consumption state of the stream partition group in the
source
+ *
+ * This is useful in systems that require preserving consumption state on
the source in order to resume or replay
Review comment:
Pinot preserves the state for all streams. The state is in the
StreamPartitionMsgOffset. I don't understand this comment.
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionGroupConsumer.java
##########
@@ -39,4 +58,19 @@
*/
MessageBatch fetchMessages(StreamPartitionMsgOffset startOffset,
StreamPartitionMsgOffset endOffset, int timeoutMs)
throws TimeoutException;
+
+ /**
+ * Checkpoints the consumption state of the stream partition group in the
source
+ *
+ * This is useful in systems that require preserving consumption state on
the source in order to resume or replay
+ * consumption of data.
+ * The offset returned will be used for offset comparisons and persisted to
the ZK segment metadata. Hence, the
Review comment:
Are the offsets returned for local comparison (within the server while
catching up, for example) different from offsets returned for comparisons with
zk segment metadata?
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionGroupConsumer.java
##########
@@ -39,4 +58,19 @@
*/
MessageBatch fetchMessages(StreamPartitionMsgOffset startOffset,
StreamPartitionMsgOffset endOffset, int timeoutMs)
throws TimeoutException;
+
+ /**
+ * Checkpoints the consumption state of the stream partition group in the
source
+ *
+ * This is useful in systems that require preserving consumption state on
the source in order to resume or replay
+ * consumption of data.
+ * The offset returned will be used for offset comparisons and persisted to
the ZK segment metadata. Hence, the
+ * returned value should be same or equivalent (in comparison) to the
lastOffset provided in the input.
+ *
+ * @param lastOffset checkpoint the stream at this offset (exclusive)
+ * @return Returns the offset that should be used as the next upcoming
offset for the stream partition group
+ */
+ default StreamPartitionMsgOffset checkpoint(StreamPartitionMsgOffset
lastOffset) {
Review comment:
If it is only a checkpoint (meaning saving the offset in some persistent
place, i suppose) then I am not clear why this method should not be a void.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
