shanthoosh commented on a change in pull request #918: SAMZA-2094: Implement
the StartpointVisitor for the KafkaSystemConsumer.
URL: https://github.com/apache/samza/pull/918#discussion_r257074119
##########
File path:
samza-kafka/src/main/java/org/apache/samza/system/kafka/KafkaSystemConsumer.java
##########
@@ -285,6 +321,27 @@ public void register(SystemStreamPartition
systemStreamPartition, String offset)
metrics.registerTopicAndPartition(toTopicAndPartition(tp));
}
+ @Override
+ public void register(SystemStreamPartition systemStreamPartition, Startpoint
startpoint) {
+ if (started.get()) {
+ String exceptionMessage = String.format("KafkaSystemConsumer: %s had
started. Registration of ssp: %s, startpoint: %s failed.", this,
systemStreamPartition, startpoint);
+ throw new SamzaException(exceptionMessage);
+ }
+
+ if (!Objects.equals(systemStreamPartition.getSystem(), systemName)) {
+ LOG.warn("{}: ignoring SSP {}, because this consumer's system doesn't
match.", this, systemStreamPartition);
+ return;
+ }
+
+ LOG.debug("Registering the ssp: {}, startpoint: {} with the consumer.",
systemStreamPartition, startpoint);
+
+ super.register(systemStreamPartition, startpoint);
+
+ TopicPartition topicPartition = toTopicPartition(systemStreamPartition);
+ topicPartitionToStartpointMap.put(topicPartition, startpoint);
Review comment:
I discussed this with you offline before about having a startpoint
comparator to choose the lowest startpoint when there're multiple startpoints
defined for a SSP.
Example: Broadcast streams. Task1 defines lower startpoint and Task-2
defines a higher startpoint for the SSP.
It's already tracked here: SAMZA-2088.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
