cameronlee314 commented on a change in pull request #918: SAMZA-2094: Implement
the StartpointVisitor for the KafkaSystemConsumer.
URL: https://github.com/apache/samza/pull/918#discussion_r258302217
##########
File path:
samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala
##########
@@ -208,7 +223,11 @@ class SystemConsumers (
if (startpoint != null) {
consumer.register(systemStreamPartition, startpoint)
} else {
- consumer.register(systemStreamPartition, offset)
+ val existingOffset = sspToRegisteredOffsets.get(systemStreamPartition)
+ val systemAdmin =
systemAdmins.getSystemAdmin(systemStreamPartition.getSystem)
+ if (existingOffset == null ||
systemAdmin.offsetComparator(existingOffset, offset) > 0) {
+ sspToRegisteredOffsets.put(systemStreamPartition, offset)
Review comment:
It seems conceptually a bit odd to call `SystemConsumer.register` in
`SystemConsumers.start`. If `SystemConsumer.register` works with multiple calls
(so it just registers the offset from the latest call to `register`), then you
could call `register` here, and then you wouldn't need to call it in `start`.
Not sure if this is feasible, but I think it would make the flow a bit
cleaner, since you wouldn't have to worry about doing anything extra in `start`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
