shanthoosh commented on a change in pull request #918: SAMZA-2094: Implement
the StartpointVisitor for the KafkaSystemConsumer.
URL: https://github.com/apache/samza/pull/918#discussion_r266697388
##########
File path:
samza-kafka/src/main/java/org/apache/samza/system/kafka/KafkaSystemConsumer.java
##########
@@ -159,30 +179,16 @@ private void startSubscription() {
*/
void startConsumer() {
// set the offset for each TopicPartition
- if (topicPartitionsToOffset.size() <= 0) {
+ if (topicPartitionToStartpointMap.size() <= 0) {
LOG.error ("{}: Consumer is not subscribed to any SSPs", this);
}
- topicPartitionsToOffset.forEach((tp, startingOffsetString) -> {
- long startingOffset = Long.valueOf(startingOffsetString);
-
- try {
- synchronized (kafkaConsumer) {
- kafkaConsumer.seek(tp, startingOffset); // this value should already
be the 'upcoming' value
- }
- } catch (Exception e) {
- // all recoverable execptions are handled by the client.
- // if we get here there is nothing left to do but bail out.
- String msg =
- String.format("%s: Got Exception while seeking to %s for partition
%s", this, startingOffsetString, tp);
- LOG.error(msg, e);
- throw new SamzaException(msg, e);
- }
-
- LOG.info("{}: Changing consumer's starting offset for tp = {} to {}",
this, tp, startingOffsetString);
-
+ topicPartitionToStartpointMap.forEach((topicPartition, startpoint) -> {
+ Partition partition = new Partition(topicPartition.partition());
+ SystemStreamPartition systemStreamPartition = new
SystemStreamPartition(systemName, topicPartition.topic(), partition);
+ startpoint.apply(systemStreamPartition,
kafkaStartpointRegistrationHandler);
// add the partition to the proxy
- proxy.addTopicPartition(topicPartitionsToSSP.get(tp), startingOffset);
+ proxy.addTopicPartition(systemStreamPartition,
kafkaConsumer.position(topicPartition));
Review comment:
@cameronlee314 My changes in `KafkaSystemConsumer` were operating on
`KafkaConsumerProxy`. However, after SAMZA-2120, this was not the case. My
local changes of registering SSP with proxy in `RegistrationHandler` did not
apply cleanly with the recent master(Since the recent master had
`KafkaConsumerProxyFactory`). Updated the patch after re-implementing some of
my changes and now SSP registration with proxy happens in
`RegistrationHandler`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
