[
https://issues.apache.org/jira/browse/KAFKA-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456955#comment-17456955
]
Matthias J. Sax edited comment on KAFKA-13500 at 12/10/21, 8:32 AM:
--------------------------------------------------------------------
Not 100% sure: for individual configs, maybe. – But it won't solve the
monitoring issue. Users might still see that the "restore consumer" has a
non-zero lag if they monitor the consumer lag metric if there are lagging
standbys and we are not actually restoring state for active tasks.
I marked this ticket as "minor", so if we think it's not worth it (for now?)
for with me. But I don't think that our code would become (much) more
complicated and thus, it might be an easy win... (the main impact might be that
we would need to add a new `standby.producer` config prefix... so we might
technically need a KIP)
Personally still in favor to add a new consumer, but I leave it up to you.
Also, we can always make this change later on, too.
was (Author: mjsax):
Not 100% sure: for individual configs, maybe. – But it won't solve the
monitoring issue. Users might still see that the "restore consumer" has a
non-zero lag if they monitor the consumer lag metric if there a lagging
standbys and we are not actually restoring state for active tasks.
I marked this ticket as "minor", so if we think it's not worth it (for now?)
for with me. But I don't think that our code would become (much) more
complicated and thus, it might be an easy win... (the main impact might be that
we would need to add a new `standby.producer` config prefix... so we might
technically need a KIP)
Personally still in favor to add a new consumer, but I leave it up to you.
Also, we can always make this change later on, too.
> Consider adding a dedicated standby consumer
> --------------------------------------------
>
> Key: KAFKA-13500
> URL: https://issues.apache.org/jira/browse/KAFKA-13500
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Minor
>
> We currently use the restore consumer to recover state for active tasks and
> to maintain standby tasks during regular processing. This setup has a few
> disadvantages
> # During state recovery, we might want to apply different consumer configs
> compared to standby maintenance during regular processing.
> # It make monitoring confusing: because we never commit offsets for
> changelog topics, users can only monitor the client's "lag metric" to
> observer restore progress (without the need to register a restore listener).
> However, if they are interesting in a restore metric, during regular
> processing it would report the standby lag, which can be rather confusing.
> Because the restore consumer does not use consumer group management, it seems
> to be low overhead to actually use a third consumer, because there won't be
> any heartbeat thread.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
