Hello Walker, thanks for the KIP. A few thoughts: 1) Have you considered just relying on the `KafkaStreams#metrics()` that includes embedded consumer metrics that have the committed offsets instead of adding a new API? Not advocating that this is a better approach but want to make sure we considered all options before we come to the "last resort" of adding new public interfaces.
2) The javadoc mentions "tasks assigned to this client", but the returned map is on partitions. I think we should make the javadoc and the return types consistent, either tasks or topic partitions. 3) In addition, if for 2) above we ended up with topic partitions, then would they include only external source topics, or also including internal repartition / changelog topics? I think including only external source topic partitions are not sufficient for your goal of tracking progress, but exposing internal topic names are also a big commitment here for future topology evolution. 4) For "tasksIdling", I'm wondering if we can make it more general, that the returned value is not just a boolean, but a TaskState that can be an enum of "created, restoring, running, idle, closing". This could help us in the future to track other things like restoration efficiency and rebalance efficiency etc. 5) We need to clarify how is "idling" being defined here: e.g. we can clearly state that a task is considered idle only if 1) lag is increasing, indicating that there are indeed new records arrived at source, while committed offset is not advancing, AND 2) produced offset (imagine we may have punctuations that generate new data to the output topic even if there's no input for a while) is not advancing either. Guozhang On Mon, Feb 22, 2021 at 3:11 PM Boyang Chen <reluctanthero...@gmail.com> wrote: > Thanks Walker for the proposed KIP! This should definitely empower KStream > users with better visibility. > > Meanwhile I got a couple of questions/suggestions: > > > 1. typo "repost/report" in the motivation section. > > 2. What offsets do we report when the task is under restoration or > rebalancing? > > 3. IIUC, we should clearly state that our reported metrics are based off > locally assigned tasks for each instance. > > 4. In the meantime, what’s our strategy to report tasks that are not local > to the instance? Users would normally try to monitor all the possible > tasks, and it’s unfortunate we couldn’t determine whether we have lost > tasks. My brainstorming was whether it makes sense for the leader instance > to report the task progress as -1 for all “supposed to be running” tasks, > so that on the metrics collector side it could catch any missing tasks. > > 5. It seems not clear how users should use `isTaskIdling`. Why not report a > map/set for idling tasks just as what we did for committed offsets? > > 6. Why do we use TopicPartition instead of TaskId as the key in the > returned map? > 7. Could we include some details in where we got the commit offsets for > each task? Is it through consumer offset fetch, or the stream processing > progress based on the records fetched? > > > On Mon, Feb 22, 2021 at 3:00 PM Walker Carlson <wcarl...@confluent.io> > wrote: > > > Hello all, > > > > I would like to start discussion on KIP-715. This kip aims to make it > > easier to monitor Kafka Streams progress by exposing the committed offset > > in a similar way as the consumer client does. > > > > Here is the KIP: https://cwiki.apache.org/confluence/x/aRRRCg > > > > Best, > > Walker > > > -- -- Guozhang
