Hi Luke, Thanks for adding RemainingSegmentsToRecovery.
Another thought: different topics can have different segment sizes. I don't know how common it is, but it is possible. Some topics might want small segment sizes to more granular expiration of data. The downside of RemainingLogsToRecovery and RemainingSegmentsToRecovery is that the rate that they will decrement depends on the configuration and patterns of the topics and partitions and segment sizes. If someone is monitoring those metrics, they might see times where the metric decrements slowly, followed by a burst where it decrements quickly. What about RemainingBytesToRecovery? This would not depend on the configuration of the topic or of the data. It would actually be a pretty good metric, because I think that this metric would change at a constant rate (based on the disk I/O speed that the broker allocates to recovery). Because it changes at a constant rate, you would be able to use the rate-of-change to predict when it hits zero, which will let you know when the broker is going to start up. Like, I would imagine if we graphed RemainingBytesToRecovery that we'd see a fairly straight line that is decrementing at a steady rate towards zero. What do you think about adding RemainingBytesToRecovery? Or, what would you think about making the primary metric be RemainingBytesToRecovery, and getting rid of the others? I don't know if I personally would rather have all 3 metrics, or would just use RemainingBytesToRecovery. I'd too would like more community input on which of those metrics would be useful to people. About the JMX metrics, you said that if num.recovery.threads.per.data.dir=2, that there might be a separate RemainingSegmentsToRecovery counter for each thread. Is that actually how the data is structured within the Kafka recovery threads? Does each thread get a fixed set of partitions, or is there just one big pool of partitions that the threads all work on? As a more concrete example: * If I have 9 small partitions and 1 big partition, and num.recovery.threads.per.data.dir=2 Does each thread get 5 partitions, which means one thread will finish much sooner than the other? OR Do both threads just work on the set of 10 partitions, which means likely 1 thread will be busy with the big partition, while the other one ends up plowing through the 9 small partitions? If each thread gets assigned 5 partitions, then it would make sense that each thread has its own counter. If the threads works on a single pool of 10 partitions, then it would probably mean that the counter is on the pool of partitions itself, and not on each thread. -James > On May 4, 2022, at 5:55 AM, Luke Chen <show...@gmail.com> wrote: > > Hi devs, > > If there are no other comments, I'll start a vote tomorrow. > > Thank you. > Luke > > On Sun, May 1, 2022 at 5:08 PM Luke Chen <show...@gmail.com> wrote: > >> Hi James, >> >> Sorry for the late reply. >> >> Yes, this is a good point, to know how many segments to be recovered if >> there are some large partitions. >> I've updated the KIP, to add a `*RemainingSegmentsToRecover*` metric for >> each log recovery thread, to show the value. >> The example in the Proposed section here >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress#KIP831:Addmetricforlogrecoveryprogress-ProposedChanges> >> shows what it will look like. >> >> Thanks for the suggestion. >> >> Thank you. >> Luke >> >> >> >> On Sat, Apr 23, 2022 at 8:54 AM James Cheng <wushuja...@gmail.com> wrote: >> >>> The KIP describes RemainingLogsToRecovery, which seems to be the number >>> of partitions in each log.dir. >>> >>> We have some partitions which are much much larger than others. Those >>> large partitions have many many more segments than others. >>> >>> Is there a way the metric can reflect partition size? Could it be >>> RemainingSegmentsToRecover? Or even RemainingBytesToRecover? >>> >>> -James >>> >>> Sent from my iPhone >>> >>>> On Apr 20, 2022, at 2:01 AM, Luke Chen <show...@gmail.com> wrote: >>>> >>>> Hi all, >>>> >>>> I'd like to propose a KIP to expose a metric for log recovery progress. >>>> This metric would let the admins have a way to monitor the log recovery >>>> progress. >>>> Details can be found here: >>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress >>>> >>>> Any feedback is appreciated. >>>> >>>> Thank you. >>>> Luke >>> >>
