Thank you, Mickael. One more question: are you imaging these
tooling/automation to call this API at a very low frequency? since
high-frequency calls to this API are prohibitively expensive. Can you
give some examples of low-frequency call use cases? I can think of
some high-frequency call use cases which are valid in this case, but I
had a hard time coming up with low-frequency call use cases.
The one you give in the KIP is validating whether disk resize
operations have been completed. However, this looks like a
high-frequency call use case to me because we need to keep monitoring
disk usage before and after resizing.
Cong
On Fri, Jun 3, 2022 at 5:22 AM Mickael Maison <mickael.mai...@gmail.com> wrote:
>
> Hi Cong,
>
> Maybe some people can do without this KIP.
> But in many cases, especially around tooling and automation, it's
> useful to be able to retrieve disk utilization values via the Kafka
> API rather than interfacing with a metrics system.
>
> Does that clarify the motivation?
>
> Thanks,
> Mickael
>
> On Wed, Jun 1, 2022 at 7:10 PM Cong Ding <cd...@confluent.io.invalid> wrote:
> >
> > Thanks for the explanation. I think the question is that if we have disk
> > utilization in our environment, what is the use case for KIP-827? The disk
> > utilization in our environment can already do the job. Is there anything I
> > missed?
> >
> > Thanks,
> > Cong
> >
> > On Tue, May 31, 2022 at 2:57 AM Mickael Maison <mickael.mai...@gmail.com>
> > wrote:
> >
> > > Hi Cong,
> > >
> > > Kafka does not expose disk utilization metrics. This is something you
> > > need to provide in your environment. You definitively should have a
> > > mechanism for exposing metrics from your Kafka broker hosts and you
> > > should absolutely monitor disk usage and have appropriate alerts.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Thu, May 26, 2022 at 7:34 PM Jun Rao <j...@confluent.io.invalid> wrote:
> > > >
> > > > Hi, Igor,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > I agree that this KIP could be useful for improving the tool for moving
> > > > data across disks. It would be useful to clarify on the main motivation
> > > of
> > > > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > > > partition on a disk. So, it seems that UsableBytes is redundant since
> > > it's
> > > > derivable.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez <i...@soarez.me> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > This can also be quite useful to make better use of existing
> > > functionality
> > > > > in the Kafka API — moving replicas between log directories via
> > > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available
> > > the
> > > > > caller can make better decisions using the same API. It means a more
> > > > > consistent way of interacting with Kafka to manage replicas locations
> > > > > within a broker without having to correlate Kafka metrics with
> > > information
> > > > > from the Kafka API.
> > > > >
> > > > > --
> > > > > Igor
> > > > >
> > > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > > Hi, Mickael,
> > > > > >
> > > > > > Thanks for the KIP. Since this is mostly for monitoring and
> > > alerting,
> > > > > > could we expose them as metrics instead of as part of the API? We
> > > already
> > > > > > have a size metric per log. Perhaps we could extend that to add
> > > > > used/total
> > > > > > metrics per disk?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > <rve...@confluent.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello Mikael,
> > > > > >>
> > > > > >> Thanks for the KIP.
> > > > > >>
> > > > > >> I see that the API response contains some information about each
> > > > > partition.
> > > > > >> ```
> > > > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > > > >> "about": "The size of the log segments in this partition in
> > > bytes." }
> > > > > >> ```
> > > > > >> Can this be summed up to provide a used space in a `log.dir`
> > > > > >> This will also be specific to a `log.dir` (for the case where
> > > multiple
> > > > > >> log.dir are hosted on the same underlying device)
> > > > > >>
> > > > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> > > <cd...@confluent.io.invalid>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > Hey Mickael,
> > > > > >> >
> > > > > >> > Great KIP!
> > > > > >> >
> > > > > >> > I have one question:
> > > > > >> >
> > > > > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > > > > change
> > > > > >> > should not
> > > > > >> > significantly affect the latency of this API." and "That would
> > > allow
> > > > > to
> > > > > >> > easily validate whether disk operations (like a resize), or topic
> > > > > >> deletion
> > > > > >> > (log deletion only happen after a short delay) have completed." I
> > > > > wonder
> > > > > >> if
> > > > > >> > there is an existing metric/API that can allow administrators to
> > > > > >> determine
> > > > > >> > whether we need to resize? If administrators use this API to
> > > determine
> > > > > >> > whether we need a resize, would this API become a high-volume
> > > API? I
> > > > > >> > understand we don't want this API to be a high-volume one because
> > > the
> > > > > API
> > > > > >> > is already costly by returning `"name": "Topics"`.
> > > > > >> >
> > > > > >> > Cong
> > > > > >> >
> > > > > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi,
> > > > > >> > >
> > > > > >> > > I wrote a small KIP to expose the total and usable space of
> > > logdirs
> > > > > >> > > via the DescribeLogDirs API:
> > > > > >> > >
> > > > > >> > >
> > > > > >>
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > > > >> > >
> > > > > >> > > Please take a look and let me know if you have any feedback.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Mickael
> > > > > >> > >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Best Regards,
> > > > > >> Raman Verma
> > > > > >>
> > > > >
> > >
