Hi Cong,
Maybe some people can do without this KIP.
But in many cases, especially around tooling and automation, it's
useful to be able to retrieve disk utilization values via the Kafka
API rather than interfacing with a metrics system.
Does that clarify the motivation?
Thanks,
Mickael
On Wed, Jun 1, 2022 at 7:10 PM Cong Ding <cd...@confluent.io.invalid> wrote:
>
> Thanks for the explanation. I think the question is that if we have disk
> utilization in our environment, what is the use case for KIP-827? The disk
> utilization in our environment can already do the job. Is there anything I
> missed?
>
> Thanks,
> Cong
>
> On Tue, May 31, 2022 at 2:57 AM Mickael Maison <mickael.mai...@gmail.com>
> wrote:
>
> > Hi Cong,
> >
> > Kafka does not expose disk utilization metrics. This is something you
> > need to provide in your environment. You definitively should have a
> > mechanism for exposing metrics from your Kafka broker hosts and you
> > should absolutely monitor disk usage and have appropriate alerts.
> >
> > Thanks,
> > Mickael
> >
> > On Thu, May 26, 2022 at 7:34 PM Jun Rao <j...@confluent.io.invalid> wrote:
> > >
> > > Hi, Igor,
> > >
> > > Thanks for the reply.
> > >
> > > I agree that this KIP could be useful for improving the tool for moving
> > > data across disks. It would be useful to clarify on the main motivation
> > of
> > > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > > partition on a disk. So, it seems that UsableBytes is redundant since
> > it's
> > > derivable.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez <i...@soarez.me> wrote:
> > >
> > > > Hi,
> > > >
> > > > This can also be quite useful to make better use of existing
> > functionality
> > > > in the Kafka API — moving replicas between log directories via
> > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available
> > the
> > > > caller can make better decisions using the same API. It means a more
> > > > consistent way of interacting with Kafka to manage replicas locations
> > > > within a broker without having to correlate Kafka metrics with
> > information
> > > > from the Kafka API.
> > > >
> > > > --
> > > > Igor
> > > >
> > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > Hi, Mickael,
> > > > >
> > > > > Thanks for the KIP. Since this is mostly for monitoring and
> > alerting,
> > > > > could we expose them as metrics instead of as part of the API? We
> > already
> > > > > have a size metric per log. Perhaps we could extend that to add
> > > > used/total
> > > > > metrics per disk?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > <rve...@confluent.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hello Mikael,
> > > > >>
> > > > >> Thanks for the KIP.
> > > > >>
> > > > >> I see that the API response contains some information about each
> > > > partition.
> > > > >> ```
> > > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > > >> "about": "The size of the log segments in this partition in
> > bytes." }
> > > > >> ```
> > > > >> Can this be summed up to provide a used space in a `log.dir`
> > > > >> This will also be specific to a `log.dir` (for the case where
> > multiple
> > > > >> log.dir are hosted on the same underlying device)
> > > > >>
> > > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> > <cd...@confluent.io.invalid>
> > > > >> wrote:
> > > > >> >
> > > > >> > Hey Mickael,
> > > > >> >
> > > > >> > Great KIP!
> > > > >> >
> > > > >> > I have one question:
> > > > >> >
> > > > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > > > change
> > > > >> > should not
> > > > >> > significantly affect the latency of this API." and "That would
> > allow
> > > > to
> > > > >> > easily validate whether disk operations (like a resize), or topic
> > > > >> deletion
> > > > >> > (log deletion only happen after a short delay) have completed." I
> > > > wonder
> > > > >> if
> > > > >> > there is an existing metric/API that can allow administrators to
> > > > >> determine
> > > > >> > whether we need to resize? If administrators use this API to
> > determine
> > > > >> > whether we need a resize, would this API become a high-volume
> > API? I
> > > > >> > understand we don't want this API to be a high-volume one because
> > the
> > > > API
> > > > >> > is already costly by returning `"name": "Topics"`.
> > > > >> >
> > > > >> > Cong
> > > > >> >
> > > > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi,
> > > > >> > >
> > > > >> > > I wrote a small KIP to expose the total and usable space of
> > logdirs
> > > > >> > > via the DescribeLogDirs API:
> > > > >> > >
> > > > >> > >
> > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > > >> > >
> > > > >> > > Please take a look and let me know if you have any feedback.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Mickael
> > > > >> > >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Best Regards,
> > > > >> Raman Verma
> > > > >>
> > > >
> >
