Thanks for the explanation. I think the question is that if we have disk
utilization in our environment, what is the use case for KIP-827? The disk
utilization in our environment can already do the job. Is there anything I
missed?
Thanks,
Cong
On Tue, May 31, 2022 at 2:57 AM Mickael Maison <mickael.mai...@gmail.com>
wrote:
> Hi Cong,
>
> Kafka does not expose disk utilization metrics. This is something you
> need to provide in your environment. You definitively should have a
> mechanism for exposing metrics from your Kafka broker hosts and you
> should absolutely monitor disk usage and have appropriate alerts.
>
> Thanks,
> Mickael
>
> On Thu, May 26, 2022 at 7:34 PM Jun Rao <j...@confluent.io.invalid> wrote:
> >
> > Hi, Igor,
> >
> > Thanks for the reply.
> >
> > I agree that this KIP could be useful for improving the tool for moving
> > data across disks. It would be useful to clarify on the main motivation
> of
> > the KIP. Also, DescribeLogDirsResponse already includes the size of each
> > partition on a disk. So, it seems that UsableBytes is redundant since
> it's
> > derivable.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 26, 2022 at 3:30 AM Igor Soarez <i...@soarez.me> wrote:
> >
> > > Hi,
> > >
> > > This can also be quite useful to make better use of existing
> functionality
> > > in the Kafka API — moving replicas between log directories via
> > > ALTER_REPLICA_LOG_DIRS. If usable space information is also available
> the
> > > caller can make better decisions using the same API. It means a more
> > > consistent way of interacting with Kafka to manage replicas locations
> > > within a broker without having to correlate Kafka metrics with
> information
> > > from the Kafka API.
> > >
> > > --
> > > Igor
> > >
> > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > Hi, Mickael,
> > > >
> > > > Thanks for the KIP. Since this is mostly for monitoring and
> alerting,
> > > > could we expose them as metrics instead of as part of the API? We
> already
> > > > have a size metric per log. Perhaps we could extend that to add
> > > used/total
> > > > metrics per disk?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> <rve...@confluent.io.invalid
> > > >
> > > > wrote:
> > > >
> > > >> Hello Mikael,
> > > >>
> > > >> Thanks for the KIP.
> > > >>
> > > >> I see that the API response contains some information about each
> > > partition.
> > > >> ```
> > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > >> "about": "The size of the log segments in this partition in
> bytes." }
> > > >> ```
> > > >> Can this be summed up to provide a used space in a `log.dir`
> > > >> This will also be specific to a `log.dir` (for the case where
> multiple
> > > >> log.dir are hosted on the same underlying device)
> > > >>
> > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> <cd...@confluent.io.invalid>
> > > >> wrote:
> > > >> >
> > > >> > Hey Mickael,
> > > >> >
> > > >> > Great KIP!
> > > >> >
> > > >> > I have one question:
> > > >> >
> > > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > > change
> > > >> > should not
> > > >> > significantly affect the latency of this API." and "That would
> allow
> > > to
> > > >> > easily validate whether disk operations (like a resize), or topic
> > > >> deletion
> > > >> > (log deletion only happen after a short delay) have completed." I
> > > wonder
> > > >> if
> > > >> > there is an existing metric/API that can allow administrators to
> > > >> determine
> > > >> > whether we need to resize? If administrators use this API to
> determine
> > > >> > whether we need a resize, would this API become a high-volume
> API? I
> > > >> > understand we don't want this API to be a high-volume one because
> the
> > > API
> > > >> > is already costly by returning `"name": "Topics"`.
> > > >> >
> > > >> > Cong
> > > >> >
> > > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > I wrote a small KIP to expose the total and usable space of
> logdirs
> > > >> > > via the DescribeLogDirs API:
> > > >> > >
> > > >> > >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > >> > >
> > > >> > > Please take a look and let me know if you have any feedback.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Mickael
> > > >> > >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best Regards,
> > > >> Raman Verma
> > > >>
> > >
>
