Hi Cong,
Kafka does not expose disk utilization metrics. This is something you
need to provide in your environment. You definitively should have a
mechanism for exposing metrics from your Kafka broker hosts and you
should absolutely monitor disk usage and have appropriate alerts.
Thanks,
Mickael
On Thu, May 26, 2022 at 7:34 PM Jun Rao <j...@confluent.io.invalid> wrote:
>
> Hi, Igor,
>
> Thanks for the reply.
>
> I agree that this KIP could be useful for improving the tool for moving
> data across disks. It would be useful to clarify on the main motivation of
> the KIP. Also, DescribeLogDirsResponse already includes the size of each
> partition on a disk. So, it seems that UsableBytes is redundant since it's
> derivable.
>
> Thanks,
>
> Jun
>
> On Thu, May 26, 2022 at 3:30 AM Igor Soarez <i...@soarez.me> wrote:
>
> > Hi,
> >
> > This can also be quite useful to make better use of existing functionality
> > in the Kafka API — moving replicas between log directories via
> > ALTER_REPLICA_LOG_DIRS. If usable space information is also available the
> > caller can make better decisions using the same API. It means a more
> > consistent way of interacting with Kafka to manage replicas locations
> > within a broker without having to correlate Kafka metrics with information
> > from the Kafka API.
> >
> > --
> > Igor
> >
> > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > Hi, Mickael,
> > >
> > > Thanks for the KIP. Since this is mostly for monitoring and alerting,
> > > could we expose them as metrics instead of as part of the API? We already
> > > have a size metric per log. Perhaps we could extend that to add
> > used/total
> > > metrics per disk?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, May 19, 2022 at 10:21 PM Raman Verma <rve...@confluent.io.invalid
> > >
> > > wrote:
> > >
> > >> Hello Mikael,
> > >>
> > >> Thanks for the KIP.
> > >>
> > >> I see that the API response contains some information about each
> > partition.
> > >> ```
> > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > >> "about": "The size of the log segments in this partition in bytes." }
> > >> ```
> > >> Can this be summed up to provide a used space in a `log.dir`
> > >> This will also be specific to a `log.dir` (for the case where multiple
> > >> log.dir are hosted on the same underlying device)
> > >>
> > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding <cd...@confluent.io.invalid>
> > >> wrote:
> > >> >
> > >> > Hey Mickael,
> > >> >
> > >> > Great KIP!
> > >> >
> > >> > I have one question:
> > >> >
> > >> > You mentioned "DescribeLogDirs is usually a low volume API. This
> > change
> > >> > should not
> > >> > significantly affect the latency of this API." and "That would allow
> > to
> > >> > easily validate whether disk operations (like a resize), or topic
> > >> deletion
> > >> > (log deletion only happen after a short delay) have completed." I
> > wonder
> > >> if
> > >> > there is an existing metric/API that can allow administrators to
> > >> determine
> > >> > whether we need to resize? If administrators use this API to determine
> > >> > whether we need a resize, would this API become a high-volume API? I
> > >> > understand we don't want this API to be a high-volume one because the
> > API
> > >> > is already costly by returning `"name": "Topics"`.
> > >> >
> > >> > Cong
> > >> >
> > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I wrote a small KIP to expose the total and usable space of logdirs
> > >> > > via the DescribeLogDirs API:
> > >> > >
> > >> > >
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > >> > >
> > >> > > Please take a look and let me know if you have any feedback.
> > >> > >
> > >> > > Thanks,
> > >> > > Mickael
> > >> > >
> > >>
> > >>
> > >>
> > >> --
> > >> Best Regards,
> > >> Raman Verma
> > >>
> >
