Hi, Mickael,
Thanks for the explanation. The KIP looks to me now.
Jun
On Tue, May 31, 2022 at 6:44 AM Mickael Maison <mickael.mai...@gmail.com>
wrote:
> Hi Jun,
>
> Igor answered to your question.
> Users should rely on their host metrics to monitor disk usage. But
> with tooling and automation it's sometimes not ideal to retrieve
> values from metrics. So exposing disk usage via the Kafka API will
> simplify coordinating disk operations.
>
> I've updated the motivation section to make it clear why being able to
> retrieve these values via the API rather than metrics is interesting.
>
> Thanks,
> Mickael
>
> On Tue, May 31, 2022 at 3:38 PM Mickael Maison <mickael.mai...@gmail.com>
> wrote:
> >
> > Hi Raman,
> >
> > Unfortunately the replica size only includes the log files and it does
> > not include indexes or other metadata files. Obviously any extra non
> > Kafka files are also not included either.
> >
> > For these reasons, I decided to have a separate field with the actual
> > usable space reported by the volume.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Tue, May 31, 2022 at 11:57 AM Mickael Maison
> > <mickael.mai...@gmail.com> wrote:
> > >
> > > Hi Cong,
> > >
> > > Kafka does not expose disk utilization metrics. This is something you
> > > need to provide in your environment. You definitively should have a
> > > mechanism for exposing metrics from your Kafka broker hosts and you
> > > should absolutely monitor disk usage and have appropriate alerts.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Thu, May 26, 2022 at 7:34 PM Jun Rao <j...@confluent.io.invalid>
> wrote:
> > > >
> > > > Hi, Igor,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > I agree that this KIP could be useful for improving the tool for
> moving
> > > > data across disks. It would be useful to clarify on the main
> motivation of
> > > > the KIP. Also, DescribeLogDirsResponse already includes the size of
> each
> > > > partition on a disk. So, it seems that UsableBytes is redundant
> since it's
> > > > derivable.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez <i...@soarez.me> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > This can also be quite useful to make better use of existing
> functionality
> > > > > in the Kafka API — moving replicas between log directories via
> > > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also
> available the
> > > > > caller can make better decisions using the same API. It means a
> more
> > > > > consistent way of interacting with Kafka to manage replicas
> locations
> > > > > within a broker without having to correlate Kafka metrics with
> information
> > > > > from the Kafka API.
> > > > >
> > > > > --
> > > > > Igor
> > > > >
> > > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > > Hi, Mickael,
> > > > > >
> > > > > > Thanks for the KIP. Since this is mostly for monitoring and
> alerting,
> > > > > > could we expose them as metrics instead of as part of the API?
> We already
> > > > > > have a size metric per log. Perhaps we could extend that to add
> > > > > used/total
> > > > > > metrics per disk?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> <rve...@confluent.io.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello Mikael,
> > > > > >>
> > > > > >> Thanks for the KIP.
> > > > > >>
> > > > > >> I see that the API response contains some information about each
> > > > > partition.
> > > > > >> ```
> > > > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > > > >> "about": "The size of the log segments in this partition in
> bytes." }
> > > > > >> ```
> > > > > >> Can this be summed up to provide a used space in a `log.dir`
> > > > > >> This will also be specific to a `log.dir` (for the case where
> multiple
> > > > > >> log.dir are hosted on the same underlying device)
> > > > > >>
> > > > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> <cd...@confluent.io.invalid>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > Hey Mickael,
> > > > > >> >
> > > > > >> > Great KIP!
> > > > > >> >
> > > > > >> > I have one question:
> > > > > >> >
> > > > > >> > You mentioned "DescribeLogDirs is usually a low volume API.
> This
> > > > > change
> > > > > >> > should not
> > > > > >> > significantly affect the latency of this API." and "That
> would allow
> > > > > to
> > > > > >> > easily validate whether disk operations (like a resize), or
> topic
> > > > > >> deletion
> > > > > >> > (log deletion only happen after a short delay) have
> completed." I
> > > > > wonder
> > > > > >> if
> > > > > >> > there is an existing metric/API that can allow administrators
> to
> > > > > >> determine
> > > > > >> > whether we need to resize? If administrators use this API to
> determine
> > > > > >> > whether we need a resize, would this API become a high-volume
> API? I
> > > > > >> > understand we don't want this API to be a high-volume one
> because the
> > > > > API
> > > > > >> > is already costly by returning `"name": "Topics"`.
> > > > > >> >
> > > > > >> > Cong
> > > > > >> >
> > > > > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi,
> > > > > >> > >
> > > > > >> > > I wrote a small KIP to expose the total and usable space of
> logdirs
> > > > > >> > > via the DescribeLogDirs API:
> > > > > >> > >
> > > > > >> > >
> > > > > >>
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > > > >> > >
> > > > > >> > > Please take a look and let me know if you have any feedback.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Mickael
> > > > > >> > >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Best Regards,
> > > > > >> Raman Verma
> > > > > >>
> > > > >
>
