I just want to send out a small update -- I decided to include the "- *consumed*" metrics in the KIP alongside the *"-produced"* metrics after all, for reasons I address in the paragraph I added at the end of the motivation section. Please let me know if you have any questions or concerns
Cheers, Sophie On Wed, Jun 1, 2022 at 1:38 AM Sophie Blee-Goldman <sop...@confluent.io> wrote: > Just a quick question: for filling the gap of sub-topology visibilities, >> would task-level produced metrics be sufficient? > > > If I understand your question correctly, you're asking whether we could > just report at the task/subtopology > level since we mainly want the bytes/throughput produced by the > subtopology itself? > > Note that since we're only reporting this metric at the sink nodes, it's > basically the same as a task-level metric > for any subtopology that has only one sink and only scales with the number > of output topics. If you're concerned > about a potential performance impact I would say that (a) making these > task-level doesn't buy us much, if anything, > and (b) we can always revisit this if our benchmarks do indeed reveal a > regression but for now let's not over- > optimize too much :) > > Also, the general philosophy behind the KAfka Streams metrics thus far > has been to report at the finest granularity > and allow users to roll them up into whatever scope they want to aggregate > over. This KIP adopts the same approach. > > On Tue, May 31, 2022 at 12:02 PM Guozhang Wang <wangg...@gmail.com> wrote: > >> Hi Sophie, >> >> Just a quick question: for filling the gap of sub-topology visibilities, >> would task-level produced metrics be sufficient? >> >> On Mon, May 30, 2022 at 10:59 AM Bill Bejeck <bbej...@gmail.com> wrote: >> >> > Thanks for the KIP Sophie. >> > >> > I'm in favor of this change as well. I don't have any comments in >> > addition to the ones already expressed. >> > >> > -Bill >> > >> > On Mon, May 30, 2022 at 4:55 AM Sagar <sagarmeansoc...@gmail.com> >> wrote: >> > >> > > Hi Sophie, >> > > >> > > A very minor comment but you might want to remove this KIP template >> > related >> > > information from the top of the KIP: >> > > >> > > *This page is meant as a template for writing a KIP >> > > < >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals >> > > >. >> > > To create a KIP choose Tools->Copy on this page and modify with your >> > > content and replace the heading with the next KIP number and a >> > description >> > > of your issue. Replace anything in italics with your own description.* >> > > >> > > >> > > Thanks! >> > > Sagar. >> > > >> > > On Mon, May 30, 2022 at 1:04 PM Sophie Blee-Goldman >> > > <sop...@confluent.io.invalid> wrote: >> > > >> > > > > >> > > > > Why does the title of the KIP talk about task-level metrics, but >> the >> > > > > specified metrics are on processor-level? >> > > > >> > > > >> > > > Ah, my mistake -- it should indeed say "processor-level metrics". >> > Thanks >> > > > for the catch Bruno, the title has been fixed. >> > > > >> > > > Since there don't seem to be any concerns I'll proceed with kicking >> off >> > > the >> > > > vote. Thanks all! >> > > > >> > > > On Mon, May 30, 2022 at 12:01 AM Bruno Cadonna <cado...@apache.org> >> > > wrote: >> > > > >> > > > > Thanks for the KIP, Sophie! >> > > > > >> > > > > I am also in favor of this KIP! >> > > > > >> > > > > I have one minor question: >> > > > > >> > > > > Why does the title of the KIP talk about task-level metrics, but >> the >> > > > > specified metrics are on processor-level? >> > > > > >> > > > > For the rest, I am +1. >> > > > > >> > > > > Best, >> > > > > Bruno >> > > > > >> > > > > On 29.05.22 00:20, John Roesler wrote: >> > > > > > Thanks for the well motivated and documented KIP, Sophie! I’m in >> > > favor >> > > > > of this change. >> > > > > > >> > > > > > -John >> > > > > > >> > > > > > On Sat, May 28, 2022, at 06:42, Sophie Blee-Goldman wrote: >> > > > > >> Hey all, >> > > > > >> >> > > > > >> I'd like to propose a very small KIP to add two metrics that >> will >> > > help >> > > > > fill >> > > > > >> a gap in the derivable produced and consumed metrics. Please >> take >> > a >> > > > look >> > > > > >> and reply here with any questions or concerns. >> > > > > >> >> > > > > >> KIP-846: Task-level Streams metrics for bytes/records Produced >> > > > > >> < >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211886093 >> > > > > > >> > > > > >> >> > > > > >> Given the small nature of this I'm going to call for a vote >> soon, >> > > but >> > > > > >> please don't hesitate to raise anything you feel should be >> > discussed >> > > > in >> > > > > >> more detail first. >> > > > > >> >> > > > > >> Thanks! >> > > > > >> Sophie >> > > > > >> > > > >> > > >> > >> >> >> -- >> -- Guozhang >> >
