KafkaIO#commitOffsetsInFinalize[1] is likely what you want if you want to see Kafka's view of how the pipeline is consuming from it since the pipeline will ensure that offsets are committed as the pipeline has guaranteed to ingest the data.
I would suggested to use pipeline level concepts and metrics so have you considered looking at pipeline level metrics like: * PCollection elements processed/size instead of bytes-consumed-rate * watermark lag / processing time lag instead of records-lag Obviously if your trying to dig down into an existing problem then it sure does make sense to look at Kafka level metrics if pipeline level metrics are tell you that there is a problem in the part of the pipeline containing Kafka. 1: https://beam.apache.org/releases/javadoc/2.44.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#commitOffsetsInFinalize-- On Fri, Jan 20, 2023 at 8:28 AM Alexey Romanenko <aromanenko....@gmail.com> wrote: > IIRC, we don’t expose any Kafka Consumer metrics, so I’m afraid, that > there is no easy way to get them in a Beam pipeline. > > — > Alexey > > On 18 Jan 2023, at 21:43, Lydian <lydia...@gmail.com> wrote: > > Hi, > I know that Beam KafkaIO doesn't use the native kafka offset, and > therefore I cannot use kafka metrics directly. > > Wondering what would be the right way to expose those metrics of my > KafkaIO pipeline? > Things I am interested includes: > > - bytes-consumed-rate > - fetch-latency-avg > - records-lag > - commit-rate > > consumer lagWondering how people get these metrics or instead of doing > this? or we should just enable `commit_offset_in_finalize` and then use the > Kafka metrics directly? > > also wondering if there's anything to notice when enabling the > commit_offset_in_finalize? Thanks! > > Sincerely, > Lydian Lee > > >
