[
https://issues.apache.org/jira/browse/FLINK-29270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Hilmi Al Fatih updated FLINK-29270:
--------------------------------------------
Description:
I did a very simple kafka-to-kafka pipeline with flink 1.14.4 with checkpoint.
I did 4 tests to confirm: for each old API
(FlinkKafkaConsumer&FlinkKafkaProducer) and new API (KafkaSource&KafkaSink), I
run AT_LEAST_ONCE and EXACLTY_ONCE. The only difference in the code is in the
settings of Semantic part.
However, the metrics shown by producer is not correct when doing EXACTLY_ONCE
semantics in both APIs.
* For FlinkKafkaProducer, the metrics seems to be restarted every checkpoint
* For KafkaSink, the rate shows half of the actual produced rate.
For at_least_once, they are all agree with the consuming rate.
To understand the situation, I also check the incoming rate metrics for both
source and sink topics. So here are the summary:
h3. Old API
h4. At Least Once
!image-2022-09-13-11-41-51-759.png|width=616,height=190!
h4. Exactly Once
!image-2022-09-13-11-42-45-297.png|width=616,height=192!
h3. New API (KafkaSource&KafkaSink)
h4. At Least Once
!image-2022-09-13-11-43-38-226.png|width=618,height=190!
h4. Exactly Once
!image-2022-09-13-11-43-50-441.png|width=620,height=189!
was:
I did a very simple kafka-to-kafka pipeline with flink 1.14.4 with checkpoint.
I did 4 tests to confirm: for each old API
(FlinkKafkaConsumer&FlinkKafkaProducer) and new API (KafkaSource&KafkaSink), I
run AT_LEAST_ONCE and EXACLTY_ONCE.
However, the metrics shown by producer is not correct when doing EXACTLY_ONCE
semantics in both APIs.
To understand the situation, I also check the incoming rate metrics for both
source and sink topics. So here are the summary:
h3. Old API
h4. At Least Once
!image-2022-09-13-11-41-51-759.png|width=616,height=190!
h4. Exactly Once
!image-2022-09-13-11-42-45-297.png|width=616,height=192!
h3. New API (KafkaSource&KafkaSink)
h4. At Least Once
!image-2022-09-13-11-43-38-226.png|width=618,height=190!
h4. Exactly Once
!image-2022-09-13-11-43-50-441.png|width=620,height=189!
> Wrong metrics kafka producer (FlinkKafkaProducer, KafkaSink) in EXACTLY_ONCE
> ----------------------------------------------------------------------------
>
> Key: FLINK-29270
> URL: https://issues.apache.org/jira/browse/FLINK-29270
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Runtime / Metrics
> Affects Versions: 1.14.4
> Reporter: Muhammad Hilmi Al Fatih
> Priority: Minor
> Attachments: image-2022-09-13-11-41-51-759.png,
> image-2022-09-13-11-42-45-297.png, image-2022-09-13-11-43-38-226.png,
> image-2022-09-13-11-43-50-441.png
>
>
> I did a very simple kafka-to-kafka pipeline with flink 1.14.4 with checkpoint.
> I did 4 tests to confirm: for each old API
> (FlinkKafkaConsumer&FlinkKafkaProducer) and new API (KafkaSource&KafkaSink),
> I run AT_LEAST_ONCE and EXACLTY_ONCE. The only difference in the code is in
> the settings of Semantic part.
> However, the metrics shown by producer is not correct when doing EXACTLY_ONCE
> semantics in both APIs.
> * For FlinkKafkaProducer, the metrics seems to be restarted every checkpoint
> * For KafkaSink, the rate shows half of the actual produced rate.
> For at_least_once, they are all agree with the consuming rate.
> To understand the situation, I also check the incoming rate metrics for both
> source and sink topics. So here are the summary:
> h3. Old API
> h4. At Least Once
> !image-2022-09-13-11-41-51-759.png|width=616,height=190!
> h4. Exactly Once
> !image-2022-09-13-11-42-45-297.png|width=616,height=192!
> h3. New API (KafkaSource&KafkaSink)
> h4. At Least Once
> !image-2022-09-13-11-43-38-226.png|width=618,height=190!
> h4. Exactly Once
> !image-2022-09-13-11-43-50-441.png|width=620,height=189!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
