Hi, Thanks for the extra information. So, there seems to be 2 separate issues here. I’ll go through them one by one.
I was also checking our Grafana and the metric we were using was "flink_taskmanager_job_task_operator_KafkaConsumer_records_lag_max", actually. "flink_taskmanager_job_task_operator_records_lag_max" seems to be new (with the attempt thingy). After looking at the code changes in FLINK-8419, this unfortunately is a accidental “break” in the scope of the metric. In 1.4.0, the Kafka-shipped metrics were exposed under the “KafkaConsumer” metrics group. After FLINK-8419, this was changed, as you observed. In 1.5.0, however, I think the metrics are exposed under both patterns. Now, with the fact that some subtasks are returning -Inf for ‘record-lag-max’: If I understood the metric semantics correctly, this metric represents the "max record lag across **partitions subscribed by a Kafka consumer client**. So, the only possibility that could think of causing this, is that either the subtask does not have any partitions assigned to it, or simply there is a bug with the Kafka client returning this value. Is it possible that you verify that all subtasks have a partition assigned to it? That should be possible by just checking the job status in the Web UI, and observe the numRecordsOut value for each source subtask. Cheers, Gordon On 13 June 2018 at 2:05:17 PM, Julio Biason (julio.bia...@azion.com) wrote: Hi Gordon, We have Kafka 0.10.1.1 running and use the flink-connector-kafka-0.10 driver. There are a bunch of flink_taskmanager_job_task_operator_* metrics, including some about the committed offset for each partition. It seems I have 4 different records_lag_max with different attempt_id, though, 3 with -Inf and 1 with a value -- which will give me some more understand of Prometheus to extract this properly. I was also checking our Grafana and the metric we were using was "flink_taskmanager_job_task_operator_KafkaConsumer_records_lag_max", actually. "flink_taskmanager_job_task_operator_records_lag_max" seems to be new (with the attempt thingy). On the "KafkaConsumer" front, but it only has the "commited_offset" for each partition. On Wed, Jun 13, 2018 at 5:41 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: Hi, Which Kafka version are you using? AFAIK, the only recent changes to Kafka connector metrics in the 1.4.x series would be FLINK-8419 [1]. The ‘records_lag_max’ metric is a Kafka-shipped metric simply forwarded from the internally used Kafka client, so nothing should have been affected. Do you see other metrics under the pattern of ‘flink_taskmanager_job_task_operator_*’? All Kafka-shipped metrics should still follow this pattern. If not, could you find the ‘records_lag_max’ metric (or any other Kafka-shipped metrics [2]) under the user scope ‘KafkaConsumer’? The above should provide more insight into what may be wrong here. - Gordon [1] https://issues.apache.org/jira/browse/FLINK-8419 [2] https://docs.confluent.io/current/kafka/monitoring.html#fetch-metrics On 12 June 2018 at 11:47:51 PM, Julio Biason (julio.bia...@azion.com) wrote: Hey guys, I just updated our Flink install from 1.4.0 to 1.4.2, but our Prometheus monitoring is not getting the current Kafka lag. After updating to 1.4.2 and making the symlink between opt/flink-metrics-prometheus-1.4.2.jar to lib/, I got the metrics back on Prometheus, but the most important one, flink_taskmanager_job_task_operator_records_lag_max is now returning -Inf. Did I miss something? -- Julio Biason, Sofware Engineer AZION | Deliver. Accelerate. Protect. Office: +55 51 3083 8101 | Mobile: +55 51 99907 0554 -- Julio Biason, Sofware Engineer AZION | Deliver. Accelerate. Protect. Office: +55 51 3083 8101 | Mobile: +55 51 99907 0554
