1) As is there is no way to accomplish this.
2) Yes (datadog, graphite) but if you are happy with Prometheus I
instead recommend to fork the reporter and adjust it accordingly. Would
only custom prefixes/suffixes work for your use-case, or would a custom
label also work?
3) Sure, there's plenty of other metrics:
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#io
On 26/08/2021 16:46, gaurav kulkarni wrote:
Thanks for the response!
1) That's correct. I was wondering if there is any logical ID for task
manager that stays the same even if its moved to a different pod and
is a part of metrics emitted. The scenario is if a TM#1 was running on
pod#1 and then is moved to pod#2 and if we can correlate the metrics.
Please let me know if there is a way correlate metrics.
2) Are there any other reporters where it is possible?
3) I have one more question. Looks like most of the metrics emitted
are around resource usage (memory/cpu). Does flink emit any metrics
w.r.t job (for example, records processed, processing rate, overall
latency etc)?
Thanks,
Gaurav
On Thursday, August 26, 2021, 04:02:10 AM PDT, Chesnay Schepler
<ches...@apache.org> wrote:
1) Can you clarify what you mean with "same"? Are you referring to
something like an index, i.e., "This is TaskManager #2", without being
tied to a specific process?
2) It is not possible for the PrometheusReporter.
On 26/08/2021 04:22, gaurav kulkarni wrote:
Hi,
We have multiple flink clusters running in kubernetes. We plan to
enable prometheus on these clusters. Looks like flink metrics emitted
are of the format:
"flink_taskmanager_Status_JVM_GarbageCollector_G1_Young_Generation_Time{host="10_244_2_6",tm_id="10_244_2_6:6122_2e3d7a",}
65.0"
1. Since task managers can get moved across multiple pods, is there
is a way to uniquely identify metrics coming from the "same" task
manager but moved to different pods?
2. Is it possible to add formatting to the names of the metrics (for
example, add any suffix/prefix to the names).
Appreciate your help.
Thanks,
Gaurav
