Hi:
I need to collect application metrics which are counts (per unit of time eg:
minute) for certain events. There are two ways of doing this:
1. Create separate streams (using split stream etc) in the application
explicitly, then aggregate the counts in a window and save them. This mixes
metrics collection with application logic and making the application logic
complex.2. Use Flink metrics framework (counter, guage, etc) to save metrics
I have a very small test with 2 events but when I run the application the
counters are not getting saved (they show value 0) even though that part of the
code is being executed. I do see the numRecordsIn counters being updated from
the source operator. I've also tried incrementing the count by 10 (instead of
1) every time the function gets execute but still the counts remain 0.
Here is snippet of the code:
dataStream.map(new RichMapFunction<String, String>() {
protected Counter counter;
public void open(Configuration parameters) { counter
= getRuntimeContext().getMetricGroup().addGroup("test",
"split").counter("success"); } @Override
public String map(String value) throws Exception {
counter.inc(); return value; } });
As I mentioned, I do get the success metric count but the value is always 0,
even though the above map function was executed.
My questions are:
1. Are there any issues regarding counters being approximate ?2. If I want to
collect accurate counts, is it recommended to use counters or should I do it
explicitly (which is making the code too complex) ?3. Do counters participate
in flink's failure/checkpointing/recovery ?4. Is there any better way of
collecting application metric counts ?
Thanks
Mans
