Hey Nick,
you can do the following for per task stats (this is kind of an workaround):
Create an Accumulator with the subtask index in the name, e.g.
int subtaskIndex = getRuntimeContext().getIndexOfThisSubtask();
IntCounter counter = getRuntimeContext().getIntCounter("counter-" +
subtaskIndex);
This way you have one accumulator per subtask.
The web interface will display the values as they are set (I’m not sure if it
is in yet). You can also gather the stats from the execution result, e.g.
ExecutionResult res = env.execute();
res.getAllAccumulatorResults();
You can furthermore add a custom Accumulator variant, which simple sets one
value if this is what you need.
Does this help?
In any case, I agree that it would be nice to expose a special API/accumulator
for this via the runtime context.
– Ufuk
> On 12 Nov 2015, at 11:55, Maximilian Michels <m...@apache.org> wrote:
>
> Hi Nick,
>
> I don't know if you have already come across the Rest Api. If not,
> please have a look here:
> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html
>
> I know that Christian Kreutzfeldt (cc) has been working on a
> monitoring service which uses Akka messages to query the JobManager on
> a job's status and accumulators. I'm wondering if you two could engage
> in any way.
>
> Cheers,
> Max
>
> On Wed, Nov 11, 2015 at 6:44 PM, Nick Dimiduk <ndimi...@gmail.com> wrote:
>> Hello,
>>
>> I'm interested in exposing metrics from my UDFs. I see FLINK-1501 exposes
>> task manager metrics via a UI; it would be nice to plug into the same
>> MetricRegistry to register my own (ie, gauges). I don't see this exposed via
>> runtime context. This did lead me to discovering the Accumulators API. This
>> looks more oriented to simple counts, which are summed across components of
>> a batch job. In my case, I'd like to expose details of my stream processing
>> vertices so that I can monitor their correctness and health re: runtime
>> decisions. For instance, referring back to my previous thread, I would like
>> to expose the number of filters loaded into my custom RichCoFlatMap so that
>> I can easily monitor this value.
>>
>> Thanks,
>> Nick
