Hi, I've tried:
set pig.notification.listener my.PigCounterProgressNotificationListener
set pig.notification.listener.arg
'my.PigCounterProgressNotificationListener';
With implementation:
@Override
void jobFinishedNotification(String scriptId, JobStats jobStats) {
println("Print jobStats.hadoopCounters: ")
jobStats.hadoopCounters.asList().forEach{it ->
println("hadoopCounters element: $it")
}
}
I don't see anything written to console... What do I do wrong?
2015-12-14 10:53 GMT+01:00 Serega Sheypak <[email protected]>:
> Hi, is there any possibility to dump all MR counters to console when pig
> script is finished?
> I also work with Scalding, they have cool feature. You can implement flow
> listener (flow ifs a kind of Pig execution DAG). And listen to "flow
> finished" event. Then you can get access to FlowStats object and get all
> counters from all stages.
> Counters with the same names will be merged (input records, output records
> and other default MR counters), but I don't care, I need to get all my
> user-defined counters. They have unique names.
>
>
> Is it possible to find something similar in pig? I don't want to iterate
> over 100500 jobs on my JobHistory server. :)
>
