Hi Nam-Luc!
Having per-iteration statistics and accumulators is on the roadmap.
The way I have done this so far is to create accumulators like shown below,
which creates a new accumulator for each superstep:
class MyFunction extends RichMapFunction{
private LongCounter counter;
public
Are you running a fixed number of iterations or do you use a dynamic
termination criterion?
For fixed iterations, you can get the id of the current iteration ... which
allows you to find out when you are running the last iterations.
Would it be feasible for you to just log these statistics to the
Hi Ufuk,
The kind of things we'd like to log are: time spent in the iteration,
residual of the algorithm (convergence), current iteration.
Best regards,
Tran Nam-Luc
At Monday, 15/06/2015 on 16:15 Ufuk Celebi wrote:
Hey Tran Nam-Luc,
there is currently no way to do this.
The iteration sync
Hey Tran Nam-Luc,
there is currently no way to do this.
The iteration sync tasks keeps track of iteration convergence/max number of
iterations and signals termination to the iteration head. After this, the head
flushes the produced result to the next task (after the iteration) and the
intermed
Hello Everyone,
I would like to log certain stats during iterations in a bulk
iterative job. The way I do this is store the things I want at each
iteration and plan to flush everything to HDFS once all the iterations
are done. To do that I would need to know when the last iteration is
invoked in o
