Robin,
I asked myself the same thing, and indeed there is no way of accessing the
metrics from within the pipeline itself. The
only access you can have is directly to the MetricCell like that (1) but it is
runner facing, it is not user facing, so
not helping for your test.So I agree with you: such a job handler would be
useful.
[1]
https://github.com/apache/beam/blob/6497b0b24888178441f05d79ab101dc40760388b/runners/core-java/src/main/java/org/apa
che/beam/runners/core/LateDataUtils.java#L95
Best,Etienne
Le mercredi 29 août 2018 à 10:43 -0700, Robin Qiu a écrit :
> Hi everyone,
>
> I am writing a test [1] for the support of @RequiresStableInput annotation in
> Java SDK [2]. In the test I need to have
> a ParDo make some side effects and then fail. Currently, the only way to
> persist the side effect during retries
> without depending on external states is using attempted metrics. However,
> attempted metrics cannot be accessed from
> within a DoFn. So I am wondering if we should make an improvement on this by,
> for example, exposing a job handler to
> DoFn.
>
> As Luke mentioned in another email thread, this improvement will also be
> useful since:
> * cancelling a pipeline from within the pipeline is useful
> * starting a new job against the existing runner from in a pipeline is useful
> * accessing attempted metrics to test DoFn's with side effects is useful for
> error handling testing
>
> What do you think? All suggestions are appreciated.
>
> Best,
> Robin
>
> [1] https://github.com/apache/beam/pull/6220
> [2]
> https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM/edit#
>
>
