[
https://issues.apache.org/jira/browse/BEAM-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094947#comment-17094947
]
Jacob Ferriero commented on BEAM-9847:
--------------------------------------
This test seems to confirm it is not possible to emit outputs of processElement
call before it has completed.
[https://gist.github.com/jaketf/d3c2e70dde781bbb0ef1993446e34b71]
> Verify If Triggering allows emitting eager results when processing a single
> element in HL7v2IO.
> -----------------------------------------------------------------------------------------------
>
> Key: BEAM-9847
> URL: https://issues.apache.org/jira/browse/BEAM-9847
> Project: Beam
> Issue Type: Task
> Components: io-java-gcp
> Reporter: Jacob Ferriero
> Assignee: Jacob Ferriero
> Priority: Minor
>
> Due to the nature of the HL7v2 API, HL7v2IO.ListHL7v2Messages follows the
> pattern of [paginating through all ListMessages results in a single
> ProcessElement call.|#diff-9cd595f078378218ccc01ce7e19ca766R447]]
>
> Upon testing with customer against HL7v2 store with 350k messages we
> observed that the ListMessages transform was not outputting any elements
> "hanging" for a long time (which was assumed to be the single thread
> paginating through all the results).
>
> We added the following triggering in hopes that it would emit early results:
> {code:java}
> .apply(
> Window.<HL7v2Message>into(new GlobalWindows())
> .triggering(
> AfterWatermark.pastEndOfWindow()
> .withEarlyFirings(
> AfterProcessingTime.pastFirstElementInPane()
> .plusDelayOf(Duration.standardSeconds(1))))
> .discardingFiredPanes())
> {code}
> Our tests with this triggering seemed to indicate that it did not "hang" like
> the first test and seemed to output a more steady stream of elements.
> Reviewer states that bundles must be committed atomically so no output
> elements of a (single process element call) can proceed to downstream stages
> until all output elements for that process element call are ready.
> There may be other things at play here. Will seek to reproduce in a way that
> definitively confirms output elements can be eagerly output during the
> execution of a single process element call before it completes.
> CC: [~pabloem]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
