[
https://issues.apache.org/jira/browse/BEAM-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Ferriero resolved BEAM-9847.
----------------------------------
Fix Version/s: Not applicable
Resolution: Not A Bug
> Verify If Triggering allows emitting eager results when processing a single
> element in HL7v2IO.
> -----------------------------------------------------------------------------------------------
>
> Key: BEAM-9847
> URL: https://issues.apache.org/jira/browse/BEAM-9847
> Project: Beam
> Issue Type: Task
> Components: io-java-gcp
> Reporter: Jacob Ferriero
> Assignee: Jacob Ferriero
> Priority: Minor
> Fix For: Not applicable
>
>
> Due to the nature of the HL7v2 API, HL7v2IO.ListHL7v2Messages follows the
> pattern of [paginating through all ListMessages results in a single
> ProcessElement call.|#diff-9cd595f078378218ccc01ce7e19ca766R447]]
>
> Upon testing with customer against HL7v2 store with 350k messages we
> observed that the ListMessages transform was not outputting any elements
> "hanging" for a long time (which was assumed to be the single thread
> paginating through all the results).
>
> We added the following triggering in hopes that it would emit early results:
> {code:java}
> .apply(
> Window.<HL7v2Message>into(new GlobalWindows())
> .triggering(
> AfterWatermark.pastEndOfWindow()
> .withEarlyFirings(
> AfterProcessingTime.pastFirstElementInPane()
> .plusDelayOf(Duration.standardSeconds(1))))
> .discardingFiredPanes())
> {code}
> Our tests with this triggering seemed to indicate that it did not "hang" like
> the first test and seemed to output a more steady stream of elements.
> Reviewer states that bundles must be committed atomically so no output
> elements of a (single process element call) can proceed to downstream stages
> until all output elements for that process element call are ready.
> There may be other things at play here. Will seek to reproduce in a way that
> definitively confirms output elements can be eagerly output during the
> execution of a single process element call before it completes.
> CC: [~pabloem]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
