On Thu, Feb 2, 2023 at 1:56 PM Kaggal, Vinod C. (Vinod C.), M.S. via user < user@beam.apache.org> wrote:
> Hello! Thank you for all the hard work on implementing these useful > libraries. > > > > *Background:* We have been using Apache Storm in production for some time > (over 8 years) and have recently switched over to Beam. One of the > topologies that we had in Storm was to ingest data, index to elastic > (write) and right after the write to elastic we would trigger a percolator > query for a downstream task to be triggered. The DAGs using Flux allowed us > to chain these steps ( [read from source] -> [process] -> [index] -> > [percolator] -> [trigger]). > > > > *Beam:* > > We are able to accomplish much of that using beam. However, using > ElasticsearchIO, the index step results in a PDone and hence we cannot > chain the remaining tasks ( [percolator] -> [trigger other events]). > > > > The question I have is this: is there a way to trigger a percolator query > after writing a document to elastic using ElasticsearchIO.Write()? > Which version of Beam are you using ? Seems like for recent versions of Beam, we actually return PCollections from ElasticsearchIO sink so this should not be an issue. https://github.com/apache/beam/blob/24f40aab7aa4c4f9eea6dc90c0baa22bb17a962e/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L1842 Thanks, Cham > > > Your thoughts would be appreciated! > > Vinod > > >
