Does that work if cassandra returns a PDone? On Wed, Mar 24, 2021 at 10:00 AM Chamikara Jayalath <chamik...@google.com> wrote:
> If you want to wait for all records are written (per window) to Cassandra > before writing that window to PubSub, you should be able to use the Wait > transform: > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Wait.java > > Thanks, > Cham > > On Wed, Mar 24, 2021 at 9:49 AM Alexey Romanenko <aromanenko....@gmail.com> > wrote: > >> Do you want to wait for ALL records are written for Cassandra and then >> write all successfully written records to PubSub or it should be performed >> "record by record"? >> >> On 24 Mar 2021, at 04:58, Vincent Marquez <vincent.marq...@gmail.com> >> wrote: >> >> I have a common use case where my pipeline looks like this: >> CassandraIO.readAll -> Aggregate -> CassandraIO.write -> PubSubIO.write >> >> I do NOT want my pipeline to look like the following: >> >> CassandraIO.readAll -> Aggregate -> CassandraIO.write >> | >> -> >> PubsubIO.write >> >> Because I need to ensure that only items written to Pubsub have >> successfully finished a (quorum) write. >> >> Since CassandraIO.write is a PTransform<A, PDone> I can't actually use it >> here so I often roll my own 'writer', but maybe there is a recommended way >> of doing this? >> >> Thanks in advance for any help. >> >> *~Vincent* >> >> >>
