I'm currently not using Wait.on. I have a pipeline that uses BigQuery for checkpointing purposes. I only want records that are successfully checkpointed in BigQuery to be eligible for the next stage in my pipeline. With streaming inserts, I can use getSuccessfulInserts to achieve this and I was looking for something similar with Storage Write.
On Thu, Mar 2, 2023 at 4:48 PM Reuven Lax via user <user@beam.apache.org> wrote: > Are you trying to do this in order to use Wait.on? getSuccessfulInserts is > not currently supported for Storage Write API. > > On Thu, Mar 2, 2023 at 1:44 PM Matthew Ouyang <matthew.ouy...@gmail.com> > wrote: > >> Thank you to Ahmed and Reuven for the tip on >> WriteResult::getFailedStorageApiInserts. >> >> When I tried to get the successful inserts through the Storage Write API, >> I received an error message saying that "Retrieving successful inserts is >> only supported for streaming inserts. Make sure >> withSuccessfulInsertsPropagation is correctly configured for >> BigQueryIO.Write object." Did I make a mistake, or is there a reason why >> this is not possible? I tried setting triggeringFrequency + >> numStorageWriteApiStreams as required by Storage Write, and I tried to set >> successfulInsertsPropagation as directed in the error message. >> >
