Hi Jeff, Thanks for the response. Yes I have a Java pipeline and yes it is a simple transformation. While DoFns work on bundles and if a single element in the bundle fails and we ignore the error on the single element, then the bundle is considered still successfully processed am I correct? Then it would just ACK everything in the bundle
Kishore On Mon, Jun 1, 2020 at 10:27 AM Jeff Klukas <jklu...@mozilla.com> wrote: > Is this a Python or Java pipeline? > > I'm familiar with PubsubIO in Java, though I expect the behavior in Python > is similar. It will ack messages at the first checkpoint step in the > pipeline, so the behavior in your case depends on whether there is a > GroupByKey operation happening before the exception is thrown. > > If there are no GroupByKey operations (so your pipeline is basically just > transforming single messages and republishing to a new topic), then I would > expect you are safe and messages will not be ACK'd unless they have been > published to the output topic. > > And yes, if an exception is thrown the whole bundle would fail, so those > messages would be picked up by another worker and retried. > > There is a chance of data loss if you have a pipeline that needs to > checkpoint data to disk and you shut down the pipeline without draining the > data. In that case, messages may have been ack'd to pubsub and held durably > only in the checkpoint state in Dataflow, so shutting down the pipeline > uncleanly would lose data. > > On Mon, Jun 1, 2020 at 1:09 PM KV 59 <kvajjal...@gmail.com> wrote: > >> Hi, >> >> I have a Dataflow pipeline with PubSub UnboundedSource, The pipeline >> transforms the data and writes to another PubSub topic. I have a question >> regarding exceptions in DoFns. If I chose to ignore an exception processing >> an element, does it ACK the bundle? >> >> Also if I were to just throw the exception, my understanding is the >> Dataflow Runner will fail the whole bundle and keeps retrying until the >> whole bundle is successful am I correct? >> >> Thanks for your response >> >
