Hello Nicholas,

This is a more comprehensive doc, that also includes rejected alternatives.

https://docs.google.com/document/d/1OEbGZlnCcYVj8zBF-pUgLL69SFxOgkEDVzRpqK7bG_g/edit?usp=sharing

Thanks,
Jagadish

On Thu, Jul 25, 2019 at 8:51 AM Nicholas Audo <na...@naudo.de> wrote:

> Hello Jagadish,
> Is the rejected alternatives document meant to be private (link at the
> bottom of the document)?
>
> On Wed, Jul 24, 2019 at 11:46 AM Jagadish Venkatraman <jagad...@apache.org
> >
> wrote:
>
> > Dear Samza users,
> >
> > We recently discovered an issue with the way we handle state in Samza
> Beam
> > and Samza High-Level API Window operators. Under certain situations, at
> > least once processing guarantee is violated.
> >
> >
> > *Details on the issue*
> >
> > The Samza high-level API includes operators such as windows which can
> hold
> > messages and emit them at a later time. eg: a time based window will
> buffer
> > its messages and emit results only at periodic time intervals. Here’s the
> > sequence of operations when a window operator is ready to emit results:
> >
> >    1.
> >
> >    Obtain the results ready to be emitted for the window
> >    2.
> >
> >    Remove the operator state corresponding to those results from its
> >    state-store
> >    3.
> >
> >    Propagate the window results to down-stream operators in the pipeline.
> >    The results eventually makes it to a terminal operator, which emits
> the
> >    final output.
> >    4.
> >
> >    At some future point, issue a commit operation, which flushes the
> >    producers, the state stores and persists the input offsets.
> >
> >
> > Scenarios
> >
> >    -
> >
> >    An exception in a downstream operator in step 3.
> >    -
> >
> >    An unclean shutdown, e.g., due to a “kill -9” to the container before
> >    the outputs have been flushed in step 4
> >
> >
> > Both these examples violate Samza’s at-least once processing guarantee,
> > since they cause state to be modified even though the processed outputs
> may
> > not have been emitted.
> >
> >
> > *Solution*
> >
> > Here's a proposal
> > <
> >
> https://docs.google.com/document/d/1wtSAUMrns14cWCf5Wdf4YwXd7K5Hf-bm7W3SQ3sXZsQ/edit
> > >
> > to fix the above issue. We are working on addressing this with the
> highest
> > priority.
> >
> >
> > Thanks,
> >
> > Jagadish
> >
>

Reply via email to