Hi Salva, Can not you take into account the pending element that’s stuck somewhere in the transit? Snapshot it as well and during recovery reprocess it? This is exactly that’s AsyncWaitOperator is doing.
Piotrek > On 5 Apr 2020, at 15:00, Salva Alcántara <salcantara...@gmail.com> wrote: > > Hi again Piotr, > > I have further considered the mailbox executor approach and I think it will > not be enough for my purposes. Here is why: > > - My state consists of models created with a third party library > - These models have their own state, which means that when I forward events > in `ProcessElement1` to these models, the model's state will be updated > accordingly. > > So, what would happen if: > > - A new element E is processed in `ProcessElement1` and sent to the third > party library model > - A checkpoint is taken, in particular snapshotting all the library models > in use > - The element E that was sent to the library is expected to generate an > output O result when the callback is called, but a failure happens before > that > - Application recovers from the snapshot and continue processing elements, > but the callback generating the expected output O has been lost by now, so > that output will be lost > > By considering the above case, I realize that the only option for me might > be to rely on AsyncIO. However, this is far from ideal because I am not > expecting an output result for each element I send to my models. I could use > a timeout but that may slow down processing as asyncIO has a limited queue > of "active" elements. Also, most of the times, I am not expecting a result > back at all from my models (callbacks will be invoked only a few times since > my modes are detecting anomalies). > > In your opinion, what would be the best approach for handling this use case? > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
