Hi Max, Thanks, that's very helpful re the REST API sink. For now I don't need exactly once guarantees for the sink, so I'll just write a simple HTTP sink implementation. But may need to move to the idempotent version in future!
For 1), that sounds like a simple/easy solution, but how would I handle occasional updates in that case, since I guess the open() function is only called once? Do I need to periodically restart the job, or periodically trigger tasks to restart and refresh their data? Ideally I would want this job to be running constantly. Josh On Mon, May 23, 2016 at 5:56 PM, Maximilian Michels <m...@apache.org> wrote: > Hi Josh, > > 1) Use a RichFunction which has an `open()` method to load data (e.g. from > a database) at runtime before the processing starts. > > 2) No that's fine. If you want your Rest API Sink to interplay with > checkpointing (for fault-tolerance), this is a bit tricky though depending > on the guarantees you want to have. Typically, you would have "at least > once" or "exactly once" semantics on the state. In Flink, this is easy to > achieve, it's a bit harder for outside systems. > > "At Least Once" > > For example, if you increment a counter in a database, this count will be > off if you recover your job in the case of a failure. You can checkpoint > the current value of the counter and restore this value on a failure (using > the Checkpointed interface). However, your counter might decrease > temporarily when you resume from a checkpoint (until the counter has caught > up again). > > "Exactly Once" > > If you want "exactly once" semantics on outside systems (e.g. Rest API), > you'll need idempotent updates. An idempotent variant of this would be a > count with a checkpoint id (cid) in your database. > > | cid | count | > |-----+-------| > | 0 | 3 | > | 1 | 11 | > | 2 | 20 | > | 3 | 120 | > | 4 | 137 | > | 5 | 158 | > > You would then always read the newest cid value for presentation. You > would only write to the database once you know you have completed the > checkpoint (CheckpointListener). You can still fail while doing that, so > you need to keep the confirmation around in the checkpoint such that you > can confirm again after restore. It is important that confirmation can be > done multiple times without affecting the result (idempotent). On recovery > from a checkpoint, you want to delete all rows higher with a cid higher > than the one you resume from. For example, if you fail after checkpoint 3 > has been created, you'll confirm 3 (because you might have failed before > you could confirm) and then delete 4 and 5 before starting the computation > again. > > You see, that strong consistency guarantees can be a bit tricky. If you > don't need strong guarantees and undercounting is ok for you, implement a > simple checkpointing for "at least once" using the Checkpointed interface > or the KeyValue state if your counter is scoped by key. > > Cheers, > Max > > > On Mon, May 23, 2016 at 3:22 PM, Josh <jof...@gmail.com> wrote: > > Hi all, > > > > I am new to Flink and have a couple of questions which I've had trouble > > finding answers to online. Any advice would be much appreciated! > > > > What's a typical way of handling the scenario where you want to join > > streaming data with a (relatively) static data source? For example, if I > > have a stream 'orders' where each order has an 'item_id', and I want to > join > > this stream with my database of 'items'. The database of items is mostly > > static (with perhaps a few new items added every day). The database can > be > > retrieved either directly from a standard SQL database (postgres) or via > a > > REST call. I guess one way to handle this would be to distribute the > > database of items with the Flink tasks, and to redeploy the entire job if > > the items database changes. But I think there's probably a better way to > do > > it? > > I'd like my Flink job to output state to a REST API. (i.e. using the REST > > API as a sink). Updates would be incremental, e.g. the job would output > > tumbling window counts which need to be added to some property on a REST > > resource, so I'd probably implement this as a PATCH. I haven't found much > > evidence that anyone else has used a REST API as a Flink sink - is there > a > > reason why this might be a bad idea? > > > > Thanks for any advice on these, > > > > Josh > >
