Hi Josh, I am no expert in Flink yet, but here are my thoughts on this:
1. what about you stream an event to flink everytime the DB of items have an update? then in some background thread you get the new data from the DB let it be through REST (if it is only few updates a day) then load the results in memory and there is your updated static data. 2. REST API are over HTTP, how that is possible to be a sink? does not sound like flink job at all to serve http requests. simply sink the results to some DB and have some component to read from DB and serve it as REST API. -Rami On 23 May 2016, at 16:22, Josh <jof...@gmail.com<mailto:jof...@gmail.com>> wrote: Hi all, I am new to Flink and have a couple of questions which I've had trouble finding answers to online. Any advice would be much appreciated! 1. What's a typical way of handling the scenario where you want to join streaming data with a (relatively) static data source? For example, if I have a stream 'orders' where each order has an 'item_id', and I want to join this stream with my database of 'items'. The database of items is mostly static (with perhaps a few new items added every day). The database can be retrieved either directly from a standard SQL database (postgres) or via a REST call. I guess one way to handle this would be to distribute the database of items with the Flink tasks, and to redeploy the entire job if the items database changes. But I think there's probably a better way to do it? 2. I'd like my Flink job to output state to a REST API. (i.e. using the REST API as a sink). Updates would be incremental, e.g. the job would output tumbling window counts which need to be added to some property on a REST resource, so I'd probably implement this as a PATCH. I haven't found much evidence that anyone else has used a REST API as a Flink sink - is there a reason why this might be a bad idea? Thanks for any advice on these, Josh Disclaimer: This message and any attachments thereto are intended solely for the addressed recipient(s) and may contain confidential information. If you are not the intended recipient, please notify the sender by reply e-mail and delete the e-mail (including any attachments thereto) without producing, distributing or retaining any copies thereof. Any review, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. Thank you.
