Hi, Can the enriching data be keyed? Or is it something that has to be broadcasted to each operator? Either way, I think Side Inputs (an upcoming feature in the future) is the best fit for this. You can take a look at https://issues.apache.org/jira/browse/FLINK-6131.
Regarding the 3 options you listed: By using QueryableState in option B, what you mean is that you want to feed the enriching data stream to a separate job, let that job allow queryable state, and query that state from the actual application job operators, correct? If so, I think options A and B would mean the same thing; i.e., they require accessing data external to the job. If the enriching data can somehow be keyed with the stream that requires it, I would go for option C using connected streams, with the enriching data as one input and the actual data as the other. Instead of just “caching the enriching data in memory”, you should register it as a managed Link state for the CoMapFunction / CoFlatMapFunction. The actual input stream records can just access that registered state locally. Cheers, Gordon On 19 May 2017 at 7:11:07 AM, Sand Stone ([email protected]) wrote: Hi. Say I have a few reference data sets need to be used for a streaming job. The sizes range between 10M-10GB. The data is not static, will be refreshed at minutes and/or day intervals. With the new advancements in Flink, it seems there are quite a few options. A. Store all the data in an external (kv) database cluster. And use async io calls * data refresh can be done in a few different ways B. Use the new Querytable State feature * it seems there is no "easy" API to discover the queryable state at the moment. Need to use the restful API to figure out the job id. C. Ingest the reference data into the job and cache them in memory Any other option? On paper, it seems option B with the Queryable State is the cleanest solution. Any comment/suggestion is greatly appreciated in particular in terms of robustness and consistent recovery. Thanks much!
