Hi Lasse,

One approach I’ve used in a similar situation is to have a “UnionedSource” 
wrapper that first emits the (bounded) data that will be loaded in-memory, and 
then starts running the source that emits the continuous stream of data.

This outputs an Either<A, B>, which I then split, and broadcast the A, and 
key/partition the B.

You could do something similar, but occasionally keep checking if there’s more 
<A> data vs. assuming it’s bounded.

The main issue I ran into is that it doesn’t seem possible to do checkpointing, 
or at least I couldn’t think of a way to make this work properly.

— Ken


> On Sep 27, 2018, at 9:50 PM, Lasse Nedergaard <lassenederga...@gmail.com> 
> wrote:
> 
> Hi. 
> 
> We have created our own database source that pools the data with a configured 
> interval. We then use a co processed function. It takes to input one from our 
> database and one from our data input. I require that you keyby with the 
> attributes you use lookup in your map function. 
> To delay your data input until your database lookup is done first time is not 
> simple but a simple solution could be to implement a delay operation or keep 
> the data in your process function until data arrive from your database 
> stream. 
> 
> Med venlig hilsen / Best regards
> Lasse Nedergaard
> 
> 
> Den 28. sep. 2018 kl. 06.28 skrev Chirag Dewan <chirag.dewa...@yahoo.in 
> <mailto:chirag.dewa...@yahoo.in>>:
> 
>> Hi,
>> 
>> I saw Apache Flink User Mailing List archive. - static/dynamic lookups in 
>> flink streaming 
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/static-dynamic-lookups-in-flink-streaming-td10726.html>
>>  being discussed, and then I saw this FLIP 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API
>>  
>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API>.
>>  
>> 
>> I know we havent made much progress on this topic. I still wanted to put 
>> forward my problem statement around this. 
>> 
>> I am also looking for a dynamic lookup in Flink operators. I actually want 
>> to pre-fetch various Data Sources, like DB, Filesystem, Cassandra etc. into 
>> memory. Along with that, I have to ensure a refresh of in-memory lookup 
>> table periodically. The period being a configurable parameter. 
>> 
>> This is what a map operator would look like with lookup: 
>> 
>> -> Load in-memory lookup - Refresh timer start
>> -> Stream processing start
>> -> Call lookup
>> -> Use lookup result in Stream processing
>> -> Timer elapsed -> Reload lookup data source into in-memory table
>> -> Continue processing
>> 
>> 
>>  My concern around these are : 
>> 
>> 1) Possibly storing the same copy of data in every Task slots memory or 
>> state backend(RocksDB in my case).
>> 2) Having a dedicated refresh thread for each subtask instance(possibly, 
>> every Task Manager having multiple refresh thread)
>> 
>> Am i thinking in the right direction? Or missing something very obvious? It 
>> confusing.
>> 
>> Any leads are much appreciated. Thanks in advance.
>> 
>> Cheers, 
>> Chirag

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra

Reply via email to