Awesome, thanks Fabian !
I will give this a try.
Fabian Hueske-2 wrote
> Hi Philipp,
>
> If I got your requirements right you would like to:
> 1) load an initial hashmap via JDBC
> 2) update the hashmap from a stream
> 3) use the hashmap to enrich another stream.
>
> You can use a CoFlatMap to
Hi Philipp,
If I got your requirements right you would like to:
1) load an initial hashmap via JDBC
2) update the hashmap from a stream
3) use the hashmap to enrich another stream.
You can use a CoFlatMap to do this:
stream1.connect(stream2).flatMap(new YourCoFlatMapFunction).
YourCoFlatMapFunc
Hi again,
I implemented the RichMap Function (open method runs a JDBC query to
populate a HashMap with data) which I am using in the map function.
Now there is another RichMap.map function that would add to the HashMap that
was initialized in the first function.
How would I share the Map between th
You can just use plain JDBC. Just keep in mind, that the classes will be
serialized and sent through the cluster. So probably, you want to
initialize all the non-serializable database access object in the open
method itself (as opposed to the constructor (client side)).
Cheers,
Konstantin
On 12.
Thank you Konstantin, the amount of data I have to load into memory will be
very small so that should be alright.
When opening and querying the database would I use any sort of Flink magic
or just do plain JDBC ?
I read about the JDBCInput concept which one could use with the DataSet API
and was wo
Hi Philipp,
the easist way is a RichMap. In the open()-Method you can load the
relevant database table into memory (e.g. a HashMap). In the
map()-method you than just look up the entry in the HashMap.
Of course, this only works if the dataset is small enough to fit in
memory. Is it?
Cheers,
Kon
