It’s good to be here. I have a data stream coming from kinesis. I also have a
list of hashmap which holds metadata that needs to participate in the
processing.
In my flink processor class I construct this metadata (hardcoded)
public static void main(String[] args) throws Exception {
…….//
DataStream<ObjectNode> jsonStream = see.addSource(new
FlinkKinesisConsumer<>("int-harvester-kinesis-us-west-2", new
MyJsonDeserializationSchema(), consumerConfig));
DataStream<HashMap<String,String>> metaStream = env.fromCollection(metadata);
DataStream<String> outStream = metaStream
.broadcast()
.connect(jsonStream).flatMap(new
SplitterCoFlatMapFunction());
SplitterCoFlatMapFunction extends RichCoFlatMapFunction
By doing the above I can operate on my jsonStream and avail elements of hashMap
in that processing.
I am not sure , as a next step , if I am given a rest api ,which will
eventually give me the latest version of that metadata collection , what is the
right way to do that call and from where , as when flink process is started the
metadata is already prepared and propagated to the workers,where can I refresh
it.
What is the most correct way of doing it?
Thanks
Sathi
=============Notice to Recipient: This e-mail transmission, and any documents,
files or previous e-mail messages attached to it may contain information that
is confidential or legally privileged, and intended for the use of the
individual or entity named above. If you are not the intended recipient, or a
person responsible for delivering it to the intended recipient, you are hereby
notified that you must not read this transmission and that any disclosure,
copying, printing, distribution or use of any of the information contained in
or attached to this transmission is STRICTLY PROHIBITED. If you have received
this transmission in error, please immediately notify the sender by telephone
or return e-mail and delete the original transmission and its attachments
without reading or saving in any manner. Thank you. =============
