Hi.
I have a case where I have a input stream that I want to enrich with
external data. I want to cache some of the external lookup data to improve
the overall performances.
To update my cache (a CoProcessFunction) I would use iteration to send the
external enriched information back to the cache and update a mapstate. I
use CoProcesFunction as the input stream and the enrich stream contains 2
diff.object types and I don't want to mix them.
Because I use a ConnectedIterativeStream I can't use state in my
CoProcessFunction because the ConnectedIterativeStream create a DataStream
based on the Feedback signature and not the stream I close the iteration
with and it is not possible to provide a keySelector in the withFeedbackType
Form Flink source
public ConnectedIterativeStreams(DataStream<I> input,
TypeInformation<F> feedbackType, long waitTime) {
super(input.getExecutionEnvironment(), input, new
DataStream(input.getExecutionEnvironment(), new
CoFeedbackTransformation(input.getParallelism(), feedbackType,
waitTime)));
}
and both streams need to be keyed before state are assigned to the operator.
Any ideas how to workaround this problem?
My sudo code is as below.
IterativeStream.ConnectedIterativeStreams<InputObject, EnrichData>
iteration = inputStream
.keyBy(obj -> obj.getkey))
.iterate(maxWaitForIterations).withFeedbackType(TypeInformation.of(new
TypeHint<EnrichData>() {}));
DataStream<ReportMessageBase> enrichedStream = iteration
.process(new EnrichFromState());
DataStream<ReportMessageBase> notEnrichedOutput = enrichedStream
.filter(obj -> obj.enriched);
EnrichService EnrichService = new EnrichService();
DataStream<InputObject> enrichedFromApi =
EnrichService.parse(notEnrichedOutput);
DataStream<EnrichData> newEnrich = enrichedFromApi
.map(obj -> {
EnrichData newData = new EnrichData();
newData.xx = obj.xx();
return newData;
})
.keyBy(obj -> obj.*getkey*);
iteration.closeWith(newAddresses);
....
