[
https://issues.apache.org/jira/browse/FLINK-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reassigned FLINK-3659:
---------------------------------------
Assignee: Kostas Kloudas (was: Aljoscha Krettek)
> Add ConnectWithBroadcast Operation
> ----------------------------------
>
> Key: FLINK-3659
> URL: https://issues.apache.org/jira/browse/FLINK-3659
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.0.0
> Reporter: Aljoscha Krettek
> Assignee: Kostas Kloudas
>
> We should add a new operation that has a main input that can be keyed (but
> doesn't have to be) and a second input that is always broadcast. This is
> similar to a {{CoFlatMap}} or {{CoMap}} but there either both inputs have to
> be keyed or non-keyed.
> This builds on FLINK-4940 which aims at adding broadcast/global state. When
> processing an element from the broadcast input only access to broadcast state
> is allowed. When processing an element from the main input access both the
> regular keyed state and the broadcast state can be accessed.
> I'm proposing this as an intermediate/low-level operation because it will
> probably take a while until we add support for side-inputs in the API. This
> new operation would allow expressing new patterns that cannot be expressed
> with the currently expressed operations.
> This is the new proposed API (names are non-final):
> 1) Add {{DataStream.connectWithBroadcast(DataStream)}} and
> {{KeyedStream.connectWithBroadcast(DataStream)}}
> 2) Add {{ConnectedWithBroadcastStream}}, akin to {{ConnectedStreams}}/
> 3) Add {{BroadcastFlatMap}} and {{TimelyBroadcastFlatMap}} as the user
> functions.
> Sketch of the user function:
> {code}
> interface BroadcastFlatMapFunction {
> public void flatMap(IN in, Collector out);
> public void processBroadcastInput(BIN in);
> }
> {code}
> The API names, function names are a bit verbose and we have to add two new
> different ones but I don't see a way around this with the current way the
> Flink API works.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
