Hi Alexis:
The broadcast state pattern should be done by calling connect() on the
non-broadcasted stream, with the broadcaststream as an argument.
And if the main stream is keyedStream, then the processElement function
will have access to any keyed state as normal keyedstream.
Best Regards!
从 Windows 版邮件发送
发件人: Alexis Sarda-Espinosa
发送时间: 2022年10月12日 4:11
收件人: user
主题: Partial broadcast/keyed connected streams
Hi everyone,
I am currently thinking about a use case for a streaming job and, while I'm
fairly certain it cannot be done with the APIs that Flink currently provides, I
figured I'd put it out there in case other users think something like this
would be useful to a wider audience.
The current broadcasting mechanisms offered by Flink mention use cases where
"control events" are needed. In my case I would also have control events, and I
would need to broadcast them to *all parallel instances* of any downstream
operators that consume the events. However, some of those operators have to be
keyed because they are stateful. From the API's point of view, I'd imagine
something like
controlStream.connect(mainStream).broadcastFirstKeySecondBy(keySelector).process(PartiallyKeyedCoProcessFunction)
The function would also have something like processElement1 and
processElement2, but one of those methods wouldn't have access to partitioned
state (or could it have access to state for all key groups handled by that
instance?).
Since I'm not familiar with all of Flink's internals, I don't know if this
would be even remotely feasible, but I'd like to know if others have opinions
on this.
Regards,
Alexis.
