Hi Matthias,
I am facing a similar issue with the need to have state on 2 keyed input
streams, but the state can only be cleared with events occurring in a third
input stream.
Do you have maybe some examples how to utilize MultipleInputStreamOperator you
have mentioned?
Thanks and kind regards,
Christian
Von: Schwalbe Matthias
Datum: Mittwoch, 7. August 2024 um 12:54
An: Sachin Mittal , user@flink.apache.org
Betreff: RE: Can we share states across tasks/operators
This email has reached Mapp via an external source
Hi Sachin,
Just as an idea, while you cannot easily share state across operators, you can
do so within the same operator:
* For two such input streams you could connect() the two streams into a
ConnectedStreams and then process() by means of a KeyedCoProcessFunction
* For more than two input streams, implement some
MultipleInputStreamOperator …
* In both cases you can yield multiple independent output streams (if need
be), by means of multiple side outputs (see here e.g.
org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction.Context#output)
I do that all the time 😊
WDYT?
Sincere Flink greetings
Thias
From: Sachin Mittal
Sent: Wednesday, August 7, 2024 12:37 PM
To: user@flink.apache.org
Subject: Can we share states across tasks/operators
Hi,
I have a stream which starts from a source and is keyed by a field f.
With the stream process function, I can emit the processed record downstream
and also update state based on the records it received for the same key.
Now I have another stream which starts from another source and is of the same
type as the first stream and it is also keyed by the same field f.
In its process function I want to access the last state updated by the first
stream's process function for the same key, do some processing (update the
state) and also send the record downstream.
Is there any way I can achieve this in Flink by connecting to the same state
store ?
Is there any concept of global state in Flink if I cannot achieve this by using
keyed states associated with an operator's process function ?
Any other way you can think of achieving the same ?
Thanks
Sachin
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet
unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von
e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine
Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser
Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per
e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche
unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng
verboten.
This message is intended only for the named recipient and may contain
confidential or privileged information. As the confidentiality of email
communication cannot be guaranteed, we do not accept any responsibility for the
confidentiality and the intactness of this message. If you have received it in
error, please advise the sender by return e-mail and delete this message and
any attachments. Any unauthorised use or dissemination of this information is
strictly prohibited.
Mapp Digital Germany GmbH with registered offices at Schönhauser Allee 148,
10435 Berlin.
Registered with the District Court München HRB 226181
Managing Directors: Frasier, Christopher & Warren, Steve
This e-mail is from Mapp Digital Group and its international legal entities and
may contain information that is confidential.
If you are not the intended recipient, do not read, copy or distribute the
e-mail or any attachments. Instead, please notify the sender and delete the
e-mail and any attachments.
