Re: Sharing data among flink operator pipeline

[shenzhongwei]

unsubscribe


         原始邮件
         
       
发件人：Pedro Mázala <pedroh.maz...@gmail.com&gt;
发件时间：2025年8月21日 18:30
收件人：Kamal Mittal <kamal.mit...@ericsson.com&gt;
抄送：user@flink.apache.org <user@flink.apache.org&gt;
主题：Re: Sharing data among flink operator pipeline



Hello there, Kamal!

Whenever I need to keep passing data downstream, I reflect it on my POJO. 
Usually, I call those fields metadata and access them downstream. The state 
will also segment data by operator, in other words, an operator cannot access 
stored data from other operators (unless you use broadcast state, which is 
usually not a great option).




The issue with broadcast state is that it will increase your storage quite a 
bit, depending on how many operators/keys you have.






Att,
Pedro Mázala
Be awesome



On Thu, 21 Aug 2025 at 11:34, Kamal Mittal via user <user@flink.apache.org&gt; 
wrote:

Hello,

&nbsp;

I have a scenario like for kafka source where kafka headers also come along 
with kafka record/event. Kafka headers fetched need to share/pass to 
next/parallel operators in pipeline.

&nbsp;

So, is there any way to share data across operator pipeline?

&nbsp;

Explored keyed state which has limitation that only with keyby() it will work.

Explored broadcast state which limitation that it can only work with process 
functions and not with map/flatmap/filter etc.

&nbsp;

Rgds,

Kamal