unsubscribe
原始邮件
发件人：Pedro Mázala <[email protected]>
发件时间：2025年8月21日 18:30
收件人：Kamal Mittal <[email protected]>
抄送：[email protected] <[email protected]>
主题：Re: Sharing data among flink operator pipeline
Hello there, Kamal!
Whenever I need to keep passing data downstream, I reflect it on my POJO.
Usually, I call those fields metadata and access them downstream. The state
will also segment data by operator, in other words, an operator cannot access
stored data from other operators (unless you use broadcast state, which is
usually not a great option).
The issue with broadcast state is that it will increase your storage quite a
bit, depending on how many operators/keys you have.
Att,
Pedro Mázala
Be awesome
On Thu, 21 Aug 2025 at 11:34, Kamal Mittal via user <[email protected]>
wrote:
Hello,
I have a scenario like for kafka source where kafka headers also come along
with kafka record/event. Kafka headers fetched need to share/pass to
next/parallel operators in pipeline.
So, is there any way to share data across operator pipeline?
Explored keyed state which has limitation that only with keyby() it will work.
Explored broadcast state which limitation that it can only work with process
functions and not with map/flatmap/filter etc.
Rgds,
Kamal
