Hi, Vignesh Flink is a distributed parallel computing framework, each MapFunction is actually a separate thread. If you want more threads to process the data, you can increase the parallelism of the MapFunction without having to use multiple threads in a single MapFunction, which in itself violates the original design intent of Flink.
Best, Ron Vignesh Kumar Kathiresan via user <user@flink.apache.org> 于2023年8月15日周二 03:59写道： > Hello All, > > *Problem statement * > For a given element, I have to perform multiple(lets say N) operations on > it. All the N operations are independent of each other. And for achieving > lowest latency, I want to do them concurrently. I want to understand what's > the best way to perform it in flink?. > > I understand flink achieves huge parallelism across elements. But is it > anti-pattern to do parallel processing in a map func at single element > level? I do not see anything on the internet for using multithreading > inside a map function. > > I can always fan out with multiple copies of the same element and send > them to different operators. But it incurs at the least a > serialize/deserialize cost and may also incur network shuffle. Trying to > see if a multithreaded approach is better. > > Thanks, > Vignesh >
