Hi, 1. Is the key that is used by the keyBy after point 1 precisely the same as the key used by the 4a and 4b? If yes, I think you could use the reinterpretAsKeyedStream to avoid the shuffle. 2. You could use SingleOutputStreamOperator::setChainingStrategy to disable the chain or use rebalance/shuffle between the two operators if you don't care about the order. 3. If you use the unorderedWait the order would not be preserved even if you use keyBy after point1 4. BTW why do you not want the operator chain together?
Best, Guowei KristoffSC <krzysiek.chmielew...@gmail.com> 于2020年1月30日周四 下午7:54写道： > Hi, > thank you for the answer. > > I think I understand. > > In my uses case I have to keep the order of events for each key, but I dont > have to process keys in the same order that I received them. On one point > of > my pipeline I'm also using a SessionWindow. > > My Flink environment has operator chaining enabled. I woudl say, that some > of my operators can be chained. > > My pipeline is (each point is an operator after Flink's operator chainign > mechanism) > 1. ActiveMQ connector + mapper, all with parallelism 1 (btw I'm using a > org.apache.bahir connector for Active MQ which does not support parallelism > bigger than 1) > 2. Enrichment, where Im using AsyncDataStream.unorderedWait with > parallelism > 5. > 3. Event split based on some criteria (not key by) that dispatches my > stream > into two "sub streams" > 4. Both substreams are keyed > 4a. SubStream "A" has a session window applied - parallelism 6. > 4b. Substream "B" has no windowing, no aggregation, but has a business > logic > for witch order of events matters. - parallelism 6 > 5. Sink for both streams. > > > If I understand you and documentation correctly, Redistributing will > forward > messages keeping the order for a key, but events between keys can be > delivered in a different order. > "So in this example, the ordering within each key is preserved, but the > parallelism does introduce non-determinism regarding the order in which the > aggregated results for different keys arrive at the sink." > > Then I could use a keyBy at the pipeline beginning, just after point 1. > But to use Window in point 4a and my process function in 4b I need to have > a > keyedStream. I'm using a KeyedProcessFunction there. What my options with > this? > > > P.S. > Regarding the operator chaining, I'm aware that there is an API that allows > me to model which operators should be chained theatergoer and which not > even > if they have the same parallelism level. > > Thanks, > Krzysztof > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
