I have a stream of tuples <channel: Int, index: Long, value: Double> , which I form into a keyedStream using keyBy on channel. I then need to process each channel in parallel. Each parallel stream must be processed in strict sequential order by index to calculate the ratios value(index)/value(index-1). If I set parallelism to 1 all is well, each channel is processed in order of index 1,2,3,,4…
My problem is when I set parallelism to a value greater than 1 each channel’s keyedStream appears to be split across multiple processes. So a channel may be processed wrongly for example as value(2), value(5), Value(6) , value(9)….. The number of channels N is unknown. So how do I rig up N processing streams with an unknown parallelism so that each stream processes each channel by strictly increasing index v(1),v(2),…..v(t),v(t+1),…..v(t+n) Thanks in advance NIck Walton
