Re: Back pressure within a operator chain

2019-01-24 Thread Chesnay Schepler
The behavior should be identical regardless of whether the are chained or not. On 23.01.2019 09:11, Paul Lam wrote: Hi, I would like to know if back pressure applies to operators in the same operator chain? The background is that I have a simple streaming job that consumes data from Kafka,

Back pressure within a operator chain

2019-01-23 Thread Paul Lam
Hi, I would like to know if back pressure applies to operators in the same operator chain? The background is that I have a simple streaming job that consumes data from Kafka, do some transformation and writes to HDFS (all the operators are chained together), and if the Kafka partitions are muc