The behavior should be identical regardless of whether the are chained
or not.
On 23.01.2019 09:11, Paul Lam wrote:
Hi,
I would like to know if back pressure applies to operators in the same
operator chain?
The background is that I have a simple streaming job that consumes
data from Kafka, do some transformation and writes to HDFS (all the
operators are chained together), and if the Kafka partitions are much
greater that job parallelism (like 40:1), OOM happens. The the root
cause should be Kafka consumer pulling too much data. So I’m wondering
if I should separate the source and sink to make the back pressure
mechanism working.
Best,
Paul Lam
