So as you were maybe thinking, it only happens with the combination:
Direct Stream only + backpressure = works as expected
4x Receiver on Topic A + Direct Stream on Topic B + backpressure = the
direct stream is throttled even in the absence of scheduling delay
This is using Spark 1.5.0 on CDH.
Yes I'll test that next.
On Sep 9, 2016 5:36 PM, "Cody Koeninger" wrote:
> Does the same thing happen if you're only using direct stream plus back
> pressure, not the receiver stream?
>
> On Sep 9, 2016 6:41 PM, "Jeff Nadler" wrote:
>
>> Maybe this is a pretty esoteric implementation, but I'm s
Does the same thing happen if you're only using direct stream plus back
pressure, not the receiver stream?
On Sep 9, 2016 6:41 PM, "Jeff Nadler" wrote:
> Maybe this is a pretty esoteric implementation, but I'm seeing some bad
> behavior with backpressure plus multiple Kafka streams / direct stre
Maybe this is a pretty esoteric implementation, but I'm seeing some bad
behavior with backpressure plus multiple Kafka streams / direct streams.
Here's the scenario:
We have 1 Kafka topic using the reliable receiver (4 receivers, union the
result).In the same app, we consume another Kafka topi
