Resolved by downgrading Client to 2.2.2 and implementing an application level
heartbeat on every Producer to avoid he UNKNOWN_PRODUCER_ID issue.
> On 9/03/2020, at 16:08, James Olsen wrote:
>
> P.S. I guess the big question is what is the best way to handle or avoid
> UNKNOWN_PRODUCER_ID when
P.S. I guess the big question is what is the best way to handle or avoid
UNKNOWN_PRODUCER_ID when running versions that don’t include KAFKA-7190 /
KAFKA-8710 ?
We are using non-transactional idempotent Producers.
> On 9/03/2020, at 12:59 PM, James Olsen wrote:
>
> For completeness I have also
For completeness I have also tested 2.4.0 Broker with 2.4.0 Client. All works
correctly. Unfortunately as we are on AWS MSK we don’t have the option to use
2.4.0 for the Brokers.
So now I guess the question changes to what combo is best for us and will it
avoid UNKNOWN_PRODUCER_ID problems?
W
Jamie,
I’ve just tested with 2.3.1 Broker and 2.3.1 Client and it works correctly. So
with that setup it does deliver the batch as soon as any partition has data.
This is what we would expect from the Kafka docs.
So it looks like an issue with the 2.4.0 Client. This is concerning as I
wante
Hi James,
My understanding is that consumers will only ever have 1 in flight request to
each broker that has leader partitions of topics that it is subscribed to. The
fetch requests will ask for records for all leader partitions on the broker so
if the consumer is consuming from more than one pa
Using 2.3.1 Brokers makes things worse. There are now 2 fetch.max.wait.ms
delays before messages are delivered even though they were available at the
beginning.
2020-03-09 11:40:23,878 DEBUG
[org.apache.kafka.clients.consumer.internals.Fetcher]
'EE-ManagedThreadFactory-default-Thread-2' [Cons
Thanks for your response. Yes the second issue can be mitigated by reducing
the fetch.max.wait.ms although reducing it too far creates excessive CPU load
on the Brokers. However I've done some further testing and found what looks
like the underlying cause.
In the scenario below the Consumer i
Hi James,
3 Consumers in a group means you are having 20 partitions per consumer (as
per your 60 partition and 1 CGroup setup), 5 means 12. There's nothing
special about these numbers as you also noticed.
Have you tried setting fetch.max.wait.ms = 0 and see whether that's making
a difference for y
I’m seeing behaviour that I don’t understand when I have Consumers fetching
from multiple Partitions from the same Topic. There are two different
conditions arising:
1. A subset of the Partitions allocated to a given Consumer not being consumed
at all. The Consumer appears healthy, the Thread
