Re: Highwater mark interpretation

2020-06-20 Thread D C
will be in sync. Cheers, D On Sunday, June 21, 2020, Nag Y wrote: > Thanks D C. Thanks a lot . That is quite a detailed explanation. > If I understand correctly, ( ignoring the case where producers > create transactions) - since the replica is down and never comes , the high > wate

Re: Highwater mark interpretation

2020-06-20 Thread D C
The short answer is : yes, a consumer can only consume messages up to the High Watermark. The long answer is not exactly, for the following reasons: At the partition level you have 3 major offsets that are important to the health of the partition and accessibility from the consumer pov: LeO (log

Re: Last Stable Offset (LSO) stuck for specific topic partition after Broker issues

2020-06-11 Thread D C
Hey peeps, Anyone else encountered this and got to the bottom of it? I'm facing a similar issue, having LSO stuck for some partitions in a topic and the consumers can't get data out of it (we're using read_committed = true). When this issue started happening we were on kafka 2.3.1 i tried: - res

Re: Last Stable Offset (LSO) stuck for specific topic partition after Broker issues

2020-06-10 Thread D C
Hey peeps, Anyone else encountered this and got to the bottom of it? I'm facing a similar issue, having LSO stuck for some partitions in a topic and the consumers can't get data out of it (we're using read_committed = true). When this issue started happening we were on kafka 2.3.1 i tried: - res

Separating internal and external traffic

2016-05-25 Thread D C
I'm sure i can do this but I'm just not stumbling on the right documentation anywhere. I have a handful of kafka servers that I am trying to get ready for production. I'm trying separate the internal and external network traffic, but I don't see how to do it. Each host has two addresses. 10.x.y.z