Re: Causes for Kafka messages being unexpectedly delivered more than once? The 'exactly once' semantic

2017-04-13 Thread Timur Fayruzov
Very enlightening presentation, thanks for sharing! On Thu, Apr 13, 2017 at 9:07 AM, Thakrar, Jayesh < jthak...@conversantmedia.com> wrote: > Hi Dmitri, > > This presentation might help you understand and take appropriate actions > to deal with data duplication (and data loss) > > https://www.sli

Single-replication topic behavior on failures

2017-04-12 Thread Timur Fayruzov
Hello, We have a number of topics with replication factor of 1 and a single partition. I want to update the cluster from 0.10.0 to 0.10.1 by taking nodes down one by one, but it is not clear to me what is the behavior will be for these topics. Brokers have topic auto-create setting set to true. W

Unexpected behavior when offset expires

2016-12-29 Thread Timur Fayruzov
Hello, Using kafka 0.8.2.1 with reactive-kafka wrapper around Java client we observe that some consumers in a group are getting stuck every once in a while. One characteristic of these stuck consumers is that (once restarted) they are attached to partitions with expired offsets, i.e. I see the fol

Re: Offset sporadically being reset

2016-10-19 Thread Timur Fayruzov
Additionally, on consumer I observe a strange behavior: it is being constantly rebalancing. There are no errors and each rebalance succeeds, but as soon as one is finished the next one is started. On Wed, Oct 19, 2016 at 4:36 PM, Timur Fayruzov wrote: > Hello, > > I run Kafka 0.8.2.

Offset sporadically being reset

2016-10-19 Thread Timur Fayruzov
Hello, I run Kafka 0.8.2.2 cluster with 3 nodes and recently started to observe strange behavior on select topics. The cluster runs in-house as well as most consumers. I have started some consumers in AWS and they _mostly_ work fine. Occasionally, I end up in a state where when I run kafka-consume