Investigating apparent data loss during preferred replica election

2016-11-17 Thread Mark Smith
old, can anybody explain what happened? I'm happy to provide more logs or whatever. Thanks! -- Mark Smith m...@qq.is

Re: Investigating apparent data loss during preferred replica election

2016-11-18 Thread Mark Smith
Correct, we've disabled unclean leader election. There were also no log messages from an unclean election. I believe that Kafka thinks it performed a clean election and still lost data. -- Mark Smith m...@qq.is On Thu, Nov 17, 2016, at 06:23 PM, Tauzell, Dave wrote: > Do

Re: Investigating apparent data loss during preferred replica election

2016-11-21 Thread Mark Smith
rg/jira/browse/KAFKA-1211 * I've read through this but I'm not entirely sure if it addresses the above. I don't think it does, though. I don't see a step in the ticket about become-leader making a call to the old leader to get the latest generation snapshot? -- Mark Smith m...@

Re: Investigating apparent data loss during preferred replica election

2016-11-21 Thread Mark Smith
this case, I still don't think any discussion about multiple- failovers is germane to the problem we saw. Each of our partitions only had a single failover, and yet 4 of them still truncated committed data. -- Mark Smith m...@qq.is On Mon, Nov 21, 2016, at 05:12 PM, Jun Rao wrote:

Re: Investigating apparent data loss during preferred replica election

2016-11-22 Thread Mark Smith
kes sense and means my understanding was wrong and this wasn't an issue. Thanks for helping clear that up. This means there is still an unresolved issue, unfortunately. I can replicate the conditions that led to it and see if I can reproduce the problem. If so, I'll update this thread

Quick replication question - loss of committed messages during preferred replica election

2017-01-18 Thread Mark Smith
this is being referred to sort-of in Scenario 1, however, that scenario is mentioning broker failure -- and my concern is that data loss is possible even in the normal scenario with no broker failures. Any thoughts? -- Mark Smith m...@qq.is

Re: Quick replication question - loss of committed messages during preferred replica election

2017-01-23 Thread Mark Smith
of our 'data loss', which isn't actually loss but a bad interaction of failover and catching a stale HWM leading to errors being thrown by the broker when it maybe doesn't need to. Thoughts? -- Mark Smith m...@qq.is On Wed, Jan 18, 2017, at 02:11 PM, Jun Rao wrote: >