compaction + delete not working for me

2017-01-06 Thread Mike Gould
Hi I'm trying to configure log compaction + deletion as per KIP-71 in kafka 0.10.1 but so far haven't had any luck. My tests show more than 50% duplicate keys when reading from the beginning even several minutes after all the events were sent. The documentation in section 3.1 doesn't seem very cle

Re: compaction + delete not working for me

2017-01-10 Thread Mike Gould
shows that my test doesn't match real life. 😃 On Fri, 6 Jan 2017 at 21:36, Ewen Cheslack-Postava wrote: > On Fri, Jan 6, 2017 at 3:57 AM, Mike Gould wrote: > > > > > Hi > > > > > > I'm trying to configure log compaction + deletion as per KIP-71 in

Streams error handling

2017-04-13 Thread Mike Gould
Hi Are there any better error handling options for Kafka streams in java. Any errors in the serdes will break the stream. The suggested implementation is to use the byte[] serde and do the deserialisation in a map operation. However this isn't ideal either as there's no great way to handle excep

Streams error handling

2017-04-13 Thread Mike Gould
Hi Are there any better error handling options for Kafka streams in java. Any errors in the serdes will break the stream. The suggested implementation is to use the byte[] serde and do the deserialisation in a map operation. However this isn't ideal either as there's no great way to handle excep

Re: Streams error handling

2017-04-13 Thread Mike Gould
I can't give you a better answer right now :( > > > -Matthias > > > On 4/13/17 8:16 AM, Mike Gould wrote: > > Hi > > Are there any better error handling options for Kafka streams in java. > > > > Any errors in the serdes will break the stream. The sugge

Partition assignment with multiple topics

2017-05-23 Thread Mike Gould
Hi We have a couple of hundred topics - each carrying a similar but distinct message type but to keep the total partition count down each only has 3 partitions. If I start Kafka-streams consuming all topics only 3 threads ever get assigned any partitions. I think the first thread to start gets th

Re: Streams error handling

2017-05-23 Thread Mike Gould
one you described in your > >> email, but its on our roadmap to improve the API -- and its priority got > >> just increase by your request. > >> > >> I am sorry, that I can't give you a better answer right now :( > >> > >> > >> -Matth

Re: Streams error handling

2017-05-23 Thread Mike Gould
Thu, Apr 13, 2017 at 9:13 PM, Mike Gould wrote: > > > Great to know I've not gone off in the wrong direction > > Thanks > > > > On Thu, 13 Apr 2017 at 16:34, Matthias J. Sax > > wrote: > > > > > Mike, > > > > > > thanks for y

Re: Partition assignment with multiple topics

2017-05-23 Thread Mike Gould
case. > > If not, your other option is to break up your application into a number of > KafkaStreams instances, each dealing with a subset of topics. > Hope that helps. > Michał > > > On 23/05/17 08:47, Mike Gould wrote: > > Hi > We have a couple of hundred topics -

Re: Streams error handling

2017-05-24 Thread Mike Gould
tracking this and other improvements in > exception handling at https://issues.apache.org/jira/browse/KAFKA-5156 < > https://issues.apache.org/jira/browse/KAFKA-5156>. > > Thanks > Eno > > On 23 May 2017, at 17:31, Mike Gould wrote: > > > > That's great for