Kafka stops on cleaner-offset-checkpoint not found

2019-01-01 Thread Gerrit Avenant
Hi, My Kafka service just stops after about running for a week to 2 weeks. It seems the cleaner-offset-checkpoint gets deleted and then it fails upon reading the file. Restarting the service creates the cleaner-offset-checkpoint again. I need ideas of where I can start debugging. It does not look

UnknownProducerIdException problems

2019-01-01 Thread 이도현
I'm using kafka 1.1.1 and idempotent producer. UnknownProducerIdException occurs particular topics when it is a low traffic situation such as under 10 logs per day. Increasing the retention of the topics reduce the frequency of occurrence. ERROR [ReplicaManager broker=1] Error processing append op

UnknownProducerIdException problems

2019-01-01 Thread 이도현
I'm using kafka 1.1.1 and idempotent producer. UnknownProducerIdException occurs particular topics when it is a low traffic situation such as under 10 logs per day. Increasing the retention of the topics reduce the frequency of occurrence. ERROR [ReplicaManager broker=1] Error processing append op

Re: Monitoring Mirror Maker Lag & source

2019-01-01 Thread Pere Urbón Bayes
Hi, as otis have already shared this is accessible through the jmx metrics in Apache Kafka. You can find a comprehensive list from https://kafka.apache.org/documentation/#monitoring at the end of the day you can pull this into any monitoring solution that can pull from jmx, for example Jolokia+E

Re: Kafka tests on a remote cluster

2019-01-01 Thread Pere Urbón Bayes
Hi, if I understand your question properly you are aiming to validate failure scenarios. I usually see this done for learning purposes, basically to answer how Kafka would react to such and such situation? or to validate the current setup / configuration an Organisation have. For the first one, o

A Kafka cluster health api idea, does it makes sense?

2019-01-01 Thread Pere Urbón Bayes
Good night, After being for a long time working with Elasticsearch one gets used to it’s great REST api where you can check all sort of things from the server. As I usually get ask during my day life helping companies with their Kafka I decided to build an small Kafka Health API POC. This is a