Can't resolve under-replicated partitions,

2018-09-27 Thread Shannon Carey
For some reason, I am not able to get the “under-replicated partitions” metric on my Kafka cluster to zero across all nodes. Even after I manually reassign all the partitions, one server still has 928 under-replicated partitions. Also, the number of partitions each server is leading is very unev

KafkaStreams: ProducerFencedException causing StreamThread death

2018-09-27 Thread Aravind Dongara
During a rebalance triggered by kafka-coordinator-heartbeat-thread losing connection to ‘Group coordinator’, we noticed that a stream thread is shutting down when it catches a ProducerFencedExcpetion while flushing the state store. This also causes the stream-state on that node to be stuck in ‘

Managing/Versioning Topic Configurations for CI/CD

2018-09-27 Thread Patrik Kleindl
Hello everyone, we are currently trying to improve the management of our topic configurations. At the moment we are managing the configurations on the producer side and checking/creating/changing topics on each application (instance) startup. This has worked fine for many cases, but does not real

Re: How to setup Kafka security ? but only for some

2018-09-27 Thread Tobias Eriksson
What I mean by internal services, is our own company has a number of producers and consumers, and I do not want these “applications” to use SSL/ACL as they are already trusted and inside our network External is any 3;rd party system that we bring in to do “whatever” -Tobias On 2018-09-27, 10:22,

Re: How to setup Kafka security ? but only for some

2018-09-27 Thread M. Manna
There is a good tutorial written by Ismael Juma on SSL/ACL/SASL etc. setup - https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ if I get this right, you are trying to do the following: 1) "Internal Services" - mean inter-broker exchanges? 2) "3rd Party ap

How to setup Kafka security ? but only for some

2018-09-27 Thread Tobias Eriksson
We have Kafka v1.1.0 Is there a really good tutorial somewhere on how to set up security with SSL and ACL I would like to have ONE cluster, where * Our internal services does not have to use SSL / ACL * The 3;rd party applications HAVE TO use SSL / ACL is this possible ? -Tobias -- Tobias Erik