Note you shouldn't cross-post to both the users and dev list -- this kind of question belongs on the user list.
The fundamental things you need to go investigate: * Kubernetes Stateful Sets * Kafka packaged for use on Kubernetes -- I have been happy with https://github.com/Yolean/kubernetes-kafka. Even if you're interested in a more DIY solution, you can get an understanding about how they manage things like `broker.id` assignment (see the config in https://github.com/Yolean/kubernetes-kafka/blob/master/kafka/10broker-config.yml for example). If you want a more automated solution, then look into Kubernetes operators for managing Kafka on Kubernetes. There are quite a few of these. An open source one is Strimzi (https://github.com/strimzi/strimzi-kafka-operator), and Confluent is working on one too. Hope that gets you started. Regards, Raman On Thu, Apr 4, 2019 at 7:21 AM M. Manna <manme...@gmail.com> wrote: > > Hello, > > The question might trigger people to reply with "Confluent" - but it's not > related to confluent as the kubernetes offering is not for publi/community > edition. So, discussing Helm charts and intro to Confluent isn't our > objective here. > > What I am trying to understand is how does the log files (kafka message > logs, consumer offsets) are managed in Kubernetes (e.g. persistent volume, > statefulsets, etc). I have a 3 node cluster running over 3 physical Linux > VMs, and would like to move this setup to Kubernetes. > > The only part where we are strugging is with the following: > > 1) how does logs.dir configuration work per Pod? > 2) Assuming I have 3 PODs (3 brokers), and one of the Pod Goes down - how > do I manage the message log and offset files ? If a POD goes does, that > means it will delete everything in the logs.dir location, won't it? > 3) I am assuming broker.id will be supplied using some form of configMap, > but if there is anything worth knowing here, please do share. > > We have a reliable service on bare metal cloud, so we don't want to disrupt > it unless we are sure about the changes. > > Does anyone have any experience with this? If so, it would be great if you > can share any gists or configs. > > Much appreciated. > > Regards,
