Kafka on dedicated hosts running in docker under marathon under Mesos. It was a real bear to get working, but is really beautiful once I did manage to get it working. I simply run with a unique hostname constraint and number of instances = replication factor. If a broker dies and it isn't a hardware or network issue, marathon restarts it.
The hardest part was that Kafka was registering to ZK with the internal (to docker) port. My workaround was that you have to use the same port inside and outside docker or it will register to ZK with whatever the port is inside the container. FYI this is an on premise dedicated Mesos cluster running on bare metal :) On Friday, February 27, 2015, James Cheng <jch...@tivo.com> wrote: > Hi, > > I know that Netflix might be talking about "Kafka on AWS" at the March > meetup, but I wanted to bring up the topic anyway. > > I'm sure that some people are running Kafka in AWS. Is anyone running > Kafka within docker in production? How does that work? > > For both of these, how do you persist data? If on AWS, do you use EBS? Do > you use ephemeral storage and then rely on replication? And if using > docker, do you persist data outside the docker container and on the host > machine? > > And related, how do you deal with broker failure? Do you simply replace > it, and repopulate a new broker via replication? Or do you bring back up > the broker with the persisted files? > > Trying to learn about what people are doing, beyond "on premises and > dedicated hardware". > > Thanks, > -James > > -- Text by Jeff, typos by iPhone
