Hi Valentin, Kafka is available on DC/OS in the Catalog (aka Universe) as part of the `kafka` package. Mesosphere has put a lot of effort into making Kafka work on DC/OS. Since Kafka requires persistent disk it's required to make sure after initial deployment brokers stay put on their assigned Mesos agents. Deployment and common ops tasks are supported with the help of the Kafka scheduler developed in the mesosphere/dcos-commons repo. For example, configuration changes to brokers can be made through the DC/OS Kafka service (through the UI or the CLI) and deployed out to brokers as a rolling upgrade, where one at a time each broker server.config's are updated and the server is cleanly bounced. The Kafka scheduler also supports other features such as upgrades for when Mesosphere releases a new scheduler update or when a new version of Kafka is available. Common ops tasks like replacing a failed broker or adding more brokers is supported by using the DC/OS CLI and Kafka scheduler configuration changes. In short, most of the the ops tasks are handled by the Kafka scheduler, but all other tasks are just Kafka as usual.
The biggest thing to watch out for is that running Kafka in DC/OS implies a shared mixed-use environment. It's possible other services could be running on the Mesos agents brokers are installed on, which could have resource conflicts, etc. By default DC/OS Kafka shares the ZooKeeper instances with Mesos and other services, you may want to consider a standalone cluster for Kafka. All these concerns can be mitigated with configuration, but you'll need to get familiar with DC/OS and the Kafka scheduler before you run anything in prod. Latest DC/OS Kafka release: https://docs.mesosphere.com/service-docs/kafka/2.0.1-0.11.0/ Regards, Sean On Tue, Oct 3, 2017 at 5:20 AM, Valentin Forst <valen...@aseno.de> wrote: > Hi Avinash, > > Thanks for this hint. > > It would have been great, if someone could share experience using this > framework on the production environment. > > Thanks in advance > Valentin > > > Am 02.10.2017 um 19:39 schrieb Avinash Shahdadpuri < > avinashp...@gmail.com>: > > > > There is a a native kafka framework which runs on top of DC/OS. > > > > https://docs.mesosphere.com/service-docs/kafka/ > > > > This will most likely be a better way to run kafka on DC/OS rather than > > running it as a marathon framework. > > > > > > -- Senior Software Engineer, Lightbend, Inc. <http://lightbend.com> @seg1o <https://twitter.com/seg1o>
