I’m not sure how your requirements of Kafka are related to your requirements
for marathon. Kafka is a streaming-log system and marathon is a scheduler.
Mesos, as your resource manager, simply “manages” resources. Are you asking
about multitenancy? If so, I highly recommend that you separate your Kafka
cluster (and zookeeper) from your other services. Kafka leverages the OS page
cache to optimize read performance and it seems likely this would interfere
with Mesos resource management policy.
-David
On 10/2/17, 6:39 AM, "Valentin Forst" <valen...@aseno.de> wrote:
Hi there,
Working in a huge compony we are about to install Kafka on DC/OS (Mesos)
and intend to use Marathon as a Scheduler. Since I am new to DC/OS and
Marathon, I was wondering if this is a recommended way of using Kafka in the
production environment.
My doubts are:
- Kafka manages Broker rebalancing (e.g. Failover, etc.) using its own
semantic. Can I trust Marathon that it will match the requirements here?
- Since our Container Platform - DC/OS is going to be used by other „micro
services“ - soon or later this is going to raise a performance issue. Should we
better use a dedicated DC/OS instance for our Kafka-Cluster? Or Kafka-Cluster
on its own?
- Is there something else we should consider important if using Kafka on
DC/OS + Marathon?
Thanks in advance for your time.
Valentin
