So long as you put some basic monitoring in place, it should run nicely with very little intervention and let you be confident everything is as it should be.
Key things to watch: * disk space - a disk filling up really makes things difficult for you. Make sure your retention fits your footprint. * consumer lag * isr (in sync replica) state - really, you just want to know about out of sync replicas * simple rates (produce/consume) to make sure things are sane. * standard up/down monitoring (though isr status will typically catch dead brokers) If you dont already have jmx monitoring tools, I suggest you look at jmxtrans and graphite/grafana, or elk stack. Both will enable you to monitor all the jmx stats available, and visualize easily, so you can hopefully just fire and (mostly) forget. We run about 100 brokers, moving 60billion+ messages a day, in a few different clusters, and aside from glancing at the dashboard occasionally, there is really no day-to-day maintenance required. Kafka just works. Probably our biggest operational overhead is when it comes time to roll a cluster to apply a change or do an upgrade. Waiting for everything to get back in sync and stable before starting another broker means it can take a couple days to complete an upgrade/change safely. Cheers Todd Sent from my BlackBerry 10 smartphone on the TELUS network. Original Message From: Kyle Mathews Sent: Sunday, February 14, 2016 7:07 PM To: users@kafka.apache.org Reply To: users@kafka.apache.org Subject: Kafka advice for small startup Hi I'm the technical co-founder of a startup in San Francisco and I've been investigating using Kafka for some time. I'm thinking about biting the bullet and integrating it soon into our system. My main concern is how much operational overhead it'll add. I'm the only engineer right now and am already stretched thin. Since Kafka isn't a must have yet, it only makes sense to adopt it if it's pretty care free. What are people's experiences in similar situations to mine? Is Kafka something you can setup and forget like say redis or is something I'll be babysitting a decent bit and spending a lot of time on upgrades etc.? Also I'd love to take anyone in San Francisco with Kafka experience out to lunch or coffee. Thanks! Kyle
