Kafka is pretty nice and as long as you have basic monitoring in place, doesn't take too much attention, but keep in mind that it still depends on zookeeper and I've seen that being the bottleneck. in the past. I also think as a single engineering person in your startup, if you don't need kafka or any other component, don't get it. All said and done, it still is one extra component to manage.
-- Sharninder On Mon, Feb 15, 2016 at 6:07 AM, Todd Snyder <tsny...@blackberry.com> wrote: > So long as you put some basic monitoring in place, it should run nicely > with very little intervention and let you be confident everything is as it > should be. > > Key things to watch: > > * disk space - a disk filling up really makes things difficult for you. > Make sure your retention fits your footprint. > * consumer lag > * isr (in sync replica) state - really, you just want to know about out of > sync replicas > * simple rates (produce/consume) to make sure things are sane. > * standard up/down monitoring (though isr status will typically catch dead > brokers) > > If you dont already have jmx monitoring tools, I suggest you look at > jmxtrans and graphite/grafana, or elk stack. Both will enable you to > monitor all the jmx stats available, and visualize easily, so you can > hopefully just fire and (mostly) forget. > > We run about 100 brokers, moving 60billion+ messages a day, in a few > different clusters, and aside from glancing at the dashboard occasionally, > there is really no day-to-day maintenance required. Kafka just works. > > Probably our biggest operational overhead is when it comes time to roll a > cluster to apply a change or do an upgrade. Waiting for everything to get > back in sync and stable before starting another broker means it can take a > couple days to complete an upgrade/change safely. > > Cheers > > Todd > > Sent from my BlackBerry 10 smartphone on the TELUS network. > Original Message > From: Kyle Mathews > Sent: Sunday, February 14, 2016 7:07 PM > To: users@kafka.apache.org > Reply To: users@kafka.apache.org > Subject: Kafka advice for small startup > > > Hi I'm the technical co-founder of a startup in San Francisco and I've been > investigating using Kafka for some time. > > I'm thinking about biting the bullet and integrating it soon into our > system. My main concern is how much operational overhead it'll add. I'm the > only engineer right now and am already stretched thin. Since Kafka isn't a > must have yet, it only makes sense to adopt it if it's pretty care free. > > What are people's experiences in similar situations to mine? Is Kafka > something you can setup and forget like say redis or is something I'll be > babysitting a decent bit and spending a lot of time on upgrades etc.? > > Also I'd love to take anyone in San Francisco with Kafka experience out to > lunch or coffee. > > Thanks! > Kyle > -- -- Sharninder
