How to enable `kafka.(producer|consumer)` JMX metrics?

2018-01-03 Thread Tim Visher
Hello Everyone, I'm trying to get my datadog kafka integration working as expected and I noticed that the documented `kafka.(producer|consumer)` JMX MBeans do not appear to be available. I verified this by using JConsole to browse the MBeans. Sure enough, kafka.server, kafka.log, kafka.network et

Re: How to enable `kafka.(producer|consumer)` JMX metrics?

2018-01-03 Thread Tim Visher
:facepalm: They are in the producer and consumer JVMs, as would be totally expected. I'm only collecting JMX metrics from the brokers. Duh. Sorry for the noise. On Wed, Jan 3, 2018 at 12:01 PM, Tim Visher wrote: > Hello Everyone, > > I'm trying to get my datadog kafka int

Re: Any suggestion about Kafka/Zookeeper migration.

2018-01-04 Thread Tim Visher
On Wed, Jan 3, 2018 at 11:39 PM, Tony Liu wrote: > This post here is aimed to ask experience about what did you do migration > `Kafka/zookeeper` ? :) > > I think the easiest way to migrate kafka is to do a blue/green deploy. Stand up an entire new cluster, cut over producers to it, wait for the

Re: Any suggestion about Kafka/Zookeeper migration.

2018-01-04 Thread Tim Visher
getting something in front of kafka that can do a cut over for you like HAProxy or DNS. Obviously that's a future consideration though. For now, definitely go with the in place migration. > > > On Thu, Jan 4, 2018 at 6:06 AM, Tim Visher wrote: > > > On Wed, Jan 3, 2

Re: Upgrading Kafka from Version 0.10.2 to 1.0.0

2018-01-16 Thread Tim Visher
On Tue, Jan 9, 2018 at 4:50 PM, ZigSphere Tech wrote: > Is it easy to upgrade from Kafka version 0.10.2 to 1.0.0 or do I need to > upgrade to version 0.11.0 first? Anything to expect? > We just did (almost) exactly this upgrade. 2.11-0.10.1.0 to 2.11-1.0.0. The main issue we faced was the broke

Disadvantages of Upgrading Kafka server without upgrading client libraries?

2016-11-29 Thread Tim Visher
Hi Everyone, I have an install of Kafka 0.8.2.1 which I'm upgrading to 0.10.1.0. I see that Kafka 0.10.1.0 should be backwards compatible with client libraries written for older versions but that newer client libraries are only compatible with their version and up. My question is what disadvantag

log.dirs balance?

2016-11-29 Thread Tim Visher
Hello, My kafka deploy has 5 servers with 3 log disks each. Over the weekend I noticed that on 2 of the 5 servers the partitions appear to be imbalanced amongst the log.dirs. ``` kafka3 /var/lib/kafka/disk1 3 /var/lib/kafka/disk2 3 /var/lib/kafka/disk3 3 kafka5 /var/lib/kafka/disk1 3 /var/lib/kaf

Re: log.dirs balance?

2016-11-30 Thread Tim Visher
, just wanted to > share my painful experience. > > > > On 2016.11.29 21:35, Tim Visher wrote: > >> Hello, >> >> My kafka deploy has 5 servers with 3 log disks each. Over the weekend I >> noticed that on 2 of the 5 servers the partitions appear to be imba

Deleting a topic without delete.topic.enable=true?

2016-12-09 Thread Tim Visher
Hi Everyone, I'm really confused at the moment. We created a topic with brokers set to delete.topic.enable=false. We now need to delete that topic. To do that we shut down all the brokers, deleted everything under log.dirs and logs.dir on all the kafka brokers, `rmr`ed the entire chroot that kafk

Re: Deleting a topic without delete.topic.enable=true?

2016-12-09 Thread Tim Visher
topic.enable=true? > > > > On Dec 9, 2016 13:40, "Tim Visher" wrote: > > > > > Hi Everyone, > > > > > > I'm really confused at the moment. We created a topic with brokers set > to > > > delete.topic.enable=false. > > >

Re: Deleting a topic without delete.topic.enable=true?

2016-12-09 Thread Tim Visher
own (the consumers) and the topic came back. I'm glad that we were doing the right steps though. > > -Todd > > > On Fri, Dec 9, 2016 at 8:25 AM, Tim Visher wrote: > > > I did all of that because setting delete.topic.enable=true wasn't > > effective.

Re: Deleting a topic without delete.topic.enable=true?

2016-12-12 Thread Tim Visher
up > > recreating a topic that has been deleted as it issues a metadata request > to > > try and find out what happened after an offset request for the topic > fails. > > > > -Todd > > > > > > On Fri, Dec 9, 2016 at 8:37 AM, Tim Visher wrote