backup(dump) and restore environment

Hi I'm looking for the best way to "dump" current system state, and recreate it on the new, autonomic environment. Lets say I'd like to create a copy of Production, and based on that, create new, separate environment for testing. Can You suggest some solutions? greetings Jakub

Re: backup(dump) and restore environment

Hello Jakub, Maybe it will work for you to combine MirrorMaker https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 and Burrow: https://github.com/linkedin/Burrow See recent announcement for Burrow http://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCACrdVJpS3k

Re: backup(dump) and restore environment

Thanks for the quick anwser. The only think I'm not happy about is _no_ env separation. As far as I do understand, this is "online" operation, am I right? So the best would be to create mirror, and then cut it off form Production. greetings Jakub 2015-06-09 11:17 GMT+02:00 Stevo Slavić : > Hel

Query on kafka broker and topic metadata

Hi, I am new to kafka and I have a doubt. How to read specified broker and topic statistics from kafka server? I want to read below parameters about existing topic from kafka. 1) How many activeMessages 2) How many activeSubscriptions 3) How many totalMessages 4) How many totalSubscriptions 5)

Kafka topic registration documentation.

Hi, I was wondering if the process of new topic registration is described somewhere. I'm using Go client to talk to kafka cluster and it would be awesome to ensure topic existence. I know there is auto creation configuration flag, but using this there is no way to use custom configuration for ever

Mirror maker doesn't rebalance after getting ZkNoNodeException

Hi, I have two mirror makers A and B both subscripting to the same whitelist. During topic rebalancing one of the mirror maker A encountered ZkNoNodeException and then stopped all connections. but mirror maker B didn't pick up the topics that were consumed by A and left some of the topics unassign

ConsumerFetcherManager MaxLag

Hi - I am using kafka 0.8.2.1 I was hoping to monitor overall lag (for all partitions) for a high level consumer via JMX, but ConsumerFetcherManager MaxLag always returns value 0, despite the fact that I can see consumer offsets updating in zk (consumer is connected properly), and kafka.tools.Co

Re: Query on kafka topic metadata

Hi Pavan, 1) you cannot read the current number of messages for a topic, but you can read the number of bytes for a topic in Kafka metrics. 2) you cannot read the current number of subscribers directly. However you can read from ZK and parse the results to get the subscriber counts. 3) / 4) I am

Re: Mirror maker doesn't rebalance after getting ZkNoNodeException

Which version of MM are you running? On 6/9/15, 4:49 AM, "tao xiao" wrote: >Hi, > >I have two mirror makers A and B both subscripting to the same whitelist. >During topic rebalancing one of the mirror maker A encountered >ZkNoNodeException and then stopped all connections. but mirror maker B >di

offsets.storage=kafka, dual.commit.enabled=false still requires ZK

We are setting up a new Kafka project (0.8.2.1) and are trying to go straight to consumer offsets stored in Kafka. Unfortunately it looks like the Java consumer will try to connect to ZooKeeper regardless of the settings. Will/When will this dependency go away completely? It would simplify our dep

Re: offsets.storage=kafka, dual.commit.enabled=false still requires ZK

The existing consumer uses Zookeeper both to commit offsets and to assign partitions to different consumers and threads in the same consumer group. While offsets can be committed to Kafka in 0.8.2 releases, which greatly reduces the load on Zookeeper, the consumer still requires Zookeeper to manage

Stale topic for existing producer

BackgroundWe have a few consumer that *used* to consume a particular topic. This topic still exist and the consumers still exist, however, these consumer no longer subscribe to the topic. When we check the offset for the Consumer Group, kafka.tools.ConsumerOffsetChecker, ZK believes this tier

Re: offsets.storage=kafka, dual.commit.enabled=false still requires ZK

The new consumer implementation, which should be included in 0.8.3, only needs a bootstrap.servers setting and does not use a zookeeper connection. On Tue, Jun 9, 2015 at 1:26 PM, noah wrote: > We are setting up a new Kafka project (0.8.2.1) and are trying to go > straight to consumer offsets st

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

Hi Todd, Thanks for open sourcing this, I'm excited to take a look. It looks like it's specific to offsets stored in kafka (and not zookeeper) correct? I assume by that that LinkedIn is using the kafka storage now in production? Jason On Thu, Jun 4, 2015 at 9:43 PM, Todd Palino wrote: > I am

producer metadata behavior when topic not created

Hi, I am talking about the 0.8.2 java producer. In our deployment, we disables auto topic creation, because we would like to control the precise number of partitions created for each topic and the placement of partitions (e.g. zone-aware). I did some experimentation and checked the code. metadat

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

For mirror maker and our audit application, we've been using Kafka-committed offsets for some time now. We've got a few other consumers who are using it, but we haven't actively worked on moving the bulk of them over. It's been less critical since we put the ZK transaction logs on SSD. And yeah, t

Re: Mirror maker doesn't rebalance after getting ZkNoNodeException

I use commit 9e894aa0173b14d64a900bcf780d6b7809368384 from trunk code On Wed, 10 Jun 2015 at 01:09 Jiangjie Qin wrote: > Which version of MM are you running? > > On 6/9/15, 4:49 AM, "tao xiao" wrote: > > >Hi, > > > >I have two mirror makers A and B both subscripting to the same whitelist. > >Du

Usage of Kafka Mirror Maker

Hi Team, I would like to know under which all circumstances I can use MirrorMaker process ? i read on a website that : "*it is not really intended as a fault-tolerance mechanism*" , so curious to know where all i can think of using MirrorMaker. Also, how to know the version of MirrorMaker proces