connecting to multiple kafka clusters with kerberos enabled

2016-10-30 Thread Digumarthi, Prabhakar Venkata Surya
We need to use the following configuration to connect to kafka cluster , I should I configure a single client to connect to multiple clusters with different JAAS configuration. Is it possible to do that ? KafkaClient { com.sun.security.auth.module.Krb5LoginModule required u

Running Zookeeper on same instance as kafka brokers in multi node cluster

2016-09-19 Thread Digumarthi, Prabhakar Venkata Surya
Hi Team, What are the downsides of installing Zookeeper and kafka on same machine, in multi broker environment? We are trying to install Zookeeper and kafka in AWS world and its becoming difficult for us to maintain ZK and Kafka with some issues. Also re-provisioning ZK and Kafka instances s

kafka 0.9 running on AWS environment caching zookeeper IP instead of using the DNS

2016-08-15 Thread Digumarthi, Prabhakar Venkata Surya
Hi All, We are using kafka version 0.9 and we are facing issues, in kafka connecting to zookeeper. Here are mode details: Zookeeper is in front AWS Elastic Load Balancer and the ELB has been associated with a DNS name, say zookerper1.com. Our broker configuration looks the below zookeeper.

keeping a high value for zookeeper.connection.timeout.ms

2016-08-08 Thread Digumarthi, Prabhakar Venkata Surya
Hi All, Right now we are running kafka in AWS EC2 servers and zookeeper is also running on separate EC2 instances. We have created a service (system units ) for kafka and zookeeper to make sure that they are started in case the server gets rebooted. The problem is sometimes zookeeper severs

Using automatic brokerId generation

2016-08-02 Thread Digumarthi, Prabhakar Venkata Surya
Hi , I am right now using kafka version 0.9.1.0 If I choose to enable automatic brokerId generation, and let’s say if one of my broker dies and a new broker gets started with a different brokerId. Is there a way I can get the new broker Id part of the replica set of a partition automatically?

Kafka In cloud environment

2016-08-02 Thread Digumarthi, Prabhakar Venkata Surya
In case I use automatic brokerId generation and if a broker dies, and a new broker is added with a different broker Id . Will the replica set gets updated automatically ? The information contained in this e-mail is confidential and/or pro

Using the new consumer client API 0.0.9

2016-03-29 Thread Digumarthi, Prabhakar Venkata Surya
Hi All, We are planning to use consumer client API 0.0.9, but the kafka documentation still says its beta quality. Is there any timeline by when it becomes production ready ? The reason why we want to go with 0.9 is to avoid rework of migrating from 0.8.X client to 0.9. Note: we are runnin