Re: Log file of server start up error

2015-05-24 Thread Sanjay Mistry
Hello, When run this command "bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testing". i am getting "INFO Closing socket connection to /127.0.0.1. (kafka.network.Processor)" error in server konsol. Please help me. On Sat, May 23, 2015 at 12:19 PM, Sanjay Mistry wrote: > [2

Re: Replication tools to move topics/partitions gradually

2015-05-24 Thread Todd Palino
See, this is why I should never say anything :) The version I have right now is very limited - it only does a clone (we needed it for some hardware testing) and a leader balance (does it using partition reassignment without actually moving partitions). We have some scripts that the other SREs h

Re: Kafka broker - Ip-address instead of host naem

2015-05-24 Thread Achanta Vamsi Subhash
Oh ok. Got it. Thanks Gwen and Daniel. On Mon, May 25, 2015 at 5:15 AM, Daniel Compton < daniel.compton.li...@gmail.com> wrote: > Correct me if I'm wrong, but I thought that the zk connect config was only > needed explicitly at startup, and the zk cluster would update the active > zookeepers as t

Re: Replication tools to move topics/partitions gradually

2015-05-24 Thread Henry Cai
Todd, This is very promising. Do you know when will we be able to see your tools released to public? On Sun, May 24, 2015 at 7:54 PM, Todd Palino wrote: > We've built tools on top of it that both build the list based on less > information (like "clone this broker to that one") and break it dow

Re: Log file of server start up error

2015-05-24 Thread Jaikiran Pai
Hi Sanjay, Did you check that no other Kafka process is using the /tmp/kafk-logs folder? What command(s) did you use to verify that? -Jaikiran On Saturday 23 May 2015 12:19 PM, Sanjay Mistry wrote: [2015-05-23 12:16:41,624] INFO Initiating client connection, connectString=localhost:2181 sessi

Re: Replication tools to move topics/partitions gradually

2015-05-24 Thread Todd Palino
We've built tools on top of it that both build the list based on less information (like "clone this broker to that one") and break it down into a configurable number of discrete moves so it doesn't tank the cluster. And yes, I've finally started the process of departing them from the LinkedIn-s

Replication tools to move topics/partitions gradually

2015-05-24 Thread Henry Cai
We have a kafka cluster with 10 brokers and we are using the kafka replication tool (kafka-reassign-partitions.sh) when we need to add more brokers to the cluster. But this tool tends to move too many topic/partitions around at the same time which causes instability. Do we have an option to do it

Re: Kafka broker - Ip-address instead of host naem

2015-05-24 Thread Daniel Compton
Correct me if I'm wrong, but I thought that the zk connect config was only needed explicitly at startup, and the zk cluster would update the active zookeepers as they joined and exited. You only need to specify one zk to join the cluster, and it will bootstrap the rest. So zk changes won't require

Re: Architecture for multiple consumers of a given message

2015-05-24 Thread Daniel Compton
Hi Warren If you're using the high level consumer, then you can just have multiple consumer groups (one for each purpose), and run 1 consumer thread per consumer group. On Mon, 25 May 2015 at 8:43 am Warren Henning wrote: > I'm working on a simple web application where I want the same message of

Architecture for multiple consumers of a given message

2015-05-24 Thread Warren Henning
I'm working on a simple web application where I want the same message of a given topic to be consumed by multiple separate consumers (one that writes to a database, another that writes to a search index like Solr/Elasticsearch). Right now as this is just a small personal project I only have one pa

Re: Kafka broker - Ip-address instead of host naem

2015-05-24 Thread Achanta Vamsi Subhash
Ok. I was talking about a scenario where there is no DNS/hostNames for the zk nodes. If the connection string is given with all ip addresess of zk hosts and a new host is brought up in the zk cluster replacing a old node with a different ip address, we still need to re-load the zk connection strin

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

2015-05-24 Thread Adam Dubiel
Hi Daniel, First of all sorry for late response, i enjoyed short vacation :) I guess the documentation might be bit misleading here, and so we should improve it: we do not aim (and can't) provide higher guarantees than Kafka. We want to be as bullteproofs as possible in REST interface segments.

Re: Kafka broker - Ip-address instead of host naem

2015-05-24 Thread Gwen Shapira
You can't dynamically re-load server properties. However, the norm in zookeeper is to configure the connection string with all the nodes in the zk cluster, so there will be no need to modify properties when you replace zk nodes. On Sun, May 24, 2015 at 4:13 PM, Achanta Vamsi Subhash < achanta.va.

Re: Kafka broker - Ip-address instead of host naem

2015-05-24 Thread Achanta Vamsi Subhash
Thanks Gwen. One more question: Is there a way we can dynamically re-load the contents of the Kafka server.properties contents without restarting the broker? Example use-case is when a zk-node goes down and a new one is brought up, we need to update the server.properties file to reflect this. Curr

Re: Kafka broker - Ip-address instead of host naem

2015-05-24 Thread Gwen Shapira
If you set advertised.hostname in server.properties to the ip address, the IP will be registered in ZooKeeper. On Fri, May 22, 2015 at 2:20 PM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi, > > Currently Kakfa brokers register the hostname in zookeeper. > > [zk: localhost:2181