Re: Message sent ordering guarantees

2016-09-01 Thread Gerard Klijs
For async you could set ack to -1, but it keep slow, because it has to wait for the broker(s), to know it is received in order, before sending the next one, you need this if order is very important. When sending async the order gets changed in case the leader becomes temporarily unavailable and oth

Re: [jira] [Created] (KAFKA-4078) VIP for Kafka doesn't work

2016-08-23 Thread Gerard Klijs
If you change the url the broker is available on, you need to change the public advertised hostname in the configuration of the broker, this is not a bug, there is no way the broker could know how it can be reached from the outside, and when it changes. On Tue, Aug 23, 2016 at 10:33 AM chao (JIRA)

Re: Jars in Kafka 0.10

2016-07-29 Thread Gerard Klijs
No, if you don't use streams you don't need them. If you have no clients (so also no mirror maker) running on the same machine you also don't need the client jar, if you run zookeeper separately you also don't need those. On Fri, Jul 29, 2016 at 4:22 PM Bhuvaneswaran Gopalasami < bhuvanragha...@gm

Re: [jira] [Commented] (KAFKA-3841) MirrorMaker topic renaming

2016-06-15 Thread Gerard Klijs
We have also something similar, it's real easy to use a message handler, in our case we check for a bit to determine if we already copied the event, and change the topic name in dev-mode. You can configure a string to pass to your messageHandler for each instance. On Wed, Jun 15, 2016 at 1:38 AM N

Re: Migrating from 07.1 .100

2016-06-13 Thread Gerard Klijs
There has been a lot of changes, and it can also be quit a challenge to get the SSL working. And with only a null-pointer, there is little to go on. I would first focus on getting it to work with 0.10 (if possible also on a clustered test setup), and if it all works, try to configure the ssl. On M

Re: [jira] [Updated] (KAFKA-3772) MirrorMaker crashes on Corrupted Message

2016-06-01 Thread Gerard Klijs
Just had a look at the code, as I would like some way to prevent such a scenario happening to us. It seems you can't prevent the mirror maker from exiting. In the 0.10 mirror maker, any exception which is not an ConsumerTimeoutException, or an WakeupException, will cause the whole mirror maker to s

Re: Migrating Kafka from old VMs to new VMs in a different Cluster

2016-05-11 Thread Gerard Klijs
Depends on your use case but I guess something like this: - Install al fresh on the new VM's - Start a mirror maker in the the new VM's to copy data from the old ones - Be sure it's working right - Shut down the old VM's and start using the new ones The last step is the trickiest and depends a lot

Re: [jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka

2016-03-21 Thread Gerard Klijs
I had good experiences using the vagrant setup as it is on a mac, but did had to change some things. We are using docker now. I'm not sure about the general preference, but I would like a docker compose over the vagrant setup. Don't know if you really want it Kafka itself, and to give it support th

Re: [jira] [Created] (KAFKA-3377) add REST interface to JMX

2016-03-10 Thread Gerard Klijs
I would like to know why you want/need it to be integrated into Kafka? For our current project we tried out zabbix, https://www.zabbix.com/documentation/3.0/manual/config/items/itemtypes/jmx_monitoring, it takes some configuration, but then you can fetch all the jmx you want and put them into graph

Re: [jira] [Commented] (KAFKA-3044) Consumer.poll doesnot return messages when poll interval is less

2015-12-29 Thread Gerard Klijs
You can also set fetch.min.bytes, It will be a trade-off between getting the messages as fast as possible, with fetch.min.bytes=0 and the poll at 0. This will result in a lot of empty returns, and a lot of io overhead as compared to getting a lot of massages in one go. If you for example set the fe