Broker Server Crash with HW failure. Broker throwing java.lang.NumberFormatException and will not restart without removing all partitions
About a week ago one of our brokers crashed with a hardware failure. When the server restarted the Kafka broker would not start. The error is listed below. I tried a couple of time to restart, but no success. The only think that work was to physically remove the partitions for this broker. I was able to restart the broker. The partitions were recreated and were in-sync within about 30 minutes. My questions for the group are: Has anyone seen this error before? If so, what was your resolution? Is there a more elegant way to "re-sync" a broker node? Is there a way to identify which partition could be causing the issue and possibly just remove that one? We have seen this a couple of other time in our environment, twice from HW failures and once from someone killing the broker with a KILL -9. Is this a known issue? If anyone has any insight as to why this would occur I would greatly appreciate it. Gene Robichaux Manager, Database Operations Match.com 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 <> 02/18/2015 09:43:11 PM | FATAL | KafkaServerStartable | Fatal error during KafkaServerStable startup. Prepare to shutdown java.lang.NumberFormatException: For input string: " ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^
Re: Broker Server Crash with HW failure. Broker throwing java.lang.NumberFormatException and will not restart without removing all partitions
Hi Gene, Looks like you might be running into this https://issues.apache.org/jira/browse/KAFKA-1758 . -Harsha On Tue, Feb 24, 2015, at 07:17 AM, Gene Robichaux wrote: > About a week ago one of our brokers crashed with a hardware failure. When > the server restarted the Kafka broker would not start. The error is > listed below. I tried a couple of time to restart, but no success. The > only think that work was to physically remove the partitions for this > broker. I was able to restart the broker. The partitions were recreated > and were in-sync within about 30 minutes. My questions for the group are: > > Has anyone seen this error before? If so, what was your resolution? > Is there a more elegant way to "re-sync" a broker node? > Is there a way to identify which partition could be causing the issue and > possibly just remove that one? > > We have seen this a couple of other time in our environment, twice from > HW failures and once from someone killing the broker with a KILL -9. > > Is this a known issue? > > If anyone has any insight as to why this would occur I would greatly > appreciate it. > > Gene Robichaux > Manager, Database Operations > Match.com > 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 > > <> > > 02/18/2015 09:43:11 PM | FATAL | KafkaServerStartable | Fatal error > during KafkaServerStable startup. Prepare to shutdown > java.lang.NumberFormatException: For input string: " > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^
Re: KafkaProducer from kafka.clients hangs when some partitions are not available
Jun,
I am trying to test how KafkaProducer behaves with topic replication factor
= 1
1. One broker is offline BEFORE KafkaProducer starts sending messages.
Because of the bug I mentioned, KafkaProducer sends to the offline
partition and hangs forever.
2. One broker goes offline WHILE KafkaProducer is sending messages.
KafkaProducer seems to be hanging forever in this case. I am still looking.
Do you mind take a look?
Thanks
On Mon, Feb 23, 2015 at 7:01 PM, Jun Rao wrote:
> The logic in that code is to cycle through all partitions and return as
> soon as we see a partition with the leader. I do see an issue that if there
> are multiple threads sending messages to the same producer concurrently, we
> may not cycle through all partitions and therefore we could return an
> unavailable partition even when available partitions are present.
>
> Do you see this issue with just a single thread producing messages? The
> current logic seems to work correctly in that case.
>
> Thanks,
>
> Jun
>
> On Fri, Feb 20, 2015 at 12:45 PM, Xiaoyu Wang
> wrote:
>
> > Found the problem - it is a bug with Partitions of kafka client. Can you
> > guys confirm and patch in kafka clients?
> >
> > for (int i = 0; i < numPartitions; i++) {
> > int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
> > if (partitions.get(partition).leader() != null) {
> > return partitions.get(partition).partition();
> > }
> > }
> >
> >
> >
> > On Fri, Feb 20, 2015 at 2:35 PM, Xiaoyu Wang
> wrote:
> >
> > > Update:
> > >
> > > I am using kafka.clients 0.8.2-beta. Below are the test steps
> > >
> > >1. setup local kafka clusters with 2 brokers, 0 and 1
> > >2. create topic X with replication fact 1 and 4 partitions
> > >3. verify that each broker has two partitions
> > >4. shutdown broker 1
> > >5. start a producer sending data to topic X using KafkaProducer with
> > >required ack = 1
> > >6. producer hangs and does not exit.
> > >
> > > Offline partitions were take care of when the partitions is null (code
> > > attached below). However, the timeout setting does not seem to work.
> Not
> > > sure what caused KafkaProducer to hang.
> > >
> > > // choose the next available node in a round-robin fashion
> > > for (int i = 0; i < numPartitions; i++) {
> > > int partition = Utils.abs(counter.getAndIncrement()) %
> numPartitions;
> > > if (partitions.get(partition).leader() != null)
> > > return partition;
> > > }
> > > // no partitions are available, give a non-available partition
> > > return Utils.abs(counter.getAndIncrement()) % numPartitions;
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Feb 20, 2015 at 1:48 PM, Xiaoyu Wang
> > wrote:
> > >
> > >> Hello,
> > >>
> > >> I am experimenting sending data to kafka using KafkaProducer and found
> > >> that when a partition is completely offline, e.g. a topic with
> > replication
> > >> factor = 1 and some broker is down, KafkaProducer seems to be hanging
> > >> forever. Not even exit with the timeout setting. Can you take a look?
> > >>
> > >> I checked code and found that the partitioner create partition based
> on
> > >> the total partition number - including those offline partitions. Is it
> > >> possible that we change ProducerClient to ignore offline partitions?
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> -Xiaoyu
> > >>
> > >>
> > >
> >
>
Re: KafkaProducer from kafka.clients hangs when some partitions are not available
Hi, Xiaoyu,
1. Could you explain a bit more what the bug is? The code does try to avoid
picking an unavailable partition. There does seem to be an issue when there
are more than one thread producing data to the same producer instance. This
is being tracked in KAFKA-1984. How many producing threads do you have in
your test?
Thanks,
Jun
On Tue, Feb 24, 2015 at 7:56 AM, Xiaoyu Wang wrote:
> Jun,
>
> I am trying to test how KafkaProducer behaves with topic replication factor
> = 1
>
>1. One broker is offline BEFORE KafkaProducer starts sending messages.
>Because of the bug I mentioned, KafkaProducer sends to the offline
>partition and hangs forever.
>2. One broker goes offline WHILE KafkaProducer is sending messages.
>KafkaProducer seems to be hanging forever in this case. I am still
> looking.
>Do you mind take a look?
>
> Thanks
>
>
>
>
> On Mon, Feb 23, 2015 at 7:01 PM, Jun Rao wrote:
>
> > The logic in that code is to cycle through all partitions and return as
> > soon as we see a partition with the leader. I do see an issue that if
> there
> > are multiple threads sending messages to the same producer concurrently,
> we
> > may not cycle through all partitions and therefore we could return an
> > unavailable partition even when available partitions are present.
> >
> > Do you see this issue with just a single thread producing messages? The
> > current logic seems to work correctly in that case.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Feb 20, 2015 at 12:45 PM, Xiaoyu Wang
> > wrote:
> >
> > > Found the problem - it is a bug with Partitions of kafka client. Can
> you
> > > guys confirm and patch in kafka clients?
> > >
> > > for (int i = 0; i < numPartitions; i++) {
> > > int partition = Utils.abs(counter.getAndIncrement()) %
> numPartitions;
> > > if (partitions.get(partition).leader() != null) {
> > > return partitions.get(partition).partition();
> > > }
> > > }
> > >
> > >
> > >
> > > On Fri, Feb 20, 2015 at 2:35 PM, Xiaoyu Wang
> > wrote:
> > >
> > > > Update:
> > > >
> > > > I am using kafka.clients 0.8.2-beta. Below are the test steps
> > > >
> > > >1. setup local kafka clusters with 2 brokers, 0 and 1
> > > >2. create topic X with replication fact 1 and 4 partitions
> > > >3. verify that each broker has two partitions
> > > >4. shutdown broker 1
> > > >5. start a producer sending data to topic X using KafkaProducer
> with
> > > >required ack = 1
> > > >6. producer hangs and does not exit.
> > > >
> > > > Offline partitions were take care of when the partitions is null
> (code
> > > > attached below). However, the timeout setting does not seem to work.
> > Not
> > > > sure what caused KafkaProducer to hang.
> > > >
> > > > // choose the next available node in a round-robin fashion
> > > > for (int i = 0; i < numPartitions; i++) {
> > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > numPartitions;
> > > > if (partitions.get(partition).leader() != null)
> > > > return partition;
> > > > }
> > > > // no partitions are available, give a non-available partition
> > > > return Utils.abs(counter.getAndIncrement()) % numPartitions;
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Feb 20, 2015 at 1:48 PM, Xiaoyu Wang
> > > wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> I am experimenting sending data to kafka using KafkaProducer and
> found
> > > >> that when a partition is completely offline, e.g. a topic with
> > > replication
> > > >> factor = 1 and some broker is down, KafkaProducer seems to be
> hanging
> > > >> forever. Not even exit with the timeout setting. Can you take a
> look?
> > > >>
> > > >> I checked code and found that the partitioner create partition based
> > on
> > > >> the total partition number - including those offline partitions. Is
> it
> > > >> possible that we change ProducerClient to ignore offline partitions?
> > > >>
> > > >>
> > > >> Thanks,
> > > >>
> > > >> -Xiaoyu
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: KafkaProducer from kafka.clients hangs when some partitions are not available
Hi Jun,
If I understand it correctly. the highlighted line is for avoiding
offline partitions, is it?
for (int i = 0; i < numPartitions; i++) {
int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
if (partitions.get(partition).leader() != null) {
return partition; --> should be changed to return the actual
partition number?
}
}
On Tue, Feb 24, 2015 at 11:30 AM, Jun Rao wrote:
> Hi, Xiaoyu,
>
> 1. Could you explain a bit more what the bug is? The code does try to avoid
> picking an unavailable partition. There does seem to be an issue when there
> are more than one thread producing data to the same producer instance. This
> is being tracked in KAFKA-1984. How many producing threads do you have in
> your test?
>
> Thanks,
>
> Jun
>
> On Tue, Feb 24, 2015 at 7:56 AM, Xiaoyu Wang wrote:
>
> > Jun,
> >
> > I am trying to test how KafkaProducer behaves with topic replication
> factor
> > = 1
> >
> >1. One broker is offline BEFORE KafkaProducer starts sending messages.
> >Because of the bug I mentioned, KafkaProducer sends to the offline
> >partition and hangs forever.
> >2. One broker goes offline WHILE KafkaProducer is sending messages.
> >KafkaProducer seems to be hanging forever in this case. I am still
> > looking.
> >Do you mind take a look?
> >
> > Thanks
> >
> >
> >
> >
> > On Mon, Feb 23, 2015 at 7:01 PM, Jun Rao wrote:
> >
> > > The logic in that code is to cycle through all partitions and return as
> > > soon as we see a partition with the leader. I do see an issue that if
> > there
> > > are multiple threads sending messages to the same producer
> concurrently,
> > we
> > > may not cycle through all partitions and therefore we could return an
> > > unavailable partition even when available partitions are present.
> > >
> > > Do you see this issue with just a single thread producing messages? The
> > > current logic seems to work correctly in that case.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Feb 20, 2015 at 12:45 PM, Xiaoyu Wang
> > > wrote:
> > >
> > > > Found the problem - it is a bug with Partitions of kafka client. Can
> > you
> > > > guys confirm and patch in kafka clients?
> > > >
> > > > for (int i = 0; i < numPartitions; i++) {
> > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > numPartitions;
> > > > if (partitions.get(partition).leader() != null) {
> > > > return partitions.get(partition).partition();
> > > > }
> > > > }
> > > >
> > > >
> > > >
> > > > On Fri, Feb 20, 2015 at 2:35 PM, Xiaoyu Wang
> > > wrote:
> > > >
> > > > > Update:
> > > > >
> > > > > I am using kafka.clients 0.8.2-beta. Below are the test steps
> > > > >
> > > > >1. setup local kafka clusters with 2 brokers, 0 and 1
> > > > >2. create topic X with replication fact 1 and 4 partitions
> > > > >3. verify that each broker has two partitions
> > > > >4. shutdown broker 1
> > > > >5. start a producer sending data to topic X using KafkaProducer
> > with
> > > > >required ack = 1
> > > > >6. producer hangs and does not exit.
> > > > >
> > > > > Offline partitions were take care of when the partitions is null
> > (code
> > > > > attached below). However, the timeout setting does not seem to
> work.
> > > Not
> > > > > sure what caused KafkaProducer to hang.
> > > > >
> > > > > // choose the next available node in a round-robin fashion
> > > > > for (int i = 0; i < numPartitions; i++) {
> > > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > > numPartitions;
> > > > > if (partitions.get(partition).leader() != null)
> > > > > return partition;
> > > > > }
> > > > > // no partitions are available, give a non-available partition
> > > > > return Utils.abs(counter.getAndIncrement()) % numPartitions;
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 20, 2015 at 1:48 PM, Xiaoyu Wang >
> > > > wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >> I am experimenting sending data to kafka using KafkaProducer and
> > found
> > > > >> that when a partition is completely offline, e.g. a topic with
> > > > replication
> > > > >> factor = 1 and some broker is down, KafkaProducer seems to be
> > hanging
> > > > >> forever. Not even exit with the timeout setting. Can you take a
> > look?
> > > > >>
> > > > >> I checked code and found that the partitioner create partition
> based
> > > on
> > > > >> the total partition number - including those offline partitions.
> Is
> > it
> > > > >> possible that we change ProducerClient to ignore offline
> partitions?
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> -Xiaoyu
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: KafkaProducer from kafka.clients hangs when some partitions are not available
Ah, yes. You are right. That's a more obvious bug. Will fix that in
KAFKA-1984.
Thanks,
Jun
On Tue, Feb 24, 2015 at 8:37 AM, Xiaoyu Wang wrote:
> Hi Jun,
>
> If I understand it correctly. the highlighted line is for avoiding
> offline partitions, is it?
>
> for (int i = 0; i < numPartitions; i++) {
> int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
> if (partitions.get(partition).leader() != null) {
> return partition; --> should be changed to return the actual
> partition number?
> }
> }
>
>
> On Tue, Feb 24, 2015 at 11:30 AM, Jun Rao wrote:
>
> > Hi, Xiaoyu,
> >
> > 1. Could you explain a bit more what the bug is? The code does try to
> avoid
> > picking an unavailable partition. There does seem to be an issue when
> there
> > are more than one thread producing data to the same producer instance.
> This
> > is being tracked in KAFKA-1984. How many producing threads do you have in
> > your test?
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Feb 24, 2015 at 7:56 AM, Xiaoyu Wang
> wrote:
> >
> > > Jun,
> > >
> > > I am trying to test how KafkaProducer behaves with topic replication
> > factor
> > > = 1
> > >
> > >1. One broker is offline BEFORE KafkaProducer starts sending
> messages.
> > >Because of the bug I mentioned, KafkaProducer sends to the offline
> > >partition and hangs forever.
> > >2. One broker goes offline WHILE KafkaProducer is sending messages.
> > >KafkaProducer seems to be hanging forever in this case. I am still
> > > looking.
> > >Do you mind take a look?
> > >
> > > Thanks
> > >
> > >
> > >
> > >
> > > On Mon, Feb 23, 2015 at 7:01 PM, Jun Rao wrote:
> > >
> > > > The logic in that code is to cycle through all partitions and return
> as
> > > > soon as we see a partition with the leader. I do see an issue that if
> > > there
> > > > are multiple threads sending messages to the same producer
> > concurrently,
> > > we
> > > > may not cycle through all partitions and therefore we could return an
> > > > unavailable partition even when available partitions are present.
> > > >
> > > > Do you see this issue with just a single thread producing messages?
> The
> > > > current logic seems to work correctly in that case.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Fri, Feb 20, 2015 at 12:45 PM, Xiaoyu Wang
> > > > wrote:
> > > >
> > > > > Found the problem - it is a bug with Partitions of kafka client.
> Can
> > > you
> > > > > guys confirm and patch in kafka clients?
> > > > >
> > > > > for (int i = 0; i < numPartitions; i++) {
> > > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > > numPartitions;
> > > > > if (partitions.get(partition).leader() != null) {
> > > > > return partitions.get(partition).partition();
> > > > > }
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 20, 2015 at 2:35 PM, Xiaoyu Wang >
> > > > wrote:
> > > > >
> > > > > > Update:
> > > > > >
> > > > > > I am using kafka.clients 0.8.2-beta. Below are the test steps
> > > > > >
> > > > > >1. setup local kafka clusters with 2 brokers, 0 and 1
> > > > > >2. create topic X with replication fact 1 and 4 partitions
> > > > > >3. verify that each broker has two partitions
> > > > > >4. shutdown broker 1
> > > > > >5. start a producer sending data to topic X using
> KafkaProducer
> > > with
> > > > > >required ack = 1
> > > > > >6. producer hangs and does not exit.
> > > > > >
> > > > > > Offline partitions were take care of when the partitions is null
> > > (code
> > > > > > attached below). However, the timeout setting does not seem to
> > work.
> > > > Not
> > > > > > sure what caused KafkaProducer to hang.
> > > > > >
> > > > > > // choose the next available node in a round-robin fashion
> > > > > > for (int i = 0; i < numPartitions; i++) {
> > > > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > > > numPartitions;
> > > > > > if (partitions.get(partition).leader() != null)
> > > > > > return partition;
> > > > > > }
> > > > > > // no partitions are available, give a non-available partition
> > > > > > return Utils.abs(counter.getAndIncrement()) % numPartitions;
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 20, 2015 at 1:48 PM, Xiaoyu Wang <
> xw...@rocketfuel.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Hello,
> > > > > >>
> > > > > >> I am experimenting sending data to kafka using KafkaProducer and
> > > found
> > > > > >> that when a partition is completely offline, e.g. a topic with
> > > > > replication
> > > > > >> factor = 1 and some broker is down, KafkaProducer seems to be
> > > hanging
> > > > > >> forever. Not even exit with the timeout setting. Can you take a
> > > look?
> > > > > >>
> > > > > >> I checked code and found that the partitioner create partition
> > based
> > > > on
> > > > > >> the total partition number - including those offline partitions.
> > Is
> > > it
> >
Re: Can Mirroring Preserve Every Topic's Partition?
Guozhang, Thank you for the reply, but could you be a little bit more detailed? When will this new MM with exact mirroring be rolled out? I went to the following URL to read up on Kafka-1650 https://issues.apache.org/jira/browse/KAFKA-1650 but that issue doesn't appear to be about mirroring, but rather about clean shutdowns. What exactly would I need to alter in the MM code? I've cloned the 8.2.0 Kafka source and read through the kafka.tools.MirrorMaker.scala class, and cannot find a single mention of partition within its source. Looking at the ProducerThread and ConsumerThread classes, I would expect them to be passed some parameter that corresponded to the partition number of the message they're producing/consuming. However the closest thing I see to anything partition related is the "data.key" object. Some more guidance on this would be extremely helpful as this exact mirroring process seems like a common use case, and it's not clear how one should go about doing this. Thanks Guozhang, Alex Melville On Sun, Feb 22, 2015 at 3:53 PM, Guozhang Wang wrote: > Hi Alex, > > What I originally meant is that you probably need to manually modify the MM > code in order to achieve your needs. However, MM has been improved a lot > since last time we synced up, in the next major release the MM will support > exact mirroring (details in KAFKA-1650) with some functional extensions. > Basically you just need to ensure the source and destination cluster have > the some partitions for each topic, which can be done via consumer > rebalance callbacks. > > Guozhang > > On Fri, Feb 20, 2015 at 12:34 AM, Alex Melville > wrote: > > > Neha and Guozhang, > > > > > > This thread is several months old now but I'd like to follow up on it as > I > > have a couple more questions related to it. > > > > > > 1. Guozhang, you suggested 3 steps I take to ensure each piece of data > > remains on the same partition from source to target cluster. In > particular > > you suggest > > > > > > *2. When the consumer of the MM gets a message, put the message > to > > the* > > *producer's queue based on its partition id; i.e. if the partition id is > > n,* > > > > > > *put to n's producer queue.* > > *3. When producer sends the data, specify the partition id; so each > > producer**will only send to a single partition.* > > > > Since the MM is just using the default producer/consumer with no custom > > logic written by me, how am I supposed to find a particular message's > > partition id, and once I do that how am I supposed to specify which > > producer queue to put that message in? > > > > > > > > 2. What does specifying the number of producer's in the MM do? Does each > > producer only push to one partition within a cluster? > > > > > > > > > > 3. I wrote a SimpleProducer that publishes a record with a String key and > > byte[] message to a kafka cluster. Originally the producer config file I > > was passing to Mirrormaker had the following important parameters > > > > > > > > *producer.type=sync* > > > > *compression.codec=none* > > > > *serializer.class=kafka.serializer.DefaultEncoder* > > *key.serializer.class=kafka.serializer.StringEncoder* > > *partitioner.class=org.rubicon.hmc.SimplePartitioner* > > > > > > > > where SimplePartitioner just took the String key, hashed it, and then > took > > the resulting number mod the number of partitions and used that number as > > the partition to publish to. However I kept getting an error when the > > producer tried to publish to the target cluster which said something > along > > the lines of "[B cannot be cast to java.lang.String". I at last figured > > this was because I had specified > > key.serializer.class=kafka.serializer.StringEncoder in the producer, but > > what it was receiving from the consumer thread was just a byte[] msg and > > when it tried to use the StringEncoder an error was returned. Now I'm > using > > the default partitionar and default encoder, I'm not getting that > > exception, and data is being copied from source to target cluster. > However > > I've lost the ability to partition data on a String key, because the MM > > producer is just given a raw byte array. My question is, am I correct in > > this reasoning behind the "[B cannot be cast to java.lang.String" error? > > > > > > > > > > > > > > Thank you in advance Neha, Guozhang, and the rest of the Kafka community! > > > > > > Alex Melville > > > > On Fri, Dec 5, 2014 at 5:30 PM, Neha Narkhede wrote: > > > > > Going back to your previous requirement of ensuring that the data in > the > > > target cluster is in the same order as the source cluster, all you need > > is > > > to specify a key with every record in your data. The mirror maker and > its > > > producer takes care of placing all the data for a particular key in the > > > same partition on the target cluster. Effectively, all your data will > be > > in > > > the same order (though there may be a few duplicates as I mentioned > > > before). >
Re: KafkaProducer from kafka.clients hangs when some partitions are not available
Jun,
Can you also take a look at the second problem I am having?
> > > I am trying to test how KafkaProducer behaves with topic replication
> > factor
> > > = 1
> > >
> > >1. One broker is offline BEFORE KafkaProducer starts sending
> messages.
> > >Because of the bug I mentioned, KafkaProducer sends to the offline
> > >partition and hangs forever.
*> > >2. One broker goes offline WHILE KafkaProducer is sending
messages. > > >KafkaProducer seems to be hanging forever in this case.
I am still > > > looking.*
On Tue, Feb 24, 2015 at 12:03 PM, Jun Rao wrote:
> Ah, yes. You are right. That's a more obvious bug. Will fix that in
> KAFKA-1984.
>
> Thanks,
>
> Jun
>
> On Tue, Feb 24, 2015 at 8:37 AM, Xiaoyu Wang wrote:
>
> > Hi Jun,
> >
> > If I understand it correctly. the highlighted line is for avoiding
> > offline partitions, is it?
> >
> > for (int i = 0; i < numPartitions; i++) {
> > int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
> > if (partitions.get(partition).leader() != null) {
> > return partition; --> should be changed to return the actual
> > partition number?
> > }
> > }
> >
> >
> > On Tue, Feb 24, 2015 at 11:30 AM, Jun Rao wrote:
> >
> > > Hi, Xiaoyu,
> > >
> > > 1. Could you explain a bit more what the bug is? The code does try to
> > avoid
> > > picking an unavailable partition. There does seem to be an issue when
> > there
> > > are more than one thread producing data to the same producer instance.
> > This
> > > is being tracked in KAFKA-1984. How many producing threads do you have
> in
> > > your test?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Feb 24, 2015 at 7:56 AM, Xiaoyu Wang
> > wrote:
> > >
> > > > Jun,
> > > >
> > > > I am trying to test how KafkaProducer behaves with topic replication
> > > factor
> > > > = 1
> > > >
> > > >1. One broker is offline BEFORE KafkaProducer starts sending
> > messages.
> > > >Because of the bug I mentioned, KafkaProducer sends to the
> offline
> > > >partition and hangs forever.
> > > >2. One broker goes offline WHILE KafkaProducer is sending
> messages.
> > > >KafkaProducer seems to be hanging forever in this case. I am still
> > > > looking.
> > > >Do you mind take a look?
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Feb 23, 2015 at 7:01 PM, Jun Rao wrote:
> > > >
> > > > > The logic in that code is to cycle through all partitions and
> return
> > as
> > > > > soon as we see a partition with the leader. I do see an issue that
> if
> > > > there
> > > > > are multiple threads sending messages to the same producer
> > > concurrently,
> > > > we
> > > > > may not cycle through all partitions and therefore we could return
> an
> > > > > unavailable partition even when available partitions are present.
> > > > >
> > > > > Do you see this issue with just a single thread producing messages?
> > The
> > > > > current logic seems to work correctly in that case.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Feb 20, 2015 at 12:45 PM, Xiaoyu Wang <
> xw...@rocketfuel.com>
> > > > > wrote:
> > > > >
> > > > > > Found the problem - it is a bug with Partitions of kafka client.
> > Can
> > > > you
> > > > > > guys confirm and patch in kafka clients?
> > > > > >
> > > > > > for (int i = 0; i < numPartitions; i++) {
> > > > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > > > numPartitions;
> > > > > > if (partitions.get(partition).leader() != null) {
> > > > > > return partitions.get(partition).partition();
> > > > > > }
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 20, 2015 at 2:35 PM, Xiaoyu Wang <
> xw...@rocketfuel.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Update:
> > > > > > >
> > > > > > > I am using kafka.clients 0.8.2-beta. Below are the test steps
> > > > > > >
> > > > > > >1. setup local kafka clusters with 2 brokers, 0 and 1
> > > > > > >2. create topic X with replication fact 1 and 4 partitions
> > > > > > >3. verify that each broker has two partitions
> > > > > > >4. shutdown broker 1
> > > > > > >5. start a producer sending data to topic X using
> > KafkaProducer
> > > > with
> > > > > > >required ack = 1
> > > > > > >6. producer hangs and does not exit.
> > > > > > >
> > > > > > > Offline partitions were take care of when the partitions is
> null
> > > > (code
> > > > > > > attached below). However, the timeout setting does not seem to
> > > work.
> > > > > Not
> > > > > > > sure what caused KafkaProducer to hang.
> > > > > > >
> > > > > > > // choose the next available node in a round-robin fashion
> > > > > > > for (int i = 0; i < numPartitions; i++) {
> > > > > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > > > > numPartitions;
> > > > > > > if (partitions.get(partition).leader() != null)
> > > > > > > return partition;
> > > > > > > }
> > > > > > > /
SimpleConsumer Example seems has some problems
Hi, I try to use the example https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example(and BTW where is 0.8.2 example?). It does not work, so I make a simplest possible snippet (still not working) out of coce for findLeader(). Here it is below, and it produces the error: Error communicating with Broker [127.0.0.1] to find Leader for [test, 0] Reason: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.null Which this igf I use console consumer (kafka-console-consumer.sh) with the same topic, localhost, port - it works, so broker is alive. Please help to find what's wrong. And the whole example: does somebody know how to fix it?
Stream naming conventions?
Hello all, Just wondering if those with a good amount of experience using Kafka in production with many streams have converged on any sort of naming convention. If so would you be willing to share? Thanks in advance, Taylor Gautier
Re: Anyone using log4j Appender for Kafka?
Here is sample log4j.properties https://github.com/stealthly/scala-kafka/blob/master/src/test/resources/log4j.properties#L54-L67 I _almost_ have always pulled the class https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala internal to private repo and changed it as things came up... e.g. setSource(), setTags() blah blah... Paul Otto has an open source version https://github.com/potto007/kafka-appender-layout that you could try out too that he built to tackle some of the layout things. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville wrote: > ^^ I would really appreciate this as well. It's unclear how to get log4j > working with Kafka when you have no prior experience with log4j. > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman > wrote: > > > Thanks. But we're using log4j. I tried setting the type to async but it > > generated a warning of no such field. Is there any real documentation on > > the log4j appender? > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > sschlans...@opentable.com> wrote: > > > > > We just configure our logback.xml to have two Appenders, an > AsyncAppender > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as appropriate). > > > > > > AsyncAppender removes more failure cases too, e.g. a health check > hanging > > > rather than returning rapidly could block you application. > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > anthony.musy...@gmail.com > > > > > > wrote: > > > > > > > Theres also another one here. > > > > > > > > https://github.com/danielwegener/logback-kafka-appender. > > > > > > > > It has a fallback appender which might address the issue of Kafka > being > > > > un-available. > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > sschlans...@opentable.com> wrote: > > > > > > > >> Here’s my attempt at a Logback version, should be fairly easily > > ported: > > > >> > > > >> https://github.com/opentable/otj-logging/blob/master/kafka/ > > > src/main/java/com/opentable/logging/KafkaAppender.java > > > >> > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman > > > wrote: > > > >> > > > >>> I am just starting to use it and could use a little guidance. I was > > > able > > > >> to > > > >>> get it working with 0.8.2 but am not clear on best practices for > > using > > > >> it. > > > >>> > > > >>> Anyway willing to help me out a bit? Got a few questions, like how > to > > > >>> protect applications from when kafka is down or unreachable. > > > >>> > > > >>> It seems like a great idea for being able to get logs from existing > > > >>> applications to be collected by kafka. > > > >>> > > > >>> Thanks in advance! > > > >> > > > >> > > > > > > > > >
Re: Anyone using log4j Appender for Kafka?
Yea, when I try to set type to async (exactly like the example) I get: log4j:WARN No such property [producerType] in kafka.producer.KafkaLog4jAppender. On Tue Feb 24 2015 at 1:35:54 PM Joe Stein wrote: > Here is sample log4j.properties > https://github.com/stealthly/scala-kafka/blob/master/src/tes > t/resources/log4j.properties#L54-L67 > > I _almost_ have always pulled the class > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ > scala/kafka/producer/KafkaLog4jAppender.scala > internal > to private repo and changed it as things came up... e.g. setSource(), > setTags() blah blah... > > Paul Otto has an open source version > https://github.com/potto007/kafka-appender-layout that you could try out > too that he built to tackle some of the layout things. > > ~ Joe Stein > - - - - - - - - - - - - - - - - - > > http://www.stealth.ly > - - - - - - - - - - - - - - - - - > > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville > wrote: > > > ^^ I would really appreciate this as well. It's unclear how to get log4j > > working with Kafka when you have no prior experience with log4j. > > > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman > > wrote: > > > > > Thanks. But we're using log4j. I tried setting the type to async but it > > > generated a warning of no such field. Is there any real documentation > on > > > the log4j appender? > > > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > > sschlans...@opentable.com> wrote: > > > > > > > We just configure our logback.xml to have two Appenders, an > > AsyncAppender > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as > appropriate). > > > > > > > > AsyncAppender removes more failure cases too, e.g. a health check > > hanging > > > > rather than returning rapidly could block you application. > > > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > > anthony.musy...@gmail.com > > > > > > > > wrote: > > > > > > > > > Theres also another one here. > > > > > > > > > > https://github.com/danielwegener/logback-kafka-appender. > > > > > > > > > > It has a fallback appender which might address the issue of Kafka > > being > > > > > un-available. > > > > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > >> Here’s my attempt at a Logback version, should be fairly easily > > > ported: > > > > >> > > > > >> https://github.com/opentable/otj-logging/blob/master/kafka/ > > > > src/main/java/com/opentable/logging/KafkaAppender.java > > > > >> > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman > > > > wrote: > > > > >> > > > > >>> I am just starting to use it and could use a little guidance. I > was > > > > able > > > > >> to > > > > >>> get it working with 0.8.2 but am not clear on best practices for > > > using > > > > >> it. > > > > >>> > > > > >>> Anyway willing to help me out a bit? Got a few questions, like > how > > to > > > > >>> protect applications from when kafka is down or unreachable. > > > > >>> > > > > >>> It seems like a great idea for being able to get logs from > existing > > > > >>> applications to be collected by kafka. > > > > >>> > > > > >>> Thanks in advance! > > > > >> > > > > >> > > > > > > > > > > > > > >
Re: How to find the number of message in the topic
Does that count get frozen on a fixed number or any random number?
-Zakee
On Mon, Feb 23, 2015 at 9:48 AM, Stuart Reynolds
wrote:
> See SimpleConsumer. getOffsetsBefore
> and the getLastOffset example here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> whichTime=-1 or -2 (depending whether you want the latest or earliest
> offset.
>
> However, I've been banging my head against the wall with this API --
> it seems that, even with a single broker and a single host, sometimes
> the latest offset gets frozen as new data comes in. :-/
>
>
>
> public static long getLastOffset(SimpleConsumer consumer, String
> topic, int partition,
> long whichTime, String clientName) {
> TopicAndPartition topicAndPartition = new
> TopicAndPartition(topic, partition);
> Map requestInfo
> = new HashMap();
> requestInfo.put(topicAndPartition, new
> PartitionOffsetRequestInfo(whichTime, 1));
> kafka.javaapi.OffsetRequest request = new
> kafka.javaapi.OffsetRequest(requestInfo,
> kafka.api.OffsetRequest.CurrentVersion(),clientName);
> OffsetResponse response = consumer.getOffsetsBefore(request);
>
> if (response.hasError()) {
> System.out.println("Error fetching data Offset Data the
> Broker. Reason: " + response.errorCode(topic, partition) );
> return 0;
> }
> long[] offsets = response.offsets(topic, partition);
> return offsets[0];
> }
>
> On Sun, Feb 22, 2015 at 9:05 PM, Bhuvana Baskar
> wrote:
> > Hi,
> >
> > Please let me know how to find the total number of messages in a
> particular
> > topic.
> >
> > Regards,
> > Bhuvana
>
> Heavy rains mean flooding
> Anywhere it rains it can flood. Learn your risk. Get flood insurance.
> http://thirdpartyoffers.netzero.net/TGL3255/54eb6db072a8f6db07624mp13duc
>
>
Re: Anyone using log4j Appender for Kafka?
and kafka too :) On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein wrote: > are you including > https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in > your project? > > ~ Joe Stein > - - - - - - - - - - - - - - - - - > > http://www.stealth.ly > - - - - - - - - - - - - - - - - - > > On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman > wrote: > >> Yea, when I try to set type to async (exactly like the example) I get: >> log4j:WARN No such property [producerType] in >> kafka.producer.KafkaLog4jAppender. >> >> On Tue Feb 24 2015 at 1:35:54 PM Joe Stein wrote: >> >> > Here is sample log4j.properties >> > https://github.com/stealthly/scala-kafka/blob/master/src/tes >> > t/resources/log4j.properties#L54-L67 >> > >> > I _almost_ have always pulled the class >> > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ >> > scala/kafka/producer/KafkaLog4jAppender.scala >> > internal >> > to private repo and changed it as things came up... e.g. setSource(), >> > setTags() blah blah... >> > >> > Paul Otto has an open source version >> > https://github.com/potto007/kafka-appender-layout that you could try >> out >> > too that he built to tackle some of the layout things. >> > >> > ~ Joe Stein >> > - - - - - - - - - - - - - - - - - >> > >> > http://www.stealth.ly >> > - - - - - - - - - - - - - - - - - >> > >> > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville >> > wrote: >> > >> > > ^^ I would really appreciate this as well. It's unclear how to get >> log4j >> > > working with Kafka when you have no prior experience with log4j. >> > > >> > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman >> > > wrote: >> > > >> > > > Thanks. But we're using log4j. I tried setting the type to async >> but it >> > > > generated a warning of no such field. Is there any real >> documentation >> > on >> > > > the log4j appender? >> > > > >> > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < >> > > > sschlans...@opentable.com> wrote: >> > > > >> > > > > We just configure our logback.xml to have two Appenders, an >> > > AsyncAppender >> > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as >> > appropriate). >> > > > > >> > > > > AsyncAppender removes more failure cases too, e.g. a health check >> > > hanging >> > > > > rather than returning rapidly could block you application. >> > > > > >> > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < >> > > anthony.musy...@gmail.com >> > > > > >> > > > > wrote: >> > > > > >> > > > > > Theres also another one here. >> > > > > > >> > > > > > https://github.com/danielwegener/logback-kafka-appender. >> > > > > > >> > > > > > It has a fallback appender which might address the issue of >> Kafka >> > > being >> > > > > > un-available. >> > > > > > >> > > > > > >> > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < >> > > > > > sschlans...@opentable.com> wrote: >> > > > > > >> > > > > >> Here’s my attempt at a Logback version, should be fairly easily >> > > > ported: >> > > > > >> >> > > > > >> https://github.com/opentable/otj-logging/blob/master/kafka/ >> > > > > src/main/java/com/opentable/logging/KafkaAppender.java >> > > > > >> >> > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman < >> sc...@woofplanet.com> >> > > > > wrote: >> > > > > >> >> > > > > >>> I am just starting to use it and could use a little guidance. >> I >> > was >> > > > > able >> > > > > >> to >> > > > > >>> get it working with 0.8.2 but am not clear on best practices >> for >> > > > using >> > > > > >> it. >> > > > > >>> >> > > > > >>> Anyway willing to help me out a bit? Got a few questions, like >> > how >> > > to >> > > > > >>> protect applications from when kafka is down or unreachable. >> > > > > >>> >> > > > > >>> It seems like a great idea for being able to get logs from >> > existing >> > > > > >>> applications to be collected by kafka. >> > > > > >>> >> > > > > >>> Thanks in advance! >> > > > > >> >> > > > > >> >> > > > > >> > > > > >> > > > >> > > >> > >> > >
Re: Anyone using log4j Appender for Kafka?
are you including https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in your project? ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman wrote: > Yea, when I try to set type to async (exactly like the example) I get: > log4j:WARN No such property [producerType] in > kafka.producer.KafkaLog4jAppender. > > On Tue Feb 24 2015 at 1:35:54 PM Joe Stein wrote: > > > Here is sample log4j.properties > > https://github.com/stealthly/scala-kafka/blob/master/src/tes > > t/resources/log4j.properties#L54-L67 > > > > I _almost_ have always pulled the class > > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ > > scala/kafka/producer/KafkaLog4jAppender.scala > > internal > > to private repo and changed it as things came up... e.g. setSource(), > > setTags() blah blah... > > > > Paul Otto has an open source version > > https://github.com/potto007/kafka-appender-layout that you could try out > > too that he built to tackle some of the layout things. > > > > ~ Joe Stein > > - - - - - - - - - - - - - - - - - > > > > http://www.stealth.ly > > - - - - - - - - - - - - - - - - - > > > > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville > > wrote: > > > > > ^^ I would really appreciate this as well. It's unclear how to get > log4j > > > working with Kafka when you have no prior experience with log4j. > > > > > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman > > > wrote: > > > > > > > Thanks. But we're using log4j. I tried setting the type to async but > it > > > > generated a warning of no such field. Is there any real documentation > > on > > > > the log4j appender? > > > > > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > > > sschlans...@opentable.com> wrote: > > > > > > > > > We just configure our logback.xml to have two Appenders, an > > > AsyncAppender > > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as > > appropriate). > > > > > > > > > > AsyncAppender removes more failure cases too, e.g. a health check > > > hanging > > > > > rather than returning rapidly could block you application. > > > > > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > > > anthony.musy...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Theres also another one here. > > > > > > > > > > > > https://github.com/danielwegener/logback-kafka-appender. > > > > > > > > > > > > It has a fallback appender which might address the issue of Kafka > > > being > > > > > > un-available. > > > > > > > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > >> Here’s my attempt at a Logback version, should be fairly easily > > > > ported: > > > > > >> > > > > > >> https://github.com/opentable/otj-logging/blob/master/kafka/ > > > > > src/main/java/com/opentable/logging/KafkaAppender.java > > > > > >> > > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman < > sc...@woofplanet.com> > > > > > wrote: > > > > > >> > > > > > >>> I am just starting to use it and could use a little guidance. I > > was > > > > > able > > > > > >> to > > > > > >>> get it working with 0.8.2 but am not clear on best practices > for > > > > using > > > > > >> it. > > > > > >>> > > > > > >>> Anyway willing to help me out a bit? Got a few questions, like > > how > > > to > > > > > >>> protect applications from when kafka is down or unreachable. > > > > > >>> > > > > > >>> It seems like a great idea for being able to get logs from > > existing > > > > > >>> applications to be collected by kafka. > > > > > >>> > > > > > >>> Thanks in advance! > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
Re: Anyone using log4j Appender for Kafka?
I'm including log4j-1.2.17, slf4j-api-1.7.6, slf4j-log4j12-1.6.1, kafka-clients-0.8.2.0, scala-library-2.11.5, and kafka_2.11-0.8.2.0 (java app) On Tue Feb 24 2015 at 2:23:40 PM Joe Stein wrote: > are you including > https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in > your project? > > ~ Joe Stein > - - - - - - - - - - - - - - - - - > > http://www.stealth.ly > - - - - - - - - - - - - - - - - - > > On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman > wrote: > > > Yea, when I try to set type to async (exactly like the example) I get: > > log4j:WARN No such property [producerType] in > > kafka.producer.KafkaLog4jAppender. > > > > On Tue Feb 24 2015 at 1:35:54 PM Joe Stein wrote: > > > > > Here is sample log4j.properties > > > https://github.com/stealthly/scala-kafka/blob/master/src/tes > > > t/resources/log4j.properties#L54-L67 > > > > > > I _almost_ have always pulled the class > > > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ > > > scala/kafka/producer/KafkaLog4jAppender.scala > > > internal > > > to private repo and changed it as things came up... e.g. setSource(), > > > setTags() blah blah... > > > > > > Paul Otto has an open source version > > > https://github.com/potto007/kafka-appender-layout that you could try > out > > > too that he built to tackle some of the layout things. > > > > > > ~ Joe Stein > > > - - - - - - - - - - - - - - - - - > > > > > > http://www.stealth.ly > > > - - - - - - - - - - - - - - - - - > > > > > > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville > > > wrote: > > > > > > > ^^ I would really appreciate this as well. It's unclear how to get > > log4j > > > > working with Kafka when you have no prior experience with log4j. > > > > > > > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman > > > > > wrote: > > > > > > > > > Thanks. But we're using log4j. I tried setting the type to async > but > > it > > > > > generated a warning of no such field. Is there any real > documentation > > > on > > > > > the log4j appender? > > > > > > > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > We just configure our logback.xml to have two Appenders, an > > > > AsyncAppender > > > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as > > > appropriate). > > > > > > > > > > > > AsyncAppender removes more failure cases too, e.g. a health check > > > > hanging > > > > > > rather than returning rapidly could block you application. > > > > > > > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > > > > anthony.musy...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Theres also another one here. > > > > > > > > > > > > > > https://github.com/danielwegener/logback-kafka-appender. > > > > > > > > > > > > > > It has a fallback appender which might address the issue of > Kafka > > > > being > > > > > > > un-available. > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > > > >> Here’s my attempt at a Logback version, should be fairly > easily > > > > > ported: > > > > > > >> > > > > > > >> https://github.com/opentable/otj-logging/blob/master/kafka/ > > > > > > src/main/java/com/opentable/logging/KafkaAppender.java > > > > > > >> > > > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman < > > sc...@woofplanet.com> > > > > > > wrote: > > > > > > >> > > > > > > >>> I am just starting to use it and could use a little > guidance. I > > > was > > > > > > able > > > > > > >> to > > > > > > >>> get it working with 0.8.2 but am not clear on best practices > > for > > > > > using > > > > > > >> it. > > > > > > >>> > > > > > > >>> Anyway willing to help me out a bit? Got a few questions, > like > > > how > > > > to > > > > > > >>> protect applications from when kafka is down or unreachable. > > > > > > >>> > > > > > > >>> It seems like a great idea for being able to get logs from > > > existing > > > > > > >>> applications to be collected by kafka. > > > > > > >>> > > > > > > >>> Thanks in advance! > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Anyone using log4j Appender for Kafka?
Producer type isn't needed anymore with the new producer so in the the logger properties just leave that out in 0.8.2 and it should work. On Tue, Feb 24, 2015 at 2:24 PM, Joe Stein wrote: > Interesting, looks like a breaking change from 0.8.1 > https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala > to 0.8.2 > https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala > > On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein wrote: > >> and kafka too :) >> >> On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein wrote: >> >>> are you including >>> https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 >>> in your project? >>> >>> ~ Joe Stein >>> - - - - - - - - - - - - - - - - - >>> >>> http://www.stealth.ly >>> - - - - - - - - - - - - - - - - - >>> >>> On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman >>> wrote: >>> Yea, when I try to set type to async (exactly like the example) I get: log4j:WARN No such property [producerType] in kafka.producer.KafkaLog4jAppender. On Tue Feb 24 2015 at 1:35:54 PM Joe Stein wrote: > Here is sample log4j.properties > https://github.com/stealthly/scala-kafka/blob/master/src/tes > t/resources/log4j.properties#L54-L67 > > I _almost_ have always pulled the class > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ > scala/kafka/producer/KafkaLog4jAppender.scala > internal > to private repo and changed it as things came up... e.g. setSource(), > setTags() blah blah... > > Paul Otto has an open source version > https://github.com/potto007/kafka-appender-layout that you could try out > too that he built to tackle some of the layout things. > > ~ Joe Stein > - - - - - - - - - - - - - - - - - > > http://www.stealth.ly > - - - - - - - - - - - - - - - - - > > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville > wrote: > > > ^^ I would really appreciate this as well. It's unclear how to get log4j > > working with Kafka when you have no prior experience with log4j. > > > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman < sc...@woofplanet.com> > > wrote: > > > > > Thanks. But we're using log4j. I tried setting the type to async but it > > > generated a warning of no such field. Is there any real documentation > on > > > the log4j appender? > > > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > > sschlans...@opentable.com> wrote: > > > > > > > We just configure our logback.xml to have two Appenders, an > > AsyncAppender > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as > appropriate). > > > > > > > > AsyncAppender removes more failure cases too, e.g. a health check > > hanging > > > > rather than returning rapidly could block you application. > > > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > > anthony.musy...@gmail.com > > > > > > > > wrote: > > > > > > > > > Theres also another one here. > > > > > > > > > > https://github.com/danielwegener/logback-kafka-appender. > > > > > > > > > > It has a fallback appender which might address the issue of Kafka > > being > > > > > un-available. > > > > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > >> Here’s my attempt at a Logback version, should be fairly easily > > > ported: > > > > >> > > > > >> https://github.com/opentable/otj-logging/blob/master/kafka/ > > > > src/main/java/com/opentable/logging/KafkaAppender.java > > > > >> > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman < sc...@woofplanet.com> > > > > wrote: > > > > >> > > > > >>> I am just starting to use it and could use a little guidance. I > was > > > > able > > > > >> to > > > > >>> get it working with 0.8.2 but am not clear on best practices for > > > using > > > > >> it. > > > > >>> > > > > >>> Anyway willing to help me out a bit? Got a few questions, like > how > > to > > > > >>> protect applications from when kafka is down or unreachable. > > > > >>> > > > > >>> It seems like a great idea for being able to get logs from > existing > > > > >>> applications to be collected by kafka. > > > > >>> > > > > >>> Thanks in advance! > > > > >> > > > > >> > > > > > > > > > > > > > > >>> >>> >> >
Re: Anyone using log4j Appender for Kafka?
Interesting, looks like a breaking change from 0.8.1 https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala to 0.8.2 https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein wrote: > and kafka too :) > > On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein wrote: > >> are you including >> https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 >> in your project? >> >> ~ Joe Stein >> - - - - - - - - - - - - - - - - - >> >> http://www.stealth.ly >> - - - - - - - - - - - - - - - - - >> >> On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman >> wrote: >> >>> Yea, when I try to set type to async (exactly like the example) I get: >>> log4j:WARN No such property [producerType] in >>> kafka.producer.KafkaLog4jAppender. >>> >>> On Tue Feb 24 2015 at 1:35:54 PM Joe Stein wrote: >>> >>> > Here is sample log4j.properties >>> > https://github.com/stealthly/scala-kafka/blob/master/src/tes >>> > t/resources/log4j.properties#L54-L67 >>> > >>> > I _almost_ have always pulled the class >>> > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ >>> > scala/kafka/producer/KafkaLog4jAppender.scala >>> > internal >>> > to private repo and changed it as things came up... e.g. setSource(), >>> > setTags() blah blah... >>> > >>> > Paul Otto has an open source version >>> > https://github.com/potto007/kafka-appender-layout that you could try >>> out >>> > too that he built to tackle some of the layout things. >>> > >>> > ~ Joe Stein >>> > - - - - - - - - - - - - - - - - - >>> > >>> > http://www.stealth.ly >>> > - - - - - - - - - - - - - - - - - >>> > >>> > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville >>> > wrote: >>> > >>> > > ^^ I would really appreciate this as well. It's unclear how to get >>> log4j >>> > > working with Kafka when you have no prior experience with log4j. >>> > > >>> > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman >> > >>> > > wrote: >>> > > >>> > > > Thanks. But we're using log4j. I tried setting the type to async >>> but it >>> > > > generated a warning of no such field. Is there any real >>> documentation >>> > on >>> > > > the log4j appender? >>> > > > >>> > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < >>> > > > sschlans...@opentable.com> wrote: >>> > > > >>> > > > > We just configure our logback.xml to have two Appenders, an >>> > > AsyncAppender >>> > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as >>> > appropriate). >>> > > > > >>> > > > > AsyncAppender removes more failure cases too, e.g. a health check >>> > > hanging >>> > > > > rather than returning rapidly could block you application. >>> > > > > >>> > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < >>> > > anthony.musy...@gmail.com >>> > > > > >>> > > > > wrote: >>> > > > > >>> > > > > > Theres also another one here. >>> > > > > > >>> > > > > > https://github.com/danielwegener/logback-kafka-appender. >>> > > > > > >>> > > > > > It has a fallback appender which might address the issue of >>> Kafka >>> > > being >>> > > > > > un-available. >>> > > > > > >>> > > > > > >>> > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < >>> > > > > > sschlans...@opentable.com> wrote: >>> > > > > > >>> > > > > >> Here’s my attempt at a Logback version, should be fairly >>> easily >>> > > > ported: >>> > > > > >> >>> > > > > >> https://github.com/opentable/otj-logging/blob/master/kafka/ >>> > > > > src/main/java/com/opentable/logging/KafkaAppender.java >>> > > > > >> >>> > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman < >>> sc...@woofplanet.com> >>> > > > > wrote: >>> > > > > >> >>> > > > > >>> I am just starting to use it and could use a little >>> guidance. I >>> > was >>> > > > > able >>> > > > > >> to >>> > > > > >>> get it working with 0.8.2 but am not clear on best practices >>> for >>> > > > using >>> > > > > >> it. >>> > > > > >>> >>> > > > > >>> Anyway willing to help me out a bit? Got a few questions, >>> like >>> > how >>> > > to >>> > > > > >>> protect applications from when kafka is down or unreachable. >>> > > > > >>> >>> > > > > >>> It seems like a great idea for being able to get logs from >>> > existing >>> > > > > >>> applications to be collected by kafka. >>> > > > > >>> >>> > > > > >>> Thanks in advance! >>> > > > > >> >>> > > > > >> >>> > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >> >> >
Re: Anyone using log4j Appender for Kafka?
Yea, however I don't get async behavior. When kafka is down the log blocks, which is kinda nasty to my app. On Tue Feb 24 2015 at 2:27:09 PM Joe Stein wrote: > Producer type isn't needed anymore with the new producer so in the the > logger properties just leave that out in 0.8.2 and it should work. > > On Tue, Feb 24, 2015 at 2:24 PM, Joe Stein wrote: > > > Interesting, looks like a breaking change from 0.8.1 > > https://github.com/apache/kafka/blob/0.8.1/core/src/ > main/scala/kafka/producer/KafkaLog4jAppender.scala > > to 0.8.2 > > https://github.com/apache/kafka/blob/0.8.2/core/src/ > main/scala/kafka/producer/KafkaLog4jAppender.scala > > > > On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein wrote: > > > >> and kafka too :) > >> > >> On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein > wrote: > >> > >>> are you including > >>> https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 > >>> in your project? > >>> > >>> ~ Joe Stein > >>> - - - - - - - - - - - - - - - - - > >>> > >>> http://www.stealth.ly > >>> - - - - - - - - - - - - - - - - - > >>> > >>> On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman > >>> wrote: > >>> > Yea, when I try to set type to async (exactly like the example) I get: > log4j:WARN No such property [producerType] in > kafka.producer.KafkaLog4jAppender. > > On Tue Feb 24 2015 at 1:35:54 PM Joe Stein > wrote: > > > Here is sample log4j.properties > > https://github.com/stealthly/scala-kafka/blob/master/src/tes > > t/resources/log4j.properties#L54-L67 > > > > I _almost_ have always pulled the class > > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ > > scala/kafka/producer/KafkaLog4jAppender.scala > > internal > > to private repo and changed it as things came up... e.g. > setSource(), > > setTags() blah blah... > > > > Paul Otto has an open source version > > https://github.com/potto007/kafka-appender-layout that you could > try > out > > too that he built to tackle some of the layout things. > > > > ~ Joe Stein > > - - - - - - - - - - - - - - - - - > > > > http://www.stealth.ly > > - - - - - - - - - - - - - - - - - > > > > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville > > > wrote: > > > > > ^^ I would really appreciate this as well. It's unclear how to get > log4j > > > working with Kafka when you have no prior experience with log4j. > > > > > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman < > sc...@woofplanet.com> > > > wrote: > > > > > > > Thanks. But we're using log4j. I tried setting the type to async > but it > > > > generated a warning of no such field. Is there any real > documentation > > on > > > > the log4j appender? > > > > > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > > > sschlans...@opentable.com> wrote: > > > > > > > > > We just configure our logback.xml to have two Appenders, an > > > AsyncAppender > > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as > > appropriate). > > > > > > > > > > AsyncAppender removes more failure cases too, e.g. a health > check > > > hanging > > > > > rather than returning rapidly could block you application. > > > > > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > > > anthony.musy...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Theres also another one here. > > > > > > > > > > > > https://github.com/danielwegener/logback-kafka-appender. > > > > > > > > > > > > It has a fallback appender which might address the issue of > Kafka > > > being > > > > > > un-available. > > > > > > > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > >> Here’s my attempt at a Logback version, should be fairly > easily > > > > ported: > > > > > >> > > > > > >> https://github.com/opentable/ > otj-logging/blob/master/kafka/ > > > > > src/main/java/com/opentable/logging/KafkaAppender.java > > > > > >> > > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman < > sc...@woofplanet.com> > > > > > wrote: > > > > > >> > > > > > >>> I am just starting to use it and could use a little > guidance. I > > was > > > > > able > > > > > >> to > > > > > >>> get it working with 0.8.2 but am not clear on best > practices for > > > > using > > > > > >> it. > > > > > >>> > > > > > >>> Anyway willing to help me out a bit? Got a few questions, > like > > how > > > to > > > > > >>> protect applications from when kafka is down or > unreachable. > > > > > >>> > > > > > >>> It seems like a great idea
java.nio.channels.ClosedChannelException
Hi, I am learning kafka, ... hard way
I try to run the example given here:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Whatever I do i I have
java.nio.channels.ClosedChannelException when try to send message
Exception in thread "main" java.nio.channels.ClosedChannelExceptionat
kafka.network.BlockingChannel.send(BlockingChannel.scala:100)at
kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:78)at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
And, sometimes it's
2015-02-24 13:46:51 INFO SimpleConsumer:68 - Reconnect due to socket
error: java.io.EOFException: Received -1 when reading from channel, socket
has likely been closed.
(in all cases it fails in findLeader())
At the same some other consumers work, as e.g. the command line console-consumer
Now, I put a very small code snippet (below), that seems should do the job, but
it's not, something missing. Please help.
public class Test1 {public static void main(String[] args) {String host
= "localhost"; int port = 2181; String topic = "test"; int soTimeout = -1;
int bufferSize = 10; String clientId = "me-aly"; SimpleConsumer consumer
= new SimpleConsumer(host, port, soTimeout, bufferSize, clientId);List
topics = Collections.singletonList(topic);
TopicMetadataRequest req = new TopicMetadataRequest(topics);
kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);
System.out.println(resp); }}
what groupID does camus use?
I looked at the example camus.properties, it does not specify this. how could we set the groupId ? thanks Yang
Re: KafkaProducer from kafka.clients hangs when some partitions are not available
Xiaoyu,
For 1, I have a patch for 0.8.2 in
https://issues.apache.org/jira/browse/KAFKA-1984. Could you test it out and
see if it fixes your issue?
For 2, I did some local testing. The only issue I saw is that producer can
block on close since there are still unsent messages in the bufferpool.
This is a known issue and is being tracked in
https://issues.apache.org/jira/browse/KAFKA-1788. Could you confirm whether
your producer blocks during send or during close (you can figure it out by
taking a thread dump)?
Thanks,
Jun
On Tue, Feb 24, 2015 at 10:14 AM, Xiaoyu Wang wrote:
> Jun,
>
> Can you also take a look at the second problem I am having?
>
> > > > I am trying to test how KafkaProducer behaves with topic replication
> > > factor
> > > > = 1
> > > >
> > > >1. One broker is offline BEFORE KafkaProducer starts sending
> > messages.
> > > >Because of the bug I mentioned, KafkaProducer sends to the
> offline
> > > >partition and hangs forever.
>
>
> *> > >2. One broker goes offline WHILE KafkaProducer is sending
> messages. > > >KafkaProducer seems to be hanging forever in this case.
> I am still > > > looking.*
>
> On Tue, Feb 24, 2015 at 12:03 PM, Jun Rao wrote:
>
> > Ah, yes. You are right. That's a more obvious bug. Will fix that in
> > KAFKA-1984.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Feb 24, 2015 at 8:37 AM, Xiaoyu Wang
> wrote:
> >
> > > Hi Jun,
> > >
> > > If I understand it correctly. the highlighted line is for avoiding
> > > offline partitions, is it?
> > >
> > > for (int i = 0; i < numPartitions; i++) {
> > > int partition = Utils.abs(counter.getAndIncrement()) %
> numPartitions;
> > > if (partitions.get(partition).leader() != null) {
> > > return partition; --> should be changed to return the actual
> > > partition number?
> > > }
> > > }
> > >
> > >
> > > On Tue, Feb 24, 2015 at 11:30 AM, Jun Rao wrote:
> > >
> > > > Hi, Xiaoyu,
> > > >
> > > > 1. Could you explain a bit more what the bug is? The code does try to
> > > avoid
> > > > picking an unavailable partition. There does seem to be an issue when
> > > there
> > > > are more than one thread producing data to the same producer
> instance.
> > > This
> > > > is being tracked in KAFKA-1984. How many producing threads do you
> have
> > in
> > > > your test?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Feb 24, 2015 at 7:56 AM, Xiaoyu Wang
> > > wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > I am trying to test how KafkaProducer behaves with topic
> replication
> > > > factor
> > > > > = 1
> > > > >
> > > > >1. One broker is offline BEFORE KafkaProducer starts sending
> > > messages.
> > > > >Because of the bug I mentioned, KafkaProducer sends to the
> > offline
> > > > >partition and hangs forever.
> > > > >2. One broker goes offline WHILE KafkaProducer is sending
> > messages.
> > > > >KafkaProducer seems to be hanging forever in this case. I am
> still
> > > > > looking.
> > > > >Do you mind take a look?
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 23, 2015 at 7:01 PM, Jun Rao wrote:
> > > > >
> > > > > > The logic in that code is to cycle through all partitions and
> > return
> > > as
> > > > > > soon as we see a partition with the leader. I do see an issue
> that
> > if
> > > > > there
> > > > > > are multiple threads sending messages to the same producer
> > > > concurrently,
> > > > > we
> > > > > > may not cycle through all partitions and therefore we could
> return
> > an
> > > > > > unavailable partition even when available partitions are present.
> > > > > >
> > > > > > Do you see this issue with just a single thread producing
> messages?
> > > The
> > > > > > current logic seems to work correctly in that case.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Fri, Feb 20, 2015 at 12:45 PM, Xiaoyu Wang <
> > xw...@rocketfuel.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Found the problem - it is a bug with Partitions of kafka
> client.
> > > Can
> > > > > you
> > > > > > > guys confirm and patch in kafka clients?
> > > > > > >
> > > > > > > for (int i = 0; i < numPartitions; i++) {
> > > > > > > int partition = Utils.abs(counter.getAndIncrement()) %
> > > > > numPartitions;
> > > > > > > if (partitions.get(partition).leader() != null) {
> > > > > > > return partitions.get(partition).partition();
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 20, 2015 at 2:35 PM, Xiaoyu Wang <
> > xw...@rocketfuel.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Update:
> > > > > > > >
> > > > > > > > I am using kafka.clients 0.8.2-beta. Below are the test
> steps
> > > > > > > >
> > > > > > > >1. setup local kafka clusters with 2 brokers, 0 and 1
> > > > > > > >2. create topic X with replication fact 1 and 4 partitions
> > > > > > > >3. verify that each broker h
Re: Anyone using log4j Appender for Kafka?
Sounds like https://issues.apache.org/jira/browse/KAFKA-1788 maybe On Feb 24, 2015 2:28 PM, "Scott Chapman" wrote: > Yea, however I don't get async behavior. When kafka is down the log blocks, > which is kinda nasty to my app. > > On Tue Feb 24 2015 at 2:27:09 PM Joe Stein wrote: > > > Producer type isn't needed anymore with the new producer so in the the > > logger properties just leave that out in 0.8.2 and it should work. > > > > On Tue, Feb 24, 2015 at 2:24 PM, Joe Stein wrote: > > > > > Interesting, looks like a breaking change from 0.8.1 > > > https://github.com/apache/kafka/blob/0.8.1/core/src/ > > main/scala/kafka/producer/KafkaLog4jAppender.scala > > > to 0.8.2 > > > https://github.com/apache/kafka/blob/0.8.2/core/src/ > > main/scala/kafka/producer/KafkaLog4jAppender.scala > > > > > > On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein > wrote: > > > > > >> and kafka too :) > > >> > > >> On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein > > wrote: > > >> > > >>> are you including > > >>> > https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 > > >>> in your project? > > >>> > > >>> ~ Joe Stein > > >>> - - - - - - - - - - - - - - - - - > > >>> > > >>> http://www.stealth.ly > > >>> - - - - - - - - - - - - - - - - - > > >>> > > >>> On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman > > > >>> wrote: > > >>> > > Yea, when I try to set type to async (exactly like the example) I > get: > > log4j:WARN No such property [producerType] in > > kafka.producer.KafkaLog4jAppender. > > > > On Tue Feb 24 2015 at 1:35:54 PM Joe Stein > > wrote: > > > > > Here is sample log4j.properties > > > https://github.com/stealthly/scala-kafka/blob/master/src/tes > > > t/resources/log4j.properties#L54-L67 > > > > > > I _almost_ have always pulled the class > > > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ > > > scala/kafka/producer/KafkaLog4jAppender.scala > > > internal > > > to private repo and changed it as things came up... e.g. > > setSource(), > > > setTags() blah blah... > > > > > > Paul Otto has an open source version > > > https://github.com/potto007/kafka-appender-layout that you could > > try > > out > > > too that he built to tackle some of the layout things. > > > > > > ~ Joe Stein > > > - - - - - - - - - - - - - - - - - > > > > > > http://www.stealth.ly > > > - - - - - - - - - - - - - - - - - > > > > > > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville < > amelvi...@g.hmc.edu > > > > > > wrote: > > > > > > > ^^ I would really appreciate this as well. It's unclear how to > get > > log4j > > > > working with Kafka when you have no prior experience with log4j. > > > > > > > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman < > > sc...@woofplanet.com> > > > > wrote: > > > > > > > > > Thanks. But we're using log4j. I tried setting the type to > async > > but it > > > > > generated a warning of no such field. Is there any real > > documentation > > > on > > > > > the log4j appender? > > > > > > > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > We just configure our logback.xml to have two Appenders, an > > > > AsyncAppender > > > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as > > > appropriate). > > > > > > > > > > > > AsyncAppender removes more failure cases too, e.g. a health > > check > > > > hanging > > > > > > rather than returning rapidly could block you application. > > > > > > > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > > > > anthony.musy...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Theres also another one here. > > > > > > > > > > > > > > https://github.com/danielwegener/logback-kafka-appender. > > > > > > > > > > > > > > It has a fallback appender which might address the issue > of > > Kafka > > > > being > > > > > > > un-available. > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > > > >> Here’s my attempt at a Logback version, should be fairly > > easily > > > > > ported: > > > > > > >> > > > > > > >> https://github.com/opentable/ > > otj-logging/blob/master/kafka/ > > > > > > src/main/java/com/opentable/logging/KafkaAppender.java > > > > > > >> > > > > > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman < > > sc...@woofplanet.com> > > > > > > wrote: > > > > > > >> > > > > > > >>> I am just starting to use it and could use a little > > guidance. I > > > was > > > > > > able > > > >
Stream naming conventions?
Hello all, Just wondering if those with a good amount of experience using Kafka in production with many streams have converged on any sort of naming convention. If so would you be willing to share? Thanks in advance, Taylor
RE: Stream naming conventions?
We have a global namespace hierarchy for topics that is exactly our Avro namespace with Class Name. The template is basically: .Core. .. The up side of this for us is that since the topics are named based on the Avro schema namespace and type, we can look up the avro schema in the Avro Schema Repository using the topic name, and the schema ID coded into the message. Each product then also has the flexibility of defining whatever topics they find useful. Hope this helps, Thunder -Original Message- From: Taylor Gautier [mailto:tgaut...@yahoo.com.INVALID] Sent: Tuesday, February 24, 2015 12:11 PM To: kafka-us...@incubator.apache.org Subject: Stream naming conventions? Hello all, Just wondering if those with a good amount of experience using Kafka in production with many streams have converged on any sort of naming convention. If so would you be willing to share? Thanks in advance, Taylor
Re: Anyone using log4j Appender for Kafka?
nah, it is expected behavior for a synchronous call, it waits and timesout. Sorry, should have been more specific. I was really looking for async. On Tue Feb 24 2015 at 3:56:19 PM Joe Stein wrote: > Sounds like https://issues.apache.org/jira/browse/KAFKA-1788 maybe > On Feb 24, 2015 2:28 PM, "Scott Chapman" wrote: > > > Yea, however I don't get async behavior. When kafka is down the log > blocks, > > which is kinda nasty to my app. > > > > On Tue Feb 24 2015 at 2:27:09 PM Joe Stein wrote: > > > > > Producer type isn't needed anymore with the new producer so in the the > > > logger properties just leave that out in 0.8.2 and it should work. > > > > > > On Tue, Feb 24, 2015 at 2:24 PM, Joe Stein > wrote: > > > > > > > Interesting, looks like a breaking change from 0.8.1 > > > > https://github.com/apache/kafka/blob/0.8.1/core/src/ > > > main/scala/kafka/producer/KafkaLog4jAppender.scala > > > > to 0.8.2 > > > > https://github.com/apache/kafka/blob/0.8.2/core/src/ > > > main/scala/kafka/producer/KafkaLog4jAppender.scala > > > > > > > > On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein > > wrote: > > > > > > > >> and kafka too :) > > > >> > > > >> On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein > > > wrote: > > > >> > > > >>> are you including > > > >>> > > https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 > > > >>> in your project? > > > >>> > > > >>> ~ Joe Stein > > > >>> - - - - - - - - - - - - - - - - - > > > >>> > > > >>> http://www.stealth.ly > > > >>> - - - - - - - - - - - - - - - - - > > > >>> > > > >>> On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman < > sc...@woofplanet.com > > > > > > >>> wrote: > > > >>> > > > Yea, when I try to set type to async (exactly like the example) I > > get: > > > log4j:WARN No such property [producerType] in > > > kafka.producer.KafkaLog4jAppender. > > > > > > On Tue Feb 24 2015 at 1:35:54 PM Joe Stein > > > wrote: > > > > > > > Here is sample log4j.properties > > > > https://github.com/stealthly/scala-kafka/blob/master/src/tes > > > > t/resources/log4j.properties#L54-L67 > > > > > > > > I _almost_ have always pulled the class > > > > https://github.com/apache/kafka/blob/0.8.2/core/src/main/ > > > > scala/kafka/producer/KafkaLog4jAppender.scala > > > > internal > > > > to private repo and changed it as things came up... e.g. > > > setSource(), > > > > setTags() blah blah... > > > > > > > > Paul Otto has an open source version > > > > https://github.com/potto007/kafka-appender-layout that you > could > > > try > > > out > > > > too that he built to tackle some of the layout things. > > > > > > > > ~ Joe Stein > > > > - - - - - - - - - - - - - - - - - > > > > > > > > http://www.stealth.ly > > > > - - - - - - - - - - - - - - - - - > > > > > > > > On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville < > > amelvi...@g.hmc.edu > > > > > > > > wrote: > > > > > > > > > ^^ I would really appreciate this as well. It's unclear how to > > get > > > log4j > > > > > working with Kafka when you have no prior experience with > log4j. > > > > > > > > > > On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman < > > > sc...@woofplanet.com> > > > > > wrote: > > > > > > > > > > > Thanks. But we're using log4j. I tried setting the type to > > async > > > but it > > > > > > generated a warning of no such field. Is there any real > > > documentation > > > > on > > > > > > the log4j appender? > > > > > > > > > > > > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker < > > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > > > We just configure our logback.xml to have two Appenders, > an > > > > > AsyncAppender > > > > > > > -> KafkaAppender, and FileAppender (or ConsoleAppender as > > > > appropriate). > > > > > > > > > > > > > > AsyncAppender removes more failure cases too, e.g. a > health > > > check > > > > > hanging > > > > > > > rather than returning rapidly could block you application. > > > > > > > > > > > > > > On Feb 22, 2015, at 11:26 PM, anthony musyoki < > > > > > anthony.musy...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Theres also another one here. > > > > > > > > > > > > > > > > https://github.com/danielwegener/logback-kafka-appender > . > > > > > > > > > > > > > > > > It has a fallback appender which might address the issue > > of > > > Kafka > > > > > being > > > > > > > > un-available. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker < > > > > > > > > sschlans...@opentable.com> wrote: > > > > > > > > > > > > > > > >> Here’s my attempt at a Logback version, should be > fairly > > > easil
Re: Auto topic creation occurring when auto.create.topics.enable=false
That I don't know Jun. I dug a little into the Ruby source yesterday, and it appeared that we might be affected by the ruby client issuing a topic metadata request explicitly. I was concerned that we were triggering the behaviour seen in https://issues.apache.org/jira/browse/KAFKA-1507 However, I'll check your suggestion on the ZK bypass. On 23 February 2015 at 17:32, Jun Rao wrote: > Does the ruby library write to ZK directly to create topics? That will > bypass the checking on the broker side. > > Thanks, > > Jun > > On Mon, Feb 23, 2015 at 3:06 AM, David Corley > wrote: > > > Hey all, > > I'm trying to run some basic error-handling validation with some client > > code, and I'm attempting to handle an UnknownTopicOrPartitionException. > To > > set this scenario up, I wanted to attempt fetch messages from a topic I > > know doesn't exist. To that end, I've got a 3-broker cluster with: > > * auto.create.topics.enable=false > > * each broker restarted after setting the above > > > > The logs for each broker show the following after the restart: > > [2015-02-23 10:13:30,315] INFO Property auto.create.topics.enable is > > overridden to false (kafka.utils.VerifiableProperties) > > > > However, every time I try to fetch messages from the broker, I get no > > exception, and kafka-list-topics shows the previously non-existent topic > is > > now created! > > > > I'm using poseidon/poseidon-cluster ruby libraries for the comms, but > don't > > believe they should be over-riding the broker configuration in terms of > > topic creation. Nothing in source code suggests anything, and I don't > > believe the protocol allows for it anyway. > > > > We're using Kafka 0.8.0 final for all brokers. Anyone come across this > > issue before? > > > > /Dave > > >
latest messages do not show up on consumers
in the following code, I just create a new topic, send some messages onto the new topic with console producer, then read it with console consumer. but after the 3rd line, nothing is printed out. export topic=ttnew2 sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create --topic $topic --partitions 1 --replication-factor 1 echo 'asdfasdfsadffadsfasf' | sudo -u kafka bin/kafka-console-producer.sh --broker-list localhost:9092 --topic $topic sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 --topic $topic -group f shouldn't I expect the consumer on the 3rd line to read out the messages? in fact if I give it --from-beginning, it works fine this is kafka_2.9.2-0.8.1.1 Thanks Yang
Re: latest messages do not show up on consumers
somehow I have to let the consumer run once BEFORE the messages are produced. then run consumer after messages are produced, then it works: export topic=ttnew21 sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create --topic $topic --partitions 1 --replication-factor 1 echo "first try" sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 --consumer-timeout-ms 1000 --topic $topic -group fxx echo 'asdfasdfsadffadsfasf' | sudo -u kafka bin/kafka-console-producer.sh --broker-list localhost:9092 --topic $topic echo "second try" sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 --consumer-timeout-ms 1000 --topic $topic -group fxx the first try doesn't read anything out (of course) ; the second one works. but if I remove the first try, the remaining consumer can't read anything either. looks like a bug? On Tue, Feb 24, 2015 at 1:51 PM, Yang wrote: > in the following code, I just create a new topic, send some messages onto > the new topic with console producer, then read it with console consumer. > but after the 3rd line, nothing is printed out. > > > > export topic=ttnew2 > sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create > --topic $topic --partitions 1 --replication-factor 1 > echo 'asdfasdfsadffadsfasf' | sudo -u kafka > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic $topic > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 > --topic $topic -group f > > > shouldn't I expect the consumer on the 3rd line to read out the messages? > in fact if I give it --from-beginning, it works fine > > > > this is kafka_2.9.2-0.8.1.1 > > > Thanks > Yang >
"at least once" consumer recommendations for a load of 5 K messages/second
Hi, It is a little long, since I wanted to explain the use case and then ask questions, so thanks for your attention Use case: We have a use case where everything in the queue has to be consumed at least once. So the consumer has to have "consumed" (saved in some destination database) the message before confirming consumption to kafka (or ZK). Now it is possible and from what I have read so far we will have consumer groups and partitions. Here are some facts/numbers for our case * We will potentially have messages with peaks of 5k /second. * We can play with the message size if that makes any difference (keep it < 100 bytes for a link or put the entire message avg size of 2-5K bytes). * We do not need replication, but might have a kafka cluster to handle the load. * Also work consumption will take anywhere from 300-500ms, generally we would like the consumer to be not behind by more than 1-2 minutes. So if the message shows up in a queue, it should show up in the database within 2 minutes. The questions I have are * If this has been covered before, please point me to it. Thanks * Is that possible/recommended "controlled commit per consumed message" for this load (have read about some concerns on ZK issues)? * Are there any recommendations on configurations in terms of partitions to number of messages OR consumers? Maybe more queues/topics * Anything else that we might need to watch out for? * As for the client, I should be able to do this (control when the offset commit happens) with high level consumer I suppose? Thanks Anand
Re: How to find the number of message in the topic
At a fixed number. I think my problem was that I assuming that there
was a single partition in my code ... still working on this.
On Tue, Feb 24, 2015 at 11:06 AM, Zakee wrote:
> Does that count get frozen on a fixed number or any random number?
>
> -Zakee
>
> On Mon, Feb 23, 2015 at 9:48 AM, Stuart Reynolds
> wrote:
>
>> See SimpleConsumer. getOffsetsBefore
>> and the getLastOffset example here:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>> whichTime=-1 or -2 (depending whether you want the latest or earliest
>> offset.
>>
>> However, I've been banging my head against the wall with this API --
>> it seems that, even with a single broker and a single host, sometimes
>> the latest offset gets frozen as new data comes in. :-/
>>
>>
>>
>> public static long getLastOffset(SimpleConsumer consumer, String
>> topic, int partition,
>> long whichTime, String clientName) {
>> TopicAndPartition topicAndPartition = new
>> TopicAndPartition(topic, partition);
>> Map requestInfo
>> = new HashMap();
>> requestInfo.put(topicAndPartition, new
>> PartitionOffsetRequestInfo(whichTime, 1));
>> kafka.javaapi.OffsetRequest request = new
>> kafka.javaapi.OffsetRequest(requestInfo,
>> kafka.api.OffsetRequest.CurrentVersion(),clientName);
>> OffsetResponse response = consumer.getOffsetsBefore(request);
>>
>> if (response.hasError()) {
>> System.out.println("Error fetching data Offset Data the
>> Broker. Reason: " + response.errorCode(topic, partition) );
>> return 0;
>> }
>> long[] offsets = response.offsets(topic, partition);
>> return offsets[0];
>> }
>>
>> On Sun, Feb 22, 2015 at 9:05 PM, Bhuvana Baskar
>> wrote:
>> > Hi,
>> >
>> > Please let me know how to find the total number of messages in a
>> particular
>> > topic.
>> >
>> > Regards,
>> > Bhuvana
>>
>> Heavy rains mean flooding
>> Anywhere it rains it can flood. Learn your risk. Get flood insurance.
>> http://thirdpartyoffers.netzero.net/TGL3255/54eb6db072a8f6db07624mp13duc
>>
>>
Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 1
Joe, It turns out that KAFKA-1724 is already resolved in trunk through another jira. However, the change in that jira is big. Since KAFKA-1724 seems to only happen when there is ZK session timeout or the ZK service is not available, I don't think it's worth fixing the issue in 0.8.2. Thanks, Jun On Mon, Feb 23, 2015 at 2:50 PM, Jun Rao wrote: > Hi, Joe, > > Left a comment on KAFKA-1724. Not sure if it's a true blocker. However, we > can probably include it if it can be fixed in the next couple of days. > > Thanks, > > Jun > > On Sun, Feb 22, 2015 at 2:12 PM, Joe Stein wrote: > >> Jun, >> >> Can we also add https://issues.apache.org/jira/browse/KAFKA-1724 to the >> next RC please? >> >> Thanks! >> >> ~ Joe Stein >> - - - - - - - - - - - - - - - - - >> >> http://www.stealth.ly >> - - - - - - - - - - - - - - - - - >> >> On Sun, Feb 22, 2015 at 11:59 AM, Jun Rao wrote: >> >>> We identified at least one more blocker issue KAFKA-1971 during testing. >>> So, we will have to roll another RC for 0.8.2.1. >>> >>> Thanks, >>> >>> Jun >>> >>> On Sat, Feb 21, 2015 at 6:04 PM, Joe Stein wrote: >>> Source verified, tests pass, quick start ok. Binaries verified, tests on scala https://github.com/stealthly/scala-kafka/pull/27 and go clients https://github.com/stealthly/go_kafka_client/pull/55 passing. If the release passes we should update the release notes to include the change from KAFKA-1729 please. +1 (binding) ~ Joe Stein On Fri, Feb 20, 2015 at 9:08 PM, ted won wrote: > +1 > > On Friday, February 20, 2015, Guozhang Wang > wrote: > > > +1 binding. > > > > Checked the md5, and quick start. > > > > Some minor comments: > > > > 1. The quickstart section would better include the building step > after > > download and before starting server. > > > > 2. There seems to be a bug in Gradle 1.1x with Java 8 causing the > "gradle" > > initialization to fail: > > > > - > > > > FAILURE: Build failed with an exception. > > > > * Where: > > Build file '/home/guwang/Workspace/temp/kafka/build.gradle' line: 199 > > > > * What went wrong: > > A problem occurred evaluating root project 'kafka'. > > > Could not create task of type 'ScalaDoc'. > > -- > > > > Downgrading Java to 1.7 resolve this issue. > > > > Guozhang > > > > On Wed, Feb 18, 2015 at 7:56 PM, Connie Yang > > wrote: > > > > > +1 > > > On Feb 18, 2015 7:23 PM, "Matt Narrell" > > wrote: > > > > > > > +1 > > > > > > > > > On Feb 18, 2015, at 7:56 PM, Jun Rao > > wrote: > > > > > > > > > > This is the first candidate for release of Apache Kafka > 0.8.2.1. This > > > > > only fixes one critical issue (KAFKA-1952) in 0.8.2.0. > > > > > > > > > > Release Notes for the 0.8.2.1 release > > > > > > > > > > > > > > > https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/RELEASE_NOTES.html > > > > > > > > > > *** Please download, test and vote by Saturday, Feb 21, 7pm PT > > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the > release: > > > > > http://kafka.apache.org/KEYS in addition to the md5, sha1 > > > > > and sha2 (SHA256) checksum. > > > > > > > > > > * Release artifacts to be voted upon (source and binary): > > > > > https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/ > > > > > > > > > > * Maven artifacts to be voted upon prior to release: > > > > > https://repository.apache.org/content/groups/staging/ > > > > > > > > > > * scala-doc > > > > > > https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/scaladoc/ > > > > > > > > > > * java-doc > > > > > > https://people.apache.org/~junrao/kafka-0.8.2.1-candidate1/javadoc/ > > > > > > > > > > * The tag to be voted upon (off the 0.8.2 branch) is the > 0.8.2.1 tag > > > > > > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=c1b4c58531343dce80232e0122d085fc687633f6 > > > > > > > > > > /*** > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > -- > > -- Guozhang > > > -- You received this message because you are subscribed to the Google Groups "kafka-clients" group. To unsubscribe from this group and stop receiving emails from it, send an email to kafka-clients+unsubscr...@googlegroups.com. To post to this group, send email to kafka-clie...@googlegroups.com. Visit this group at http://groups.google.com/group/kafka-clients. To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/CAA7ooCDvUNQx2B351P3LaOYA
Re: NetworkProcessorAvgIdlePercent
What about RequestHandlerAvgIdlePercent? Thanks, Jun On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote: > Hi Jun, > > With ~100G of data being pushed per hour across 35 topics > (replication-factor 3), the NetworkProcessorAvgIdlePercent is mostly > showing below 0.5 sometimes when the producers send on a high rate. > > Thanks > -Zakee > > On Sun, Feb 22, 2015 at 10:29 PM, Jun Rao wrote: > > > What kind of load do you have on the brokers? On an idle cluster (just > > fetch requests from the follower replicas), I > > saw NetworkProcessorAvgIdlePercent at about 97%. > > > > Thanks, > > > > Jun > > > > On Thu, Feb 19, 2015 at 5:19 PM, Zakee wrote: > > > > > Jun, > > > > > > I am already using the latest release 0.8.2.1. > > > > > > -Zakee > > > > > > On Thu, Feb 19, 2015 at 2:46 PM, Jun Rao wrote: > > > > > > > Could you try the 0.8.2.1 release being voted on now? It fixes a CPU > > > issue > > > > and should reduce the CPU load in network thread. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Thu, Feb 19, 2015 at 11:54 AM, Zakee wrote: > > > > > > > > > Kafka documentation recommends <0.3 for above metric. I assume > > > processor > > > > is > > > > > busier if this goes below 0.3 and obviously it being < 0.3 for long > > > does > > > > > not seem to be a good sign. > > > > > > > > > > What should be our criteria to raise an alert, I though it should > be > > > > when > > > > > its value goes below 0.3. However, the value seems to be below 0.3 > a > > > lot > > > > of > > > > > the times, almost always if we take samples every five mins. What > > > should > > > > be > > > > > the threshold to raise an alarm ? > > > > > > > > > > What would be the impact of having this below 0.3 or even zero like > > > most > > > > of > > > > > the times? > > > > > > > > > > > > > > > -Zakee > > > > > > > > > > How Old Men Tighten Skin > > > > > 63 Year Old Man Shares DIY Skin Tightening Method You Can Do From > > Home > > > > > > > > > http://thirdpartyoffers.netzero.net/TGL3231/54e63f5bda4c23f5b6560st02vuc > > > > > > > > 8% Annuity Return Secret > > > > Earn Guaranteed Income for Life! Compare Rates Today. > > > > > > http://thirdpartyoffers.netzero.net/TGL3255/54e6782bcbe78782b37bdmp15duc > > > > > > > High School Yearbooks > > View Class Yearbooks Online Free. Reminisce & Buy a Reprint Today! > > http://thirdpartyoffers.netzero.net/TGL3255/54eac962abad049627e56mp03duc >
Re: After Leadership Election and "kafka.log" JMXBean Registration Process
These two metrics are always registered, whether the replica is the leader or the follower. Thanks, Jun On Mon, Feb 23, 2015 at 6:40 PM, Bhavesh Mistry wrote: > Hi Kafka Team or User Community , > > After leadership election or switch between follower/leader of partition > for given topic, does following metrics JMX bean gets registered (on > leader) and de-registered (on follower). > > LogEndOffset > Size > LogStartOffset > > eg: > "kafka.log":type="Log",name="TOPIC-17-*Size*" > "kafka.log":type="Log",name="TOPIC-17-*LogStartOffset*" > "kafka.log":type="Log",name="TOPIC-17-*LogEndOffset*" > > > Thanks in advance for your help !! > > Thanks, > > Bhaevsh >
Re: latest messages do not show up on consumers
If a consumer comes from a new consumer group, it will by default consume from the log end. In you first scenario, because you started consumer after producing data, when the consumer starts it will consume from the latest. So you will not be able to see the messages even though they are in the broker. In your second scenario, because consumer will commit the offsets periodically and during rebalance. The first run of consumer will commit offset to be 0. Then you produced some data and run the second consumer. Because the second run of consumer uses the same consumer group, it will pick up the committed offset and consume from 0. In this case you will see the produced messages. If you use a different consumer group in your second run of consumer, you will not be able to see the messages because the offset will be reset to latest like in the first scenario. -Jiangjie Qin On 2/24/15, 1:56 PM, "Yang" wrote: >somehow I have to let the consumer run once BEFORE the messages are >produced. then run consumer after messages are produced, then it works: > > >export topic=ttnew21 > sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create >--topic $topic --partitions 1 --replication-factor 1 >echo "first try" > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 >--consumer-timeout-ms 1000 --topic $topic -group fxx > echo 'asdfasdfsadffadsfasf' | sudo -u kafka >bin/kafka-console-producer.sh >--broker-list localhost:9092 --topic $topic >echo "second try" > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 >--consumer-timeout-ms 1000 --topic $topic -group fxx > > >the first try doesn't read anything out (of course) ; the second one >works. >but if I remove the first try, the remaining consumer can't read anything >either. > > > >looks like a bug? > >On Tue, Feb 24, 2015 at 1:51 PM, Yang wrote: > >> in the following code, I just create a new topic, send some messages >>onto >> the new topic with console producer, then read it with console consumer. >> but after the 3rd line, nothing is printed out. >> >> >> >> export topic=ttnew2 >> sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create >> --topic $topic --partitions 1 --replication-factor 1 >> echo 'asdfasdfsadffadsfasf' | sudo -u kafka >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic >>$topic >> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181 >> --topic $topic -group f >> >> >> shouldn't I expect the consumer on the 3rd line to read out the >>messages? >> in fact if I give it --from-beginning, it works fine >> >> >> >> this is kafka_2.9.2-0.8.1.1 >> >> >> Thanks >> Yang >>
Re: Producing message set
A messageSet does get written to the local log file atomically. However, the set may not be fetched atomically by the replica fetcher. So, on switching the leader of a partition, it's possible that only a subset of the messages in the set show up in the new leader. The exception is that if the messageSet is compressed, it will show up all or nothing on failover. Thanks, Jun On Wed, Feb 11, 2015 at 6:13 AM, Piotr Husiatyński wrote: > Hi, > > I'm writing new client library for kafka and I cannot find information > about > atomicity of message set publishing. I'm implementing protocol described on > wiki [1]. According to documentation, it is possible to send within single > request more than one message - message set for any partition. > In response, there is error code for every partition separately. Does this > mean, that while writing messages to different partitions can fail, writing > multiple messages to one partition as message set is atomic and can either > fail or succeed for all of them? > > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol >
Re: Issue with auto topic creation as well
Hmm, anything special with that topic (e.g., did you run any admin operation on it)? Also, does bouncing the brokers fix the issue? Thanks, Jun On Tue, Feb 10, 2015 at 1:31 PM, Sumit Rangwala wrote: > On Thu, Feb 5, 2015 at 12:37 PM, Sumit Rangwala > wrote: > > > > > > > On Wed, Feb 4, 2015 at 9:23 PM, Jun Rao wrote: > > > >> Could you try the 0.8.2.0 release? It fixed one issue related to topic > >> creation. > >> > > > Jun, > > If you need more info let me know. Seems like TopicMetadataResponse is > expecting more fields then what is present in the response. > > > Sumit > > > > > > > Jun, > > > > Tried with 0.8.2.0 and I still see the same error. > > > > I see the error given below almost incessantly on the client side for > > topic LAX1-GRIFFIN-r47-1423165897627. Things look fine one the broker > side: > > Topic:LAX1-GRIFFIN-r47-1423165897627 PartitionCount:8 ReplicationFactor:1 > > Configs: > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 0 Leader: 49817 > Replicas: > > 49817 Isr: 49817 > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 1 Leader: 49818 > Replicas: > > 49818 Isr: 49818 > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 2 Leader: 49814 > Replicas: > > 49814 Isr: 49814 > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 3 Leader: 49817 > Replicas: > > 49817 Isr: 49817 > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 4 Leader: 49818 > Replicas: > > 49818 Isr: 49818 > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 5 Leader: 49814 > Replicas: > > 49814 Isr: 49814 > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 6 Leader: 49817 > Replicas: > > 49817 Isr: 49817 > > Topic: LAX1-GRIFFIN-r47-1423165897627 Partition: 7 Leader: 49818 > Replicas: > > 49818 Isr: 49818 > > > > > > Broker logs at http://d.pr/f/1jLOf/5H8uWPqu in case you need them > > > > ERROR: > > 2015-02-05 12:02:03,764 > > > (LAX1-GRIFFIN-r47-1423165897627-pf6898-lax1-GriffinDownloader-1423166442395_f27c6501f3d5-1423166444213-d572da8f-leader-finder-thread) > > ClientUtils$ WARN: Fetching topic metadata with correlation id 152 for > > topics [Set(LAX1-GRIFFIN-r47-1423165897627)] from broker > > [id:49814,host:172.16.204.44,port:49814] failed > > java.lang.ArrayIndexOutOfBoundsException: 7 > >at > > > kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38) > >at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78) > >at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36) > >at > > > kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31) > >at > > > kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31) > >at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) > >at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) > >at scala.collection.immutable.Range.foreach(Range.scala:81) > >at > scala.collection.TraversableLike$class.map(TraversableLike.scala:233) > >at scala.collection.immutable.Range.map(Range.scala:46) > >at > > kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31) > >at kafka.producer.SyncProducer.send(SyncProducer.scala:114) > >at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) > >at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) > >at > > > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) > >at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) > > > > > > > > > > > > > >> > >> Thanks, > >> > >> Jun > >> > >> On Wed, Feb 4, 2015 at 12:58 AM, Sumit Rangwala < > sumitrangw...@gmail.com> > >> wrote: > >> > >> > I am observing the following exception with kafka client: > >> > > >> > 2015-02-04 00:17:27,345 > >> > > >> > > >> > (LAX1-GRIFFIN-r8-1423037468055-pf13797-lax1-GriffinDownloader-1423037818264_c7b1e843ff51-1423037822122-eb7afca7-leader-finder-thread) > >> > ClientUtils$ WARN: Fetching topic metadata with correlation id 112 for > >> > topics [Set(LAX1-GRIFFIN-r8-1423037468055)] from broker > >> > [id:49649,host:172.16.204.44,port:49649] failed > >> > java.lang.ArrayIndexOutOfBoundsException: 7 > >> > at > >> > > >> > > >> > kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38) > >> > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78) > >> > at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36) > >> > at > >> > > >> > > >> > kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31) > >> > at > >> > > >> > > >> > kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31) > >> > at > >> > > >> > > >> > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) > >> > at > >> > > >> > > >> > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233) > >> > at scala.collection.immutable.Range.foreach(Range.scala:81
Re: java.nio.channels.ClosedChannelException
I usually got that exception if
a) there is a firewall issue
b) you are trying to consume messages from another node and have not set
your advertised host name to the public ip address in the kafka server
settings.
On Tue, Feb 24, 2015 at 12:07 PM, alexey yakubovich <
alexe...@yahoo.com.invalid> wrote:
> Hi, I am learning kafka, ... hard way
> I try to run the example given here:
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
> Whatever I do i I have
> java.nio.channels.ClosedChannelException when try to send message
> Exception in thread "main" java.nio.channels.ClosedChannelExceptionat
> kafka.network.BlockingChannel.send(BlockingChannel.scala:100)at
> kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:78)at
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
> And, sometimes it's
> 2015-02-24 13:46:51 INFO SimpleConsumer:68 - Reconnect due to socket
> error: java.io.EOFException: Received -1 when reading from channel,
> socket has likely been closed.
> (in all cases it fails in findLeader())
>
>
> At the same some other consumers work, as e.g. the command line
> console-consumer
> Now, I put a very small code snippet (below), that seems should do the
> job, but it's not, something missing. Please help.
> public class Test1 {public static void main(String[] args) {String
> host = "localhost"; int port = 2181; String topic = "test"; int
> soTimeout = -1; int bufferSize = 10; String clientId = "me-aly";
> SimpleConsumer consumer = new SimpleConsumer(host, port, soTimeout,
> bufferSize, clientId);List topics =
> Collections.singletonList(topic);
> TopicMetadataRequest req = new TopicMetadataRequest(topics);
> kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);
> System.out.println(resp); }}
>
>
Re: KafkaProducer.send contract
> Could you explain a bit more what you want to achieve through batching? > Better throughput or atomicity? Sure! I've assumed that there's per-message atomicity and a per-partition ordering guarantee with KafkaProducer.send(), but nothing beyond that. My hopes are to reduce latency from when my subsystem is handed a message to when I receive my ack back from Kafka (acks=-1). My subsystem will receive multiple messages for a Node (msg1,msg2,msg3,msg4), but I need to acquire a semaphore for that Node before sending msg1, and the callback in the send() will need to release it before I can send msg2. As Kafka can have messages much bigger than my subsystem will get, I can do application-layer batching and build msgA which aggregates msg2,msg3,msg4 and send(msgA) to do a "bulk" send. This is the only way I can ensure that msg1 is safely written before msg2 is sent. To me, the semaphore is unfortunate since it's artificially slowing message rate down just in case the send() fails. I appreciate it'd be awesome if Kafka could track this; but if it's failing to write my message, it's likely having issues that would impede its ability to track the failure state too. JAmes On Mon, Feb 23, 2015 at 4:36 PM, Jun Rao wrote: > Could you explain a bit more what you want to achieve through batching? > Better throughput or atomicity? > > Thanks, > > Jun > > On Thu, Feb 19, 2015 at 4:09 PM, JAmes Atwill > wrote: > > > Hey Jun, > > > > That's what I've got right now, semaphore before send() and release in > the > > callback. Am I correct in understanding that there's no way to do any > > batching with KafkaProducer itself (other than have a "bulk" message > which > > would just be a single message with multiple messages for a particular > > Node)? > > > > JAmes > > > > On Thu, Feb 19, 2015 at 2:50 PM, Jun Rao wrote: > > > > > You can register a callback for each message sent. The callback will be > > > called when the message is sent successfully or failed. > > > > > > Thanks, > > > > > > Jun > > > > > > On Tue, Feb 17, 2015 at 4:11 PM, JAmes Atwill > > > wrote: > > > > > > > Hi! > > > > > > > > I'm using the new KafkaProducer in 0.8.2.0. > > > > > > > > I have thousands of "Nodes" which receive messages. Each message > > > > idempotently mutates the state of the Node, so while duplicate > messages > > > are > > > > fine, missed messages are not. > > > > > > > > I'm writing these messages into a topic with dozens of partitions. > > > > > > > > Am I correct in believing that I'll have to manually manage having > one > > > > message "in flight" per "node" at a time? Or is there a mechanism to > > say > > > > "This message and all messages after it for this partition were > > > rejected"? > > > > (or something similar) > > > > > > > > Thanks! > > > > > > > > JAmes > > > > > > > > > >
Re: After Leadership Election and "kafka.log" JMXBean Registration Process
HI Jun, Thanks for info. Thanks, Bhavesh On Tue, Feb 24, 2015 at 2:45 PM, Jun Rao wrote: > These two metrics are always registered, whether the replica is the leader > or the follower. > > Thanks, > > Jun > > On Mon, Feb 23, 2015 at 6:40 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > Hi Kafka Team or User Community , > > > > After leadership election or switch between follower/leader of partition > > for given topic, does following metrics JMX bean gets registered (on > > leader) and de-registered (on follower). > > > > LogEndOffset > > Size > > LogStartOffset > > > > eg: > > "kafka.log":type="Log",name="TOPIC-17-*Size*" > > "kafka.log":type="Log",name="TOPIC-17-*LogStartOffset*" > > "kafka.log":type="Log",name="TOPIC-17-*LogEndOffset*" > > > > > > Thanks in advance for your help !! > > > > Thanks, > > > > Bhaevsh > > >
Re: NetworkProcessorAvgIdlePercent
Similar pattern for that too. Mostly hovering below. -Zakee On Tue, Feb 24, 2015 at 2:43 PM, Jun Rao wrote: > What about RequestHandlerAvgIdlePercent? > > Thanks, > > Jun > > On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote: > > > Hi Jun, > > > > With ~100G of data being pushed per hour across 35 topics > > (replication-factor 3), the NetworkProcessorAvgIdlePercent is mostly > > showing below 0.5 sometimes when the producers send on a high rate. > > > > Thanks > > -Zakee > > > > On Sun, Feb 22, 2015 at 10:29 PM, Jun Rao wrote: > > > > > What kind of load do you have on the brokers? On an idle cluster (just > > > fetch requests from the follower replicas), I > > > saw NetworkProcessorAvgIdlePercent at about 97%. > > > > > > Thanks, > > > > > > Jun > > > > > > On Thu, Feb 19, 2015 at 5:19 PM, Zakee wrote: > > > > > > > Jun, > > > > > > > > I am already using the latest release 0.8.2.1. > > > > > > > > -Zakee > > > > > > > > On Thu, Feb 19, 2015 at 2:46 PM, Jun Rao wrote: > > > > > > > > > Could you try the 0.8.2.1 release being voted on now? It fixes a > CPU > > > > issue > > > > > and should reduce the CPU load in network thread. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Thu, Feb 19, 2015 at 11:54 AM, Zakee > wrote: > > > > > > > > > > > Kafka documentation recommends <0.3 for above metric. I assume > > > > processor > > > > > is > > > > > > busier if this goes below 0.3 and obviously it being < 0.3 for > long > > > > does > > > > > > not seem to be a good sign. > > > > > > > > > > > > What should be our criteria to raise an alert, I though it should > > be > > > > > when > > > > > > its value goes below 0.3. However, the value seems to be below > 0.3 > > a > > > > lot > > > > > of > > > > > > the times, almost always if we take samples every five mins. What > > > > should > > > > > be > > > > > > the threshold to raise an alarm ? > > > > > > > > > > > > What would be the impact of having this below 0.3 or even zero > like > > > > most > > > > > of > > > > > > the times? > > > > > > > > > > > > > > > > > > -Zakee > > > > > > > > > > > > How Old Men Tighten Skin > > > > > > 63 Year Old Man Shares DIY Skin Tightening Method You Can Do From > > > Home > > > > > > > > > > > > http://thirdpartyoffers.netzero.net/TGL3231/54e63f5bda4c23f5b6560st02vuc > > > > > > > > > > 8% Annuity Return Secret > > > > > Earn Guaranteed Income for Life! Compare Rates Today. > > > > > > > > > http://thirdpartyoffers.netzero.net/TGL3255/54e6782bcbe78782b37bdmp15duc > > > > > > > > > > High School Yearbooks > > > View Class Yearbooks Online Free. Reminisce & Buy a Reprint Today! > > > > http://thirdpartyoffers.netzero.net/TGL3255/54eac962abad049627e56mp03duc > > > > Extended Stay America > Official Site. Free WIFI, Kitchens. Our best rates here, guaranteed. > http://thirdpartyoffers.netzero.net/TGL3255/54ed04fb9538c4fb6188mp15duc
Re: "at least once" consumer recommendations for a load of 5 K messages/second
* ZK was not built for 5K/s writes type of load * Kafka 0.8.2.0 allows you to commit messages to Kafka rather than ZK. I believe this is recommended. * You can also commit batches of messages (i.e. commit every 100 messages). This will reduce the writes and give you at least once while controlling number of duplicates in case of failure. * Yes, can be done in high level consumer. I give few tips here: http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ Gwen On Tue, Feb 24, 2015 at 1:57 PM, Anand Somani wrote: > Hi, > > It is a little long, since I wanted to explain the use case and then ask > questions, so thanks for your attention > > Use case: > > We have a use case where everything in the queue has to be consumed at > least once. So the consumer has to have "consumed" (saved in some > destination database) the message before confirming consumption to kafka > (or ZK). Now it is possible and from what I have read so far we will have > consumer groups and partitions. Here are some facts/numbers for our case > > * We will potentially have messages with peaks of 5k /second. > * We can play with the message size if that makes any difference (keep it < > 100 bytes for a link or put the entire message avg size of 2-5K bytes). > * We do not need replication, but might have a kafka cluster to handle the > load. > * Also work consumption will take anywhere from 300-500ms, generally we > would like the consumer to be not behind by more than 1-2 minutes. So if > the message shows up in a queue, it should show up in the database within 2 > minutes. > > The questions I have are > * If this has been covered before, please point me to it. Thanks > * Is that possible/recommended "controlled commit per consumed message" > for this load (have read about some concerns on ZK issues)? > * Are there any recommendations on configurations in terms of partitions > to number of messages OR consumers? Maybe more queues/topics > * Anything else that we might need to watch out for? > * As for the client, I should be able to do this (control when the offset > commit happens) with high level consumer I suppose? > > > Thanks > Anand >
Re: Stream naming conventions?
Nice :) I like the idea of tying topic name to avro schemas. I have experience with other people's data, and until now I mostly recommended: ... So we end up with things like: etl.onlineshop.searches.validated Or if I have my own test dataset that I don't want to share: users.gshapira.newapp.testing1 Makes it relatively easy to share datasets across the organization, and also makes white-listing and black-listing relatively simple because of the hierarchy (until we add a real topic hierarchy to kafka...). Gwen On Tue, Feb 24, 2015 at 1:13 PM, Thunder Stumpges wrote: > We have a global namespace hierarchy for topics that is exactly our Avro > namespace with Class Name. The template is basically: > > .Core. > .. > > The up side of this for us is that since the topics are named based on the > Avro schema namespace and type, we can look up the avro schema in the Avro > Schema Repository using the topic name, and the schema ID coded into the > message. Each product then also has the flexibility of defining whatever > topics they find useful. > > Hope this helps, > Thunder > > -Original Message- > From: Taylor Gautier [mailto:tgaut...@yahoo.com.INVALID] > Sent: Tuesday, February 24, 2015 12:11 PM > To: kafka-us...@incubator.apache.org > Subject: Stream naming conventions? > > Hello all, > Just wondering if those with a good amount of experience using Kafka in > production with many streams have converged on any sort of naming > convention. If so would you be willing to share? > Thanks in advance, > Taylor >
Re: what groupID does camus use?
Camus uses the simple consumer, which doesn't have the concept of "consumer group" in the API (i.e. Camus is responsible for allocating threads to partitions on its own). The client-id is hard coded and is "hadoop-etl" in some places (when it initializes the offsets) and "camus" in other places. The name "camus" is defined in camus.properties in "kafka.client.name", so you can override it. Gwen On Tue, Feb 24, 2015 at 12:26 PM, Yang wrote: > I looked at the example camus.properties, it does not specify this. how > could we set the groupId ? > > thanks > Yang >
How to measure performance metrics
Hi, Please let me know how to measure the performance metrics while pushing/consuming the message to/from the topic. Thanks.
Re: How to measure performance metrics
Hi, There are bunch of metrics available for performance monitoring. These metrics are can be monitored by JMX monitoring tool (Jconsole). https://kafka.apache.org/documentation.html#monitoring. Some of the available metrics reporters are: https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters On Wed, Feb 25, 2015 at 11:10 AM, Bhuvana Baskar wrote: > Hi, > > Please let me know how to measure the performance metrics while > pushing/consuming the message to/from the topic. > > Thanks. >
Re: Kafka High Level Consumer
Thanks Jun. It seems it was an issue with jruby client I was using. Now, they fixed it. -Pranay On Mon, Feb 23, 2015 at 4:57 PM, Jun Rao wrote: > Did you enable auto offset commit? > > Thanks, > > Jun > > On Tue, Feb 17, 2015 at 4:22 PM, Pranay Agarwal > wrote: > > > Hi, > > > > I am trying to read kafka consumer using high level kafka Consumer API. I > > had to restart the consumers for some reason but I kept the same group > id. > > It seems the consumers have started consuming from the beginning (0 > offset) > > instead from the point they had already consumed. > > > > What am I doing wrong here? How do I make sure the consumer start only > > from the point they had left before? > > > > Thanks > > -Pranay > > >
broker restart problems
Hi all:We have a cluster of 3 brokers(id : 0,1,2). We restart(simply use stop.sh and start.sh in bin directory) broker 1. The broker started successfully. However, all the partitions' leader moved to other brokers and no data were written into broker 2. This is the status of one topic:Topic:wx_rtdc_flumesinksPartitionCount:12 ReplicationFactor:3 Configs:Topic: wx_rtdc_flumesinks Partition: 0Leader: 2 Replicas: 1,2,0 Isr: 2,0Topic: wx_rtdc_flumesinks Partition: 1Leader: 2 Replicas: 2,0,1 Isr: 2,0Topic: wx_rtdc_flumesinks Partition: 2Leader: 0 Replicas: 0,1,2 Isr: 0,2Topic: wx_rtdc_flumesinks Partition: 3 Leader: 0 Replicas: 1,0,2 Isr: 0,2Topic: wx_rtdc_flumesinks Partition: 4Leader: 2 Replicas: 2,1,0 Isr: 2,0Topic: wx_rtdc_flumesinks Partition: 5Leader: 0 Replicas: 0,2,1 Isr: 0,2Topic: wx_rtdc_flumesinks Partition: 6Leader: 2 Replicas: 1,2,0 Isr: 2,0Topic: wx_rtdc_flumesinks Partition: 7 Leader: 2 Replicas: 2,0,1 Isr: 2,0Topic: wx_rtdc_flumesinks Partition: 8Leader: 0 Replicas: 0,1,2 Isr: 0,2Topic: wx_rtdc_flumesinks Partition: 9Leader: 0 Replicas: 1,0,2 Isr: 0,2Topic: wx_rtdc_flumesinks Partition: 10 Leader: 2 Replicas: 2,1,0 Isr: 2,0Topic: wx_rtdc_flumesinks Partition: 11 Leader: 0 Replicas: 0,2,1 Isr: 0,2 It seems the broker is out of synchronize with other brokers. and nothing changed after i run preferred replication leader election tool. i think it is because the preferred replication is not in Isr, which is described in the wiki of replication tool. I want to know how to synchronize the replications of 3 brokers so that the broker 1 could work properly.any help would be appreciated.
