Editing Kafka wiki pages (Confluence)?

Hi all, I have been trying to modify one of the Kafka wiki pages [1] to correct a few outdated code examples but it turns out that my Confluence account (miguno [2]) apparently does not have edit permissions. The "Page Restrictions" for [1] are listed as: - No view restrictions are defined f

Re: Kafka behind AWS ELB

Hi Dillian, Yeah, ELB + ASG will is pretty popular however might need further tricks to use them for Kafka brokers. As I replied to Chandrashekhar in another email. You can use ELB as bootstrap.servers/metadata.broker.list to serve as client bootstrap use case. But all the producing/consuming tr

Re: Kafka behind AWS ELB

Dillian, On Mon, May 4, 2015 at 1:52 PM, Dillian Murphey wrote: > > I'm interested in this topic as well. If you put kafka brokers inside an > autoscaling group, then AWS will automatically add brokers if demand > increases, and the ELB will automatically round-robin across all of your > kafka i

Re: Topic missing Leader and Isr

Thank you, Buntu. On Sat, May 2, 2015 at 1:16 PM, Buntu Dev wrote: > This seems to be an issue thats fixed in 0.8.2 and after upgrading the > kafka cluster to 0.8.2 this issue was resolved. > https://issues.apache.org/jira/browse/KAFKA-1738 > >

Re: Kafka brokers behind AWS ELB

Can you clarify what is meant by using the ELB as a metadata broker? Let's say I have a python script that sends messages into kafka. Do I have to provide a list of brokers or can I use the ELB DNS alias that will round-robin to all my healthy brokers? Much easier to use the ELB instead of hard-c

Re: Kafka behind AWS ELB

I'm interested in this topic as well. If you put kafka brokers inside an autoscaling group, then AWS will automatically add brokers if demand increases, and the ELB will automatically round-robin across all of your kafka instances. So in your config files and code, you only need to provide a sing

Re: Kafka behind AWS ELB

Ok. You can deploy kafka in AWS. You can have brokers on AWS servers. Kafka is not a push system. So you will need someone writing to kafka and consuming from kafka. It will work. My suggestion will be to try it out on a smaller instance in AWS and see the effects. As I do not know the actual use

Re: When in-sync isn't in sync?

FYI, Basically, the ISR returned by topic metadata response is unreliable. This is discussed in KAFKA-1367 Thanks, Mayuresh On Mon, May 4, 2015 at 8:09 AM, Steve Miller wrote: > [ BTW, after some more research, I think what might be happening here is > that we had some de-facto network part

Re: Kafka behind AWS ELB

Hi Chandrashekar, Can you please elaborate the use case for Kafka here, like how you are planning to use it. Thanks, Mayuresh On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar < shekhar.kote...@gmail.com> wrote: > Hi, > > I am new to Apache Kafka. I have played with it on my laptop. > >

Re: Kafka behind AWS ELB

I am sorry but I cannot reveal those details due to confidentiality issues. I hope you understand. Regards, Chandrash3khar Kotekar Mobile - +91 8600011455 On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat wrote: > Hi Chandrashekar, > > Can you please elaborate the use case for Kafka here, like h

Re: Unclaimed partitions

Hi Dave, For simpleConsumer can you try using : /** * Fetch a set of messages from a topic. * * @param request specifies the topic name, topic partition, starting byte offset, maximum bytes to be fetched. * @return a set of fetched messages */ def fetch(request: kafka.javaapi.

When in-sync isn't in sync?

[ BTW, after some more research, I think what might be happening here is that we had some de-facto network partitioning happen as a side-effect of us renaming some network interfaces, though if that's the case, I'd like to know how to get everything back into sync. ] Hi. I'm seeing somethin

Re: Unclaimed partitions

Hi, this happens most times I restart the consumer group, but not every time. There are no log errors and nothing seems to be indicating that a rebalance is occurring. Here are the ZK logs I see on one of the processes that isn’t receiving partitions. 2015-05-04 13:55:32,365 [main] INFO org.ap

kafka.message.InvalidMessageException: Message is corrupt crc

Hi Here is my questions. kafka.message.InvalidMessageException: Message is corrupt (stored crc = 301919704, computed crc = 2370805243) at kafka.message.Message.ensureValid(Message.scala:166) at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:102)