how to configure KAFKA behind firewall

Hi all, I am encountering problem in configuring ports. Am running ZooKeeper and KAFKA in the same machine. When am starting KAFKA server, it listen to zookeeper with clientPort 2181. But KAFKA creating some other tcp connection with zookeeper. After enabling trace logs in ZooKeeper, [2014-03-1

Re: about offsetRequest time field

My question is, am I understanding the meaning of "Time" right? Or should I try other values for the "Time" field? Thanks for your time! The time field expects an absolute unix timestamp, except for the 2 special values for getting earliest and latest offsets. Thanks Neha On Thursday, March 13,

Re: zookeeper exception

No. This indicates the follower broker 1 is falling behind the leader and hence the leader removes it from the ISR. We need to improve logging. This does not indicate a problem with the broker. On Thursday, March 13, 2014, 陈小军 wrote: > Hi all > i use 0.8.1 release to do test, and get followi

Re: Invalid topic metadata response

By the "reassignment tool", do you mean the tool as described at https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool, which just writes the JSON to ZooKeeper, or the code in KafkaController that is actually responsible for reassignment? Wou

Re: Invalid topic metadata response

Most of the bugs we hit were recoverable in the sense that doing a rolling bounce on the cluster fixed the state of the topic. I updated the documentation to add the warning. Thanks, Neha On Fri, Mar 14, 2014 at 9:17 AM, Ryan Berdeen wrote: > By the "reassignment tool", do you mean the tool as

Hardware planning

Hi all, I'm looking at setting up a (small) Kafka cluster for streaming microscope data to Spark-Streaming. The producer would be a single Windows 7 machine with a 1Gb or 10Gb ethernet connection running http posts from Matlab (this bit is a little fuzzy, and I'm not the user, I'm an admin),

Re: zookeeper exception

What Information in the log indicates that broker 1 is falling behind the leader? What about the error: "ERROR Conditional update of path” in there. Is that OK? I just want to be label to read the log better. I assume below two properties determines the broker behavior of removing the follo

Re: zookeeper exception

Maung, [2014-03-14 12:11:44,310] INFO Partition [nelo2-normal-logs,0] on broker 0: Shrinking ISR for partition [nelo2-normal-logs,0] from 0,1 to 0 (kafka.cluster.Partition) [2014-03-14 12:11:44,313] INFO Partition [nelo2-symbolicated-logs,1] on broker 0: Shrinking ISR for partition [nelo2-symbolic

Re: about offsetRequest time field

Thanks for helping! Cheers, Churu On Mar 14, 2014, at 7:42 AM, Neha Narkhede wrote: > My question is, am I understanding the meaning of "Time" right? Or should I > try other values for the "Time" field? Thanks for your time! > > The time field expects an absolute unix timestamp, except for the

about consumer group name

Hi, “Consumers label themselves with a consumer group name”, and consumer group name should be global across each Kafka cluster. However, when I check the API, there is no “consumer group name” included in any kind of request(metadata, produce, fetch, offset). Does the broker know about the con

Re: about consumer group name

Hi Churu, Brokers are actually not aware of the consumer groups, the consumer groups is maintained within the consumers (and registered in ZK) to achieve load balance. After the group has decided who-consume-which-partition, then each consumer will do their fetching independently and send fetching

Re: Hardware planning

The spec looks reasonable. If you have other machines, it may be better to put ZK on its own machines. Thanks, Jun On Fri, Mar 14, 2014 at 10:52 AM, Carlile, Ken wrote: > Hi all, > > I'm looking at setting up a (small) Kafka cluster for streaming microscope > data to Spark-Streaming. > > The p