java api code and javadoc
Hi I downloaded kafka 0.8.1.1 src and went through some documentation and wikis, but could not find any documentation (javadoc or other) on the java API - info on classes like SimpleConsumer, MessageAndOffset etc. Nor could I locate the source code (.java). I see only scala files. Can anyone provide info on where I can find doc to get list of attributes, methods, signatures etc? Thanks. -Mayank.
Re: java api code and javadoc
Thanks Joseph. I built the javadoc but its incomplete. Where can I find the code itself for classes like KafkaStream, MessageAndOffset, CosumerConnector etc? On Wed, Oct 15, 2014 at 11:10 AM, Joseph Lawson wrote: > You probably have to build your own right now. Check out > https://github.com/apache/kafka#building-javadocs-and-scaladocs > > From: 4mayank <4may...@gmail.com> > Sent: Wednesday, October 15, 2014 11:38 AM > To: users@kafka.apache.org > Subject: java api code and javadoc > > Hi > > I downloaded kafka 0.8.1.1 src and went through some documentation and > wikis, but could not find any documentation (javadoc or other) on the java > API - info on classes like SimpleConsumer, MessageAndOffset etc. Nor could > I locate the source code (.java). I see only scala files. > > Can anyone provide info on where I can find doc to get list of attributes, > methods, signatures etc? > > Thanks. > -Mayank. >
Re: Consumer offsets in offsets topic 0.8.2
I did a similar change - moved from High Level Consumer to Simple Consumer. Howerver kafka-consumer-offset-checker.sh throws an exception. Its searching the zk path /consumers// which does not exist on any of my zk nodes. Is there any other tool for getting the offset lag when using Simple Consumer? Or am I using kafka-consumer-offset-checker.sh incorrectly for Simple Consumer? Output: kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181, 192.168.1.202:2181,192.168.1.203:2181 --group my-control-group Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/my-control-group/owners. kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181, 192.168.1.202:2181,192.168.1.203:2181 --group my-control-group --topic my-control Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/my-control-group/offsets/my-control/1. ZK cli output: ./zookeeper-shell.sh 192.168.1.201:2181 Connecting to 192.168.1.201:2181 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null ls /config/topics [my-control] ls /consumers [] quit Thanks. Mayank. On Fri, Mar 20, 2015 at 9:54 AM, Jiangjie Qin wrote: > Hi Vamsi, > > The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh still > works. You can use them to check the offsets. > If you need to check the offsets programmatically, you can send > OffsetsFetcheRequest to broker using simple consumer. You may refer the > ConsumerOffsetChecker.scala to see how to find correct broker where > corresponding offset manager resides. > > Jiangjie (Becket) Qin > > On 3/19/15, 11:54 PM, "Achanta Vamsi Subhash" > wrote: > > >Hi, > > > >We are using 0.8.2.1 currently. > > > >- How to get the consumer offsets from the offsets topic? > >- Is there any built-in function which I could use? (like in > >AdminUtils.scala) > >- Is it ok to start a simple consumer and read the offsets from the topic? > > > >We used to read the offsets from zookeeper previously for the > >HighLevelConsumers. But with the new broker this changed as we are using > >Kafka topic for offsets. > > > >-- > >Regards > >Vamsi Subhash > >
Re: Consumer offsets in offsets topic 0.8.2
Thanks guys. Based on the responses it looks like I need to have a tool of my own to read offsets. On Mon, Apr 13, 2015 at 3:07 PM, Jiangjie Qin wrote: > Yeah, the current ConsumerOffsetChecker has this issue (maybe bug also) if > the offset storage is Kafka and no offset has been committed. It will > throw ZK exception, which is very confusing. KAFKA-1951 was opened for > this but was not checked in. > > Thanks. > > Jiangjie (Becket) Qin > > On 4/13/15, 9:55 AM, "4mayank" <4may...@gmail.com> wrote: > > >I did a similar change - moved from High Level Consumer to Simple > >Consumer. > >Howerver kafka-consumer-offset-checker.sh throws an exception. Its > >searching the zk path /consumers// which does not exist on any of > >my > >zk nodes. > > > >Is there any other tool for getting the offset lag when using Simple > >Consumer? Or am I using kafka-consumer-offset-checker.sh incorrectly for > >Simple Consumer? > > > >Output: > >kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181, > >192.168.1.202:2181,192.168.1.203:2181 --group my-control-group > >Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException: > >KeeperErrorCode = NoNode for /consumers/my-control-group/owners. > > > > > >kafka-consumer-offset-checker.sh --zookeeper 192.168.1.201:2181, > >192.168.1.202:2181,192.168.1.203:2181 --group my-control-group --topic > >my-control > >Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException: > >KeeperErrorCode = NoNode for > >/consumers/my-control-group/offsets/my-control/1. > > > > > >ZK cli output: > > > >./zookeeper-shell.sh 192.168.1.201:2181 > >Connecting to 192.168.1.201:2181 > >Welcome to ZooKeeper! > >JLine support is disabled > > > >WATCHER:: > > > >WatchedEvent state:SyncConnected type:None path:null > >ls /config/topics > >[my-control] > >ls /consumers > >[] > >quit > > > > > >Thanks. > >Mayank. > > > >On Fri, Mar 20, 2015 at 9:54 AM, Jiangjie Qin > >wrote: > > > >> Hi Vamsi, > >> > >> The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh > >>still > >> works. You can use them to check the offsets. > >> If you need to check the offsets programmatically, you can send > >> OffsetsFetcheRequest to broker using simple consumer. You may refer the > >> ConsumerOffsetChecker.scala to see how to find correct broker where > >> corresponding offset manager resides. > >> > >> Jiangjie (Becket) Qin > >> > >> On 3/19/15, 11:54 PM, "Achanta Vamsi Subhash" > >> > >> wrote: > >> > >> >Hi, > >> > > >> >We are using 0.8.2.1 currently. > >> > > >> >- How to get the consumer offsets from the offsets topic? > >> >- Is there any built-in function which I could use? (like in > >> >AdminUtils.scala) > >> >- Is it ok to start a simple consumer and read the offsets from the > >>topic? > >> > > >> >We used to read the offsets from zookeeper previously for the > >> >HighLevelConsumers. But with the new broker this changed as we are > >>using > >> >Kafka topic for offsets. > >> > > >> >-- > >> >Regards > >> >Vamsi Subhash > >> > >> > >
Using different IPs for clients and kafka inter-node communication
I have a 3-node setup where each kafka/zookeeper node has two network interfaces - one globally routeable (10.x.x.x) and one internal (192.168.x.x). I would like the kafka nodes to use the internal interface to talk to its neighboring brokers (faster and more efficient) and use the globally routeable 10.x address while connecting to producers and consumers. Relevant portions of the config files are below: zookeeper.properties server.1=192.168.1.201:2888:3888 server.2=192.168.1.202:2888:3888 server.3=192.168.1.203:2888:3888 server.properties zookeeper.connect=192.168.1.201:2181,192.168.1.202:2181,192.168.1.203:2181 # Hostname of broker. If this is set, it will only bind to this address. If this is not set, it will bind to all interfaces, and publish one to ZK. host.name=10.a.b.c # Hostname the broker will advertise to producers and consumers. If not set, it uses the # value for "host.name" if configured. Otherwise, it will use the value returned from # java.net.InetAddress.getCanonicalHostName(). advertised.host.name=10.a.b.c This setup is using the 10.x.x.x address to talk to the other kafka brokers. If i were to unset the host.name property how do I know which IP address it will use and which one will it publish to ZK? Is there a different way to achieve this? Thanks. Mayank.
Producer waiting ~15 mins before disconnecting.
I am using kafka 0.8.2.1 old producer. When one of the kafka node in the remote cluster is down the producer is waiting about 15 minutes before it disconnects and tries to connect to another node. (kafka takes < 1 min to change leaders). Producer config used: request.required.acks=1 partitioner.class=MyKey message.send.max.retries=3 request.timeout.ms=12 retry.backoff.ms=2500 I played around with different values of request.timeout.ms, retry.backoff.ms, and topic.metadata.refresh.interval.ms with same results. Is there a way to control the producer's wait so that it returns sooner? Thanks. Mayank.
Re: Producer waiting ~15 mins before disconnecting.
Thanks Magnus. In this case the connections are not idle. There is active traffic between the producer/client and the kafka node when the node goes down. There are socket timeouts arguments for SimpleConsumer. But there are none when creating the producer. If there a configuration/poroperty item to set socket timeouts for producer? On Tue, May 19, 2015 at 2:52 PM, Magnus Edenhill wrote: > Hi Mayank, > > The client should expose a configuration property to enable TCP keepalives > (SO_KEEPALIVE) on its broker sockets, > SO_KEEPALIVE provides speedier detection of connection loss on idle > connections. > (as a positive side effect it also helps keeping connections alive through > NAT/firewalls/LBs) > > If this is not available in the client in question I suggest you file an > issue (JIRA). > > Regards, > Magnus > > > 2015-05-19 21:38 GMT+02:00 4mayank <4may...@gmail.com>: > > > I am using kafka 0.8.2.1 old producer. When one of the kafka node in the > > remote cluster is down the producer is waiting about 15 minutes before it > > disconnects and tries to connect to another node. (kafka takes < 1 min to > > change leaders). > > > > Producer config used: > > request.required.acks=1 > > partitioner.class=MyKey > > message.send.max.retries=3 > > request.timeout.ms=12 > > retry.backoff.ms=2500 > > > > I played around with different values of request.timeout.ms, > > retry.backoff.ms, and topic.metadata.refresh.interval.ms with same > > results. > > > > Is there a way to control the producer's wait so that it returns sooner? > > > > Thanks. > > Mayank. > > >
