Hello, I tried using the sources for unit tests, but have been unsuccessful in getting it to work with the low level API (kafka version 0.8.2.2). Our code is modelled on the Simple Consumer API code, and the error always occurs in the findLeader code.
SEVERE: Error communicating with broker:localhost:53412 attempting to find a leader for channel:catapult.schedule.day, partition: 0.Reason: java.nio.channels.ClosedChannelException Looking at the logs the embedded server(s) generate, I see these types of messages which seemed interesting for this particular issue (first one is that first org.apache.zookeeper.ClientCnxn always seems to fail on the currently assigned port, and later on a bunch of KeeperErrorCode = NoNode for /brokers etc): [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:53411 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@14be60b [main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:53411. Will not attempt to authenticate using SASL (unknown error) [main-SendThread(localhost:53411)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.ja va:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:53411. Will not attempt to authenticate using SASL (unknown error) [main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:53411, initiating session [NIOServerCxn.Factory:localhost/127.0.0.1:53411] INFO org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:53414 [NIOServerCxn.Factory:localhost/127.0.0.1:53411] INFO org.apache.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:53414 [SyncThread:0] INFO org.apache.zookeeper.server.persistence.FileTxnLog - Creating new log file: log.1 [SyncThread:0] INFO org.apache.zookeeper.server.ZooKeeperServer - Established session 0x15006291f440000 with negotiated timeout 6000 for client /127.0.0.1:53414 [main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:53411, sessionid = 0x15006291f440000, negotiated timeout = 6000 [ProcessThread(sid:0 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x15006291f440000 type:create cxid:0x4 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers [ProcessThread(sid:0 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x15006291f440000 type:create cxid:0xa zxid:0x7 txntype:-1 reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config [ProcessThread(sid:0 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x15006291f440000 type:create cxid:0x10 zxid:0xb txntype:-1 reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin [ProcessThread(sid:0 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x15006291f440000 type:setData cxid:0x1a zxid:0xf txntype:-1 reqpath:n/a Error Path:/controller_epoch Error:KeeperErrorCode = NoNode for /controller_epoch [ProcessThread(sid:0 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x15006291f440000 type:delete cxid:0x27 zxid:0x11 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election [ProcessThread(sid:0 cport:-1):] INFO org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x15006291f440000 Thanks Rakesh On 07/07/2015 21:16, "Gwen Shapira" <gshap...@cloudera.com> wrote: >Is this for unit tests? Or do you need an embedded Kafka / ZK inside an >app? >Or do you mean launching an external Kafka cluster, but without command >line? > >For unit tests / embedded, you can see this example: >https://gist.github.com/vmarcinko/e4e58910bcb77dac16e9 > >To start an actual Kafka server through an API call, you typically >need some kind of agent to take the API call and launch Kafka. >Cloudera Manager has an API that can do this, and I'm sure Ambari can >do the same. > >Gwen > >On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j.gon...@gmail.com> wrote: >> Hi all, >> >> Is it possible to launch Kafka/Zookeeper servers via some part of the >>API? >> I noticed in this: >> >>http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in- >>kafka-from-the-ide-using-api/ >> that it seems to be possible to create a ZK instance and add a topic to >>it, >> but is this the same as launching the ZK server as we would in the >>command >> line? >> >> Similarly, would it be possible to do this for a Kafka server and have >>it >> connect to an existing ZK connection? >> >> Thanks!