Hello,
I tried using the sources for unit tests, but have been unsuccessful in
getting it to work with the low level API (kafka version 0.8.2.2). Our
code is modelled on the Simple Consumer API code, and the error always
occurs in the findLeader code.
SEVERE: Error communicating with broker:localhost:53412 attempting to find
a leader for channel:catapult.schedule.day, partition: 0.Reason:
java.nio.channels.ClosedChannelException
Looking at the logs the embedded server(s) generate, I see these types of
messages which seemed interesting for this particular issue (first one is
that first org.apache.zookeeper.ClientCnxn always seems to fail on the
currently assigned port, and later on a bunch of KeeperErrorCode = NoNode
for /brokers etc):
[main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=localhost:53411 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@14be60b
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Opening socket connection to server localhost/0:0:0:0:0:0:0:1:53411. Will
not attempt to authenticate using SASL (unknown error)
[main-SendThread(localhost:53411)] WARN org.apache.zookeeper.ClientCnxn -
Session 0x0 for server null, unexpected error, closing socket connection
and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.ja
va:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Opening socket connection to server localhost/127.0.0.1:53411. Will not
attempt to authenticate using SASL (unknown error)
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Socket connection established to localhost/127.0.0.1:53411, initiating
session
[NIOServerCxn.Factory:localhost/127.0.0.1:53411] INFO
org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket
connection from /127.0.0.1:53414
[NIOServerCxn.Factory:localhost/127.0.0.1:53411] INFO
org.apache.zookeeper.server.ZooKeeperServer - Client attempting to
establish new session at /127.0.0.1:53414
[SyncThread:0] INFO org.apache.zookeeper.server.persistence.FileTxnLog -
Creating new log file: log.1
[SyncThread:0] INFO org.apache.zookeeper.server.ZooKeeperServer -
Established session 0x15006291f440000 with negotiated timeout 6000 for
client /127.0.0.1:53414
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Session establishment complete on server localhost/127.0.0.1:53411,
sessionid = 0x15006291f440000, negotiated timeout = 6000
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:create
cxid:0x4 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/brokers
Error:KeeperErrorCode = NoNode for /brokers
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:create
cxid:0xa zxid:0x7 txntype:-1 reqpath:n/a Error Path:/config
Error:KeeperErrorCode = NoNode for /config
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:create
cxid:0x10 zxid:0xb txntype:-1 reqpath:n/a Error Path:/admin
Error:KeeperErrorCode = NoNode for /admin
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:setData
cxid:0x1a zxid:0xf txntype:-1 reqpath:n/a Error Path:/controller_epoch
Error:KeeperErrorCode = NoNode for /controller_epoch
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:delete
cxid:0x27 zxid:0x11 txntype:-1 reqpath:n/a Error
Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for
/admin/preferred_replica_election
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Processed session
termination for sessionid: 0x15006291f440000
Thanks
Rakesh
On 07/07/2015 21:16, "Gwen Shapira" <gshap...@cloudera.com> wrote:
>Is this for unit tests? Or do you need an embedded Kafka / ZK inside an
>app?
>Or do you mean launching an external Kafka cluster, but without command
>line?
>
>For unit tests / embedded, you can see this example:
>https://gist.github.com/vmarcinko/e4e58910bcb77dac16e9
>
>To start an actual Kafka server through an API call, you typically
>need some kind of agent to take the API call and launch Kafka.
>Cloudera Manager has an API that can do this, and I'm sure Ambari can
>do the same.
>
>Gwen
>
>On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j.gon...@gmail.com> wrote:
>> Hi all,
>>
>> Is it possible to launch Kafka/Zookeeper servers via some part of the
>>API?
>> I noticed in this:
>>
>>http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-
>>kafka-from-the-ide-using-api/
>> that it seems to be possible to create a ZK instance and add a topic to
>>it,
>> but is this the same as launching the ZK server as we would in the
>>command
>> line?
>>
>> Similarly, would it be possible to do this for a Kafka server and have
>>it
>> connect to an existing ZK connection?
>>
>> Thanks!
