Hello Tony,
Could you share your Streams config values so that people can help further
investigating your issue?
Guozhang
On Mon, Feb 5, 2018 at 12:00 AM, Tony John <tonyjohnant...@gmail.com> wrote:
> Hi All,
>
> I have been running a streams application for sometime. The application
> runs fine for sometime but after a day or two I see the below log getting
> printed continuously on to the console.
>
> WARN 2018-02-05 02:50:04.060 [kafka-producer-network-thread | producer-1]
> org.apache.kafka.clients.NetworkClient - Connection to node -1 could not
> be
> established. Broker may not be available.
>
> WARN 2018-02-05 02:50:04.160 [kafka-producer-network-thread | producer-1]
> org.apache.kafka.clients.NetworkClient - Connection to node -1 could not
> be
> established. Broker may not be available.
>
> WARN 2018-02-05 02:50:04.261 [kafka-producer-network-thread | producer-1]
> org.apache.kafka.clients.NetworkClient - Connection to node -1 could not
> be
> established. Broker may not be available.
>
> WARN 2018-02-05 02:50:04.311 [kafka-producer-network-thread | producer-1]
> org.apache.kafka.clients.NetworkClient - Connection to node -1 could not
> be
> established. Broker may not be available.
>
> WARN 2018-02-05 02:50:04.361 [kafka-producer-network-thread | producer-1]
> org.apache.kafka.clients.NetworkClient - Connection to node -1 could not
> be
> established. Broker may not be available.
>
> WARN 2018-02-05 02:50:04.411 [kafka-producer-network-thread | producer-1]
> org.apache.kafka.clients.NetworkClient - Connection to node -1 could not
> be
> established. Broker may not be available.
>
> At this time, though the application is able to process the messages, I
> could also see lag building up in the consumers and the processing time for
> a batch has increased 15 folds.
>
> I am using a single zoo-keeper instance with 2 brokers and 4 application
> instances. I checked the broker and zoo-keeper status, they are all running
> fine as I could see. I have also verified the connectivity between the
> application and broker instances using telnet and it seems intact. The
> kafka broker and streams/client versions are 0.11.0.2. Results of broker
> status results from zoo-keeper below
>
>
> [root@app100 kafka]# echo dump | nc localhost 2181
>
> SessionTracker dump:
>
> Session Sets (3):
>
> 0 expire at Mon Feb 05 06:16:39 UTC 2018:
>
> 1 expire at Mon Feb 05 06:16:42 UTC 2018:
>
> 0x161562860970001
>
> 1 expire at Mon Feb 05 06:16:45 UTC 2018:
>
> 0x161562860970000
>
> ephemeral nodes dump:
>
> Sessions with Ephemerals (2):
>
> 0x161562860970000:
>
> /brokers/ids/0
>
> /controller
>
> 0x161562860970001:
>
> /brokers/ids/1
>
> [root@app100 kafka]# ./kafka_2.11-0.11.0.2/bin/zookeeper-shell.sh
> localhost:2181 <<< "get /brokers/ids/0"
>
> Connecting to localhost:2181
>
> Welcome to ZooKeeper!
>
> JLine support is disabled
>
>
> WATCHER::
>
>
> WatchedEvent state:SyncConnected type:None path:null
>
> {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"}
> ,"endpoints":["PLAINTEXT://172.31.10.35:9092"],"jmx_port":
> 55555,"host":"172.31.10.35","timestamp":"1517569007467","
> port":9092,"version":4}
>
> cZxid = 0x1c
>
> ctime = Fri Feb 02 10:56:47 UTC 2018
>
> mZxid = 0x1c
>
> mtime = Fri Feb 02 10:56:47 UTC 2018
>
> pZxid = 0x1c
>
> cversion = 0
>
> dataVersion = 0
>
> aclVersion = 0
>
> ephemeralOwner = 0x161562860970000
>
> dataLength = 197
>
> numChildren = 0
>
> [root@app100 kafka]# ./kafka_2.11-0.11.0.2/bin/zookeeper-shell.sh
> localhost:2181 <<< "get /brokers/ids/1"
>
> Connecting to localhost:2181
>
> Welcome to ZooKeeper!
>
> JLine support is disabled
>
>
> WATCHER::
>
>
> WatchedEvent state:SyncConnected type:None path:null
>
> {"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"}
> ,"endpoints":["PLAINTEXT://172.31.14.8:9092"],"jmx_port":
> 55555,"host":"172.31.14.8","timestamp":"1517569016562","
> port":9092,"version":4}
>
> cZxid = 0x21
>
> ctime = Fri Feb 02 10:56:56 UTC 2018
>
> mZxid = 0x21
>
> mtime = Fri Feb 02 10:56:56 UTC 2018
>
> pZxid = 0x21
>
> cversion = 0
>
> dataVersion = 0
>
> aclVersion = 0
>
> ephemeralOwner = 0x161562860970001
>
> dataLength = 195
>
> numChildren = 0
>
> Could you please throw some light on this as to what could be going wrong
> here?
>
> Thanks,
> Tony
>
--
-- Guozhang
