1. bootstrap.servers is only used for issuing the very first metadata request. Once the leader of a topic/partition is found, the producer will talk to the leader brokers directly.
2. Do you see any disconnects in the broker log? Thanks, Jun On Mon, Jan 19, 2015 at 11:05 AM, Sa Li <sal...@gmail.com> wrote: > Hello, Jun > > I run such command: > > bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance > test-rep-three 1000000 3000 -1 acks=-1 bootstrap.servers= > 10.100.98.100:9092, > 10.100.98.101:9092 <http://10.100.98.100:9092/>, 10.100.98.102:9092 > <http://10.100.98.100:9092/> buffer.memory=67108864 batch.size=8196 > > This is the perf test to producing data to different hosts. I have such > questions: > 1. If I use bootstrap.servers=10.100.98.100:9092, 10.100.98.101:9092 > <http://10.100.98.100:9092/>, 10.100.98.102:9092 > <http://10.100.98.100:9092/>, does this mean I send data to each of > servers > through 9092? However, if I set bootstrap.servers=10.100.98.100:9092 only, > the producer will send data to 100, while replicate the data through other > ports? > > here is the netstate I saw: > netstat -plantue | egrep -i '.98.101|.98.102' > tcp 0 0 10.100.98.100:37512 10.100.98.102:22 > ESTABLISHED 1004 7516316 5862/ssh > tcp6 0 0 10.100.98.100:9092 10.100.98.102:56819 > ESTABLISHED 0 371522 3852/java > tcp6 0 0 10.100.98.100:3888 10.100.98.101:40052 > ESTABLISHED 0 27514 1793/java > tcp6 0 202 10.100.98.100:53592 10.100.98.101:9092 > ESTABLISHED 0 497715 3852/java > tcp6 0 0 10.100.98.100:3888 10.100.98.102:32837 > ESTABLISHED 0 21701 1793/java > tcp6 0 0 10.100.98.100:9092 10.100.98.101:51053 > ESTABLISHED 0 371526 3852/java > tcp6 0 270 10.100.98.100:53591 10.100.98.101:9092 > ESTABLISHED 0 497713 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.102:57554 > ESTABLISHED 0 7491796 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.101:51055 > ESTABLISHED 0 601277 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.102:56824 > ESTABLISHED 0 614795 3852/java > tcp6 0 0 10.100.98.100:48226 10.100.98.102:9092 > ESTABLISHED 0 3659949 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.101:51054 > ESTABLISHED 0 601275 3852/java > tcp6 0 0 10.100.98.100:48225 10.100.98.102:9092 > ESTABLISHED 0 3803556 3852/java > tcp6 0 0 10.100.98.100:53593 10.100.98.101:9092 > ESTABLISHED 0 638462 3852/java > tcp6 0 236 10.100.98.100:48228 10.100.98.102:9092 > ESTABLISHED 0 3936260 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.102:56827 > ESTABLISHED 0 601276 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.101:51052 > ESTABLISHED 0 614796 3852/java > tcp6 0 230 10.100.98.100:53594 10.100.98.101:9092 > ESTABLISHED 0 637547 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.102:56826 > ESTABLISHED 0 601274 3852/java > tcp6 0 0 10.100.98.100:2181 10.100.98.102:34162 > ESTABLISHED 0 7491795 1793/java > tcp6 0 230 10.100.98.100:48227 10.100.98.102:9092 > ESTABLISHED 0 3121735 3852/java > tcp6 0 0 10.100.98.100:9092 10.100.98.102:56825 > ESTABLISHED 0 497716 3852/java > > 2. I still see the error of "server disconnect" during the perf test, but > back to normal and print the results like > > 11212 records sent, 2239.7 records/sec (6.41 MB/sec), 7287.1 ms avg > latency, 15005.0 max latency. > 11620 records sent, 2313.8 records/sec (6.62 MB/sec), 7248.9 ms avg > latency, 14807.0 max latency. > 11522 records sent, 2293.4 records/sec (6.56 MB/sec), 7110.9 ms avg > latency, 14551.0 max latency. > 11058 records sent, 2200.2 records/sec (6.29 MB/sec), 7176.8 ms avg > latency, 14774.0 max latency. > > But by monitoring the connection between nodes, we didn't see any > disconnections, but we are not sure if the kafka servers disconnection took > place or not, how to check it up? > > thanks > > AL > > > On Sun, Jan 18, 2015 at 10:21 AM, Jun Rao <j...@confluent.io> wrote: > > > Any issue with the network? > > > > Thanks, > > > > Jun > > > > On Wed, Jan 7, 2015 at 1:59 PM, Sa Li <sal...@gmail.com> wrote: > > > > > Things bother me, sometimes, the errors won't pop out, sometimes it > > comes, > > > why? > > > > > > On Wed, Jan 7, 2015 at 1:49 PM, Sa Li <sal...@gmail.com> wrote: > > > > > > > > > > > Hi, Experts > > > > > > > > Our cluster is a 3 nodes cluster, I simply test producer locally, see > > > > > > > > bin/kafka-run-class.sh > > org.apache.kafka.clients.tools.ProducerPerformance > > > > test-rep-three 1000000 3000 -1 acks=1 bootstrap.servers= > > > 10.100.98.100:9092 > > > > buffer.memory=67108864 batch.size=8196 > > > > > > > > But I got such error, I do think this is critical issue, it just > > > > temporally lose the connection and get back, what is the reason for > > this? > > > > > > > > [2015-01-07 21:44:14,180] WARN Error in I/O with > > voluminous-mass.master/ > > > > 10.100.98.101 (org.apache.kafka.common.network.Selector) > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:232) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115) > > > > at java.lang.Thread.run(Thread.java:745) > > > > [2015-01-07 21:44:14,190] WARN Error in I/O with > > voluminous-mass.master/ > > > > 10.100.98.101 (org.apache.kafka.common.network.Selector) > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:232) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115) > > > > at java.lang.Thread.run(Thread.java:745) > > > > [2015-01-07 21:44:14,200] WARN Error in I/O with > > voluminous-mass.master/ > > > > 10.100.98.101 (org.apache.kafka.common.network.Selector) > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:232) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115) > > > > at java.lang.Thread.run(Thread.java:745) > > > > [2015-01-07 21:44:14,210] WARN Error in I/O with > > voluminous-mass.master/ > > > > 10.100.98.101 (org.apache.kafka.common.network.Selector) > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:232) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115) > > > > at java.lang.Thread.run(Thread.java:745) > > > > [2015-01-07 21:44:14,220] WARN Error in I/O with > > voluminous-mass.master/ > > > > 10.100.98.101 (org.apache.kafka.common.network.Selector) > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:232) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115) > > > > at java.lang.Thread.run(Thread.java:745) > > > > [2015-01-07 21:44:14,230] WARN Error in I/O with > > voluminous-mass.master/ > > > > 10.100.98.101 (org.apache.kafka.common.network.Selector) > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:232) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115) > > > > at java.lang.Thread.run(Thread.java:745) > > > > [2015-01-07 21:44:14,240] WARN Error in I/O with > > voluminous-mass.master/ > > > > 10.100.98.101 (org.apache.kafka.common.network.Selector) > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:232) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184) > > > > at > > > > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 80 records sent, 13.6 records/sec (0.04 MB/sec), 31113.8 ms avg > > latency, > > > > 46912.0 max latency. > > > > 116695 records sent, 23339.0 records/sec (66.77 MB/sec), 5157.1 ms > avg > > > > latency, 46946.0 max latency. > > > > > > > > > > > > thanks > > > > > > > > -- > > > > > > > > Alec Li > > > > > > > > > > > > > > > > -- > > > > > > Alec Li > > > > > > > > > -- > > Alec Li >
