Hi Erik, I've not tested it on a producer that is not a part of the kafka cluster.
*Producer configuration on all machines:-* root@x.x.x.x:/a/kafka/config# cat producer.properties | egrep -v '^#|^$' metadata.broker.list=localhost:9092 producer.type=sync compression.codec=none serializer.class=kafka.serializer.DefaultEncoder *Server configuration on all machines:-* root@y.y.y.y:/a/kafka/config# cat server.properties | egrep -v '^#|^$' broker.id=0 port=9092 num.network.threads=6 num.io.threads=8 socket.send.buffer.bytes=10485760 socket.receive.buffer.bytes=10485760 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 log.cleaner.enable=false zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 num.replica.fetchers=4 *Command used from both the machines (slow and fast producer):* kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092,a.a.a.a:9092,b.b.b.b:9092 --messages 1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats --threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms 10000 --compression-codec 2 --reporting-interval 1000 Regards, Prabhjot On Thu, Sep 24, 2015 at 6:36 PM, Helleren, Erik <erik.helle...@cmegroup.com> wrote: > What happens when the new producer that is getting 70 MB/s is started on a > machine that is not part of the kafka cluster? > > Can you include your topic description/configuration, producer > configuration, and broker configuration? > > On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <prabhbha...@gmail.com> wrote: > > >Hi, > > > >I would like to dig deep into this issue. I've changed log4j.properties > >for > >logging in ALL mode in all places. I am getting lost in the logs. > > > >Any pointers would be welcome > > > >Please let me know if you would need any information regarding this > > > >Thanks, > >Prabhjot > > > >On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj <prabhbha...@gmail.com> > >wrote: > > > >> Hello Folks, > >> > >> I've noticed that 2 producer machines, that I had configured, have > >>become > >> very slow over time > >> They are giving 17-19 MB/s > >> > >> But, a producer that I setup today is giving 70MB/s as the write > >>throughput > >> > >> If I see the contents of bin, libs, config directories, nothing is > >> different in the files on any of the producer machines. > >> > >> Producer is running on the kafka machines itself > >> > >> Request your expertise > >> > >> Regards, > >> Prabhjot > >> > >> > >> > > > > > >-- > >--------------------------------------------------------- > >"There are only 10 types of people in the world: Those who understand > >binary, and those who don't" > > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"
