> bootstrap.servers = <BROKER-1-IP:9091> , <BROKER-2-IP:9091>
Is your bootstrap.servers configuration is correct ? You have specified port `9091`, but running the GetOffsetShell command on `9094` On Wed, Apr 19, 2017 at 11:58 AM, Ranjith Anbazhakan < ranjith.anbazha...@aspiresys.com> wrote: > Unfortunately, there is no specific information (error/exception) in the > logs when the buffer to queue records data goes missing i.e) When stopped > broker (say broker 2) is started and followed by stopping current running > broker that received all producer sent records in buffer (say broker 1). > > Thanks, > Ranjith > > -----Original Message----- > From: David Garcia [mailto:] > Sent: Wednesday, April 19, 2017 09:31 > To: users@kafka.apache.org > Subject: Re: Kafka Producer - Multiple broker - Data sent to buffer but > not in Queue > > What do broker logs say around the time you send your messages? > > On 4/18/17, 3:21 AM, "Ranjith Anbazhakan" <ranjith.anbazhakan@aspiresys. > com> wrote: > > Hi, > > I have been testing behavior of multiple broker instances of kafka in > same machine and facing inconsistent behavior of producer sent records to > buffer not being available in queue always. > > Tried kafka versions: > 0.10.2.0 > 0.10.1.0 > > Scenario: > > 1. Ran two broker instances in same machine. Say broker 1 as > initial leader, broker 2 as initial follower. > > 2. Stopped broker 1. Now broker 2 became leader. > > 3. Now producer sends records for a given topic TEST through > send() method, followed by flush(). Records have to go to Broker 2 > logically. No error/exception is thrown by code. (So it is assumed data has > been sent successfully to buffer) > > 4. When using command to check the records count for TEST topic > in Broker 2, the sent records are not added to existing records count for > that topic in queue. > > a. Used command - kafka-run-class.bat kafka.tools.GetOffsetShell > --broker-list localhost:9094 --topic TEST --time -1 (where TEST is the used > topic) > > NOTE: **Step 4 is not happening always and is inconsistent**. In the > scenario when it does not work, if Broker 1 is made UP and then made DOWN, > records are always been available in queue in Broker 2 post doing Step 3. > > Configurations: > Overall Producer configurations: (most are default values) > acks = all > batch.size = 16384 > block.on.buffer.full = false > bootstrap.servers = <BROKER-1-IP:9091> , > <BROKER-2-IP:9091> > buffer.memory = 33554432 > client.id = producer-1 > compression.type = none > connections.max.idle.ms = 540000 > interceptor.classes = null > key.serializer = class org.apache.kafka.common. > serialization.StringSerializer > linger.ms = 1 > max.block.ms = 60000 > max.in.flight.requests.per.connection = 5 > max.request.size = 1048576 > metadata.fetch.timeout.ms = 60000 > metadata.max.age.ms = 300000 > metric.reporters = [] > metrics.num.samples = 2 > metrics.sample.window.ms = 30000 > partitioner.class = class org.apache.kafka.clients. > producer.internals.DefaultPartitioner > receive.buffer.bytes = 32768 > reconnect.backoff.ms = 50 > request.timeout.ms = 30000 > retries = 0 > retry.backoff.ms = 100 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > sasl.kerberos.min.time.before.relogin = 60000 > sasl.kerberos.service.name = null > sasl.kerberos.ticket.renew.jitter = 0.05 > sasl.kerberos.ticket.renew.window.factor = 0.8 > sasl.mechanism = GSSAPI > security.protocol = PLAINTEXT > send.buffer.bytes = 131072 > ssl.cipher.suites = null > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > ssl.endpoint.identification.algorithm = null > ssl.key.password = null > ssl.keymanager.algorithm = SunX509 > ssl.keystore.location = null > ssl.keystore.password = null > ssl.keystore.type = JKS > ssl.protocol = TLS > ssl.provider = null > ssl.secure.random.implementation = null > ssl.trustmanager.algorithm = PKIX > ssl.truststore.location = null > ssl.truststore.password = null > ssl.truststore.type = JKS > timeout.ms = 30000 > value.serializer = class org.apache.kafka.common. > serialization.StringSerializer > > Broker 1: (server.properties) > broker.id=1 > port=9091 > advertised.host.name=<System-IP> > auto.create.topics.enable=true > default.replication.factor=2 > leader.imbalance.check.interval.seconds=20 > topic.metadata.refresh.interval.ms=-1 > > Broker 2: (server1.properties) > broker.id=2 > port=9094 > advertised.host.name=<System-IP> > auto.create.topics.enable=true > default.replication.factor=2 > leader.imbalance.check.interval.seconds=20 > topic.metadata.refresh.interval.ms=-1 > > Do let know if anyone have faced similar scenarios and why such an > issue occurs? Let know if any more details are needed. > > Thanks, > Ranjith > [Aspire Systems] > > This e-mail message and any attachments are for the sole use of the > intended recipient(s) and may contain proprietary, confidential, trade > secret or privileged information. Any unauthorized review, use, disclosure > or distribution is prohibited and may be a violation of law. If you are not > the intended recipient, please contact the sender by reply e-mail and > destroy all copies of the original message. > > > [Aspire Systems] > > This e-mail message and any attachments are for the sole use of the > intended recipient(s) and may contain proprietary, confidential, trade > secret or privileged information. Any unauthorized review, use, disclosure > or distribution is prohibited and may be a violation of law. If you are not > the intended recipient, please contact the sender by reply e-mail and > destroy all copies of the original message. >
