Given that the kafka-bytes-out metric fell to zero I am guessing that the
broker was not able to replicate the partitions it was the leader for some
reason and hence the data loss. What I can't figure out is why the Producer
clients didn't get any errors from the broker. Still no clue as to why the
Also, the latency results show no major difference when using ack=0 or
ack=1. Why is that?
On Thu, Aug 13, 2015 at 11:51 AM, Yuheng Du
wrote:
> I am running an experiment where 92 producers is publishing data into 6
> brokers and 10 consumer are reading online data simultaneously.
>
> How should
Hey all,
Just wanted to confirm, this was totally our issue. Thank so much Todd and
Matt, our cluster is much more stable now.
Apache Kafka folks: I know 0.8.3 is slated to come out soon, but this is a
pretty serious bug. I would think it would merit a minor release just to
get it out there, s
I am running an experiment where 92 producers is publishing data into 6
brokers and 10 consumer are reading online data simultaneously.
How should I do to reduce the latency? Currently when I run the producer
performance test the average latency is around 10s.
Should I disable log.flush? How to d
Are you measuring latency as time between producer and consumer ?
In that case, the ack shouldn't affect the latency, cause even tough your
producer is not going to wait for the ack, the consumer will only get the
message after its commited in the server.
About latency my best result occur with s
Thank you Alvaro,
How to use sync producers? I am running the standard ProducerPerformance
test from kafka to measure the latency of each message to send from
producer to broker only.
The command is like "bin/kafka-run-class.sh
org.apache.kafka.clients.tools.ProducerPerformance test7 5000 100
I'm using last one, but not using the ProducerPerformance, I created my
own. but I think there is a producer.properties file in config folder in
kafka.. is that configuration not for this tester ?
On Thu, Aug 13, 2015 at 4:18 PM, Yuheng Du wrote:
> Thank you Alvaro,
>
> How to use sync producers
Yes there is. But if we are using ProducerPerformance test, it's configured
as giving input when running the test command. Do you write a java program
to test the latency? Thanks.
On Thu, Aug 13, 2015 at 3:54 PM, Alvaro Gareppe wrote:
> I'm using last one, but not using the ProducerPerformance,
Till recently we were on 0.8.1 and updated to 0.8.2.1
Everything seems to work but I am no longer seeing metrics reported from
the broker that was updated to the new version.
My config file has the following lines:
kafka.metrics.polling.interval.secs=5
kafka.metrics.reporters=kafka.metrics.Kafka
What exactly do we need to wait for before being sure the topic has been
created? Do we need to copy everything from TestUtils.createTopic:
def createTopic(zkClient: ZkClient,
topic: String,
numPartitions: Int = 1,
replicationFactor: Int = 1
What exactly should be checked to determine that the topic has been
successfully created since it is async?
I see a few different things in the code, in TestUtils.createTopic it
requires a list of Brokers to check, and checks for both
TestUtils.waitUntilMetadataIsPropagated() to each broker, and
Your batch.size is 8196 and your buffer.memory is 67108864. This means
67108864/8196
~ 8188 batches are in memory ready to the sent. There is only one thread io
thread sending them. I would guess that the io thread (
kafka-producer-network-thread) would be busy. Please check it in visual vm.
In st
Hi,
As I understand it, kafka brokers will store the incoming messages into
pagecache as much as possible and then flush them into disk, right?
But in my experiment where 90 producers is publishing data into 6 brokers,
I see that the log directory on disk where broker stores the data is
constantl
I'm running kafka version 0.8.2.1. In testing I've created a bunch of consumer
groups that use kafka based offsets and not zookeeper offsets, they are no
longer needed. Is there a way to remove unneeded consumer groups that use
kafka offset storage?
Thank you!
Aah that seems like a red herring - seems like the underlying cause is that
the MBeans I was trying to poll (through our metrics system) are no longer
present. We use collectd JMX to get metrics from Kafka and here is what I
see:
GenericJMXConfMBean: No MBean matched the ObjectName
"kafka.server"
Hi, All
I have a question about kafka-python producer, here is the record I have
id (uuid) | sensor_id (character) | timestamp | period (int) | current
(numeric) | date_received | factor (bigint)
"75da661c-bd5c-40e3-8691-9034f34262e3” | “ff0057” | "2013-03-21
11:44:00-07” | 60 |
Hello,
Is there some documentation on how to run performance tests ?
This link https://gist.github.com/jkreps/c7ddb4041ef62a900e6c lists the
commands, but some documentation explaining the usage would be useful.
Regards,
Saurav
Consumers can only fetch data up to the committed offset and the reason is
reliability and durability on a broker crash (some consumers might get the
new data and some may not as the data is not yet committed and lost). Data
will be committed when it is flushed. So if you delay the flushing,
consum
Hi AL,
kafka deals in blobs, so you generally have to manage serialization /
deserialization at the producer + consumer level. kafka-python's
SimpleProducer and SimpleConsumer classes are fairly naive and operate
exclusively on bytes, so if you use those you will have to serialize before
producing
The problem was that the metric names had all changed in the latest
version. Fixing the names seems to have done it.
On Thu, Aug 13, 2015 at 3:13 PM, Rajiv Kurian wrote:
> Aah that seems like a red herring - seems like the underlying cause is
> that the MBeans I was trying to poll (through our m
On Thu, Aug 13, 2015 at 4:10 PM, Kishore Senji wrote:
> Consumers can only fetch data up to the committed offset and the reason is
> reliability and durability on a broker crash (some consumers might get the
> new data and some may not as the data is not yet committed and lost). Data
> will be co
21 matches
Mail list logo
