Got it. This was indeed my problem. Thanks so much!
On Fri, Nov 21, 2014 at 10:39 PM, Gwen Shapira
wrote:
> I think the issue is that you are:
> " running the above snippet for every broker ... I am assuming that
> item.partitionsMetadata() only returns PartitionMetadata for the partitions
> thi
I think the issue is that you are:
" running the above snippet for every broker ... I am assuming that
item.partitionsMetadata() only returns PartitionMetadata for the partitions
this broker is responsible for "
This is inaccurate. Each broker will check ZooKeeper for PartitionMetadata
and return
I have used the kafka-topics.sh script to create a topic with a number of
partitions.
bin/kafka-topics.sh --zookeeper myZKPath-create --topic myTopic
--partitions 1024 --replication-factor 3
I was expecting this topic to be created with 1024 total partitions across
3 of my kafka brokers. Instea
In the initial state all replicas are in isr list， but sometimes when I
check the topic state, the replica can never become isr even if actually it
is synchronized. I saw in the log, the leader print expand isr request，but
did not work. I found a interesting thing, the shrink and expand request
ha
Yes, we have consumers for the topic fetching at the same time and we can
see the its BytesPerSec metric. So we find it hard to explain why
BytesInPerSec would be greater than BytesOutPerSec on the broker given that
we have active consumer and replication.
On Fri, Nov 21, 2014 at 4:32 PM, Guozhan
Bhavesh,
In 0.9 consumer would not talk to ZK and will be single threaded, which
will be easier to provide monitoring mechanisms.
Guozhang
On Thu, Nov 20, 2014 at 8:15 PM, Jun Rao wrote:
> Can you just monitor the consumer byte/message/fetch rate?
>
> Thanks,
>
> Jun
>
> On Thu, Nov 20, 2014 a
Are you having consumers fetching from these topics at the same time?
BytesInPerSec only counts the bytes appended to the log from the produce
request, and BytesOutPerSec counts the bytes fetch from fetch requests;
hence both replica fetcher and normal consumer fetcher's requests count in
BytesOut
We observed that for a topic, BytesIn is greater than BytesOut. We are
under the impression that BytesOut should include replication. The topic
has two replicas for each partitions and all replicas are in sync. Then
BytesOut should be at least same as BytesIn since it always needs to
replicate to a
Ok, then your understanding is correct. If the queue is full then the
enqueue operation should result in a kafka.common.QueueFullException in 20
ms. Not sure why this is not happening since in the code, we simply call
queue.offer() with the timeout. It could be due to contention. Do you have
lots o
Hi, all
I am trying to get kafka web console work, but seems it only works few
hours and fails afterwards, below is the error messages on the screen. I am
assuming something wrong with the DB, I used to swap H2 to mysql, but
didn't help. Anyone has similar problem?
-
.
.
at sun.
All,
Thanks. Everything you've said is what I was already assuming/knew. I was
wondering if there were alternative solutions as a concern that came up was
similar to the burst situation where our existing cluster gets backed up, but
if we add more partitions we still end up with those same pa
See the description of log.retention.bytes here:
https://kafka.apache.org/08/configuration.html
You can set a basic value per log-partition, but you'll need to do some
math to work out an appropriate value based on:
1. The number of partitions per topic
2. The number of topics
3. The capacity of t
Hi All,
Can anyone give some inputs about retention policy as I am trying to save
larger data in the topics and hence going out of disk space.
Regards,
Nilesh Chhapru.
---Disclaimer--
13 matches
Mail list logo
