yes, Kafka provides those metrics for monitoring the pending requests and
idle request handler, check their documents, you should get it, otherwise,
let me know.
from the operations you did:
1. Restarting service is not help should be expected, since that broker
is already being a bottlenec
Thanks Tony.
How can I check the number of pending requests and the idle percentage
of the requests handler on node 6? Are there any metrics for those?
Controller logs confirm that the controller is not able to send
updateMetadataRequest() to that particular node acting as the follower.
Whereas t
you mentioned:
1. broker disconnection error, normally the case I have ever seen is
when some broker is busy and can not response connection quickly to other
replicas.
2. partitions under-replicated, normally pint to some broker may have a
performance issue.
3. 90% under-replicat
Still hoping for some help here.
On Fri, Dec 7, 2018 at 12:24 AM Suman B N wrote:
> Guys,
> Another observation is 90% of under-replicated partitions have the same
> node as the follower.
>
> *Any help in here is very much appreciated. We have very less time to
> stabilize kafka. Thanks a lot in
Guys,
Another observation is 90% of under-replicated partitions have the same
node as the follower.
*Any help in here is very much appreciated. We have very less time to
stabilize kafka. Thanks a lot in advance.*
-Suman
On Thu, Dec 6, 2018 at 9:08 PM Suman B N wrote:
> +users
>
> On Thu, Dec 6
+users
On Thu, Dec 6, 2018 at 9:01 PM Suman B N wrote:
> Team,
>
> We are observing ISR shrink and expand very frequently. In the logs of the
> follower, below errors are observed:
>
> [2018-12-06 20:00:42,709] WARN [ReplicaFetcherThread-2-15], Error in fetch
> kafka.server.ReplicaFetcherThread$
Hi Romain,
Zookeeper needs to achieve quorum to function. This means that a strict
majority of nodes need to be alive. If your box1 dies, you lose 2 zookeeper
services, with only one left. Your system is already offline. So you would
need to spread your zookeeper services across 3 physical nodes.
if RF=2 and min.insync.replicas=1 (the default) then you shouldn't have
offline partitions if 1 of 3 brokers is down.
I'd first double check your topic config (and broker defaults) for the one
that went offline to verify RF/Min. Be sure to check each partition they
can be different! ( kafka-topics
whats your config for min.insync.replicas?
2018-01-17 13:37 GMT+01:00 Sameer Kumar :
> Hi,
>
> I have a cluster of 3 Kafka brokers, and replication factor is 2. This
> means I can tolerate failure of 1 node without data loss.
>
> Recently, one of my node crashed and some of my partitions went off
Got some info on this, I think this is mostly controlled by acks.
The default value of “1” requires an explicit acknowledgement from the
partition leader that the write succeeded. The strongest guarantee that
Kafka provides is with “acks=all”, which guarantees that not only did the
partition leade
please refer to the following doc.
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication
It talks about both the modes, I am not sure of which one is followed by
Kafka.
Also, is there a property to switch between the two.
-Sameer.
On Wed, Aug 9, 2017 at 3:24 PM, Manikumar wrote:
Not sure about, what you mean by Asynchronous and Synchronous
replication. details about replication are here:
http://kafka.apache.org/documentation/#replication
Kafka producers can choose whether they wait for the message to be
acknowledged
by 0,1 or all (-1) replicas by using "acks" config prope
Yes. Either using a Change Data Capture product like Oracle GoldenGate
Connector for Kafka or JDBC Source & Sink Kafka Connectors like the one
included with Confluent Open Source.
-hans
> On Dec 27, 2016, at 11:47 AM, Julious.Campbell
> wrote:
>
>
> Support
>
> Can Kafka handle data repl
Hello Avi,
Yes, replication factor of 1 means just one lead replica gets assigned for
the topic partitions.
To achieve HA of messages there are more things to watch out, not only
replication factor - metadata/ZK HA, producer acks, min-in-sync replicas,
rack awareness for replica assignment to nam
Kafka tolerates 2f failures with 2f+1 replicas by default. What error are
you seeing?
Thanks,
Jun
On Fri, Dec 5, 2014 at 10:01 AM, Ankit Jain
wrote:
> Hi All,
>
> I have two nodes kafka cluster and using replication factor 2. If both the
> node is running, then i am able to push data, but if a
Thanks for your response Jun.
JIRA has been filed (see link below). Please let me know if I should add
more details/context:
https://issues.apache.org/jira/browse/KAFKA-1464
Thanks,
Marcos Juarez
On Wed, May 21, 2014 at 8:40 AM, Jun Rao wrote:
> We don't have such throttling right now. Co
We don't have such throttling right now. Could you file a jira?
Thanks,
Jun
On Tue, May 20, 2014 at 10:29 PM, Marcos Juarez Lopez wrote:
> Hi,
>
> We have several Kafka clusters in production, and we've had to reassign
> replication a few times now in production. Some of our topic/partitions
Thanks Bert for response.
Are you saying that if i set replication factor to 3 then two nodes can go
down, in that case only 3rd down can serve the data?
In that case(replication factor 3 ) also if some partitions are in
synchronization process and meanwhile a node goes down then we may face
this
Only a single broker needs to be online for data to be available. In your
example partition 2 and 3 had copies of data on broker 0 and 1. When those
two brokers went down your data was unavailable. To withstand two brokers
going offline you would want to change your replication factor to 3.
O
After restarting it start working fine.
What i understand from this scenario is:
There must be atleast 2 brokers in isr (In Sync Replica) list of each topic
partition before any node goes down which has replica, otherwise topic may
go to unstable state.
Or
In other words, If topic partitions are
After restarting it start working fine.
What i understand from this scenario is:
On Mon, Apr 28, 2014 at 8:30 PM, Jun Rao wrote:
> Partition 2 and 3 are not accessible since both replicas are down. Have you
> restarted the 2 failed brokers?
>
> Thanks,
>
> Jun
>
>
> On Sun, Apr 27, 2014 at 11
Partition 2 and 3 are not accessible since both replicas are down. Have you
restarted the 2 failed brokers?
Thanks,
Jun
On Sun, Apr 27, 2014 at 11:58 PM, Hanish Bansal <
hanish.bansal.agar...@gmail.com> wrote:
> Hi All,
>
> I have 3 nodes kafka cluster with below configuration:
>
> Replication
Right, everything has to go through the leader. If the producer waits for
ack=committed, it will get an exception when sending m3. So the producer is
aware of that m3 may not be committed by the broker. The producer can
choose to resend m3 if it wants to be sure that m3 is safely stored at the
brok
Excellent thanks!
BTW, in the slides, it shows the the message 'm3' is lost.
I guess the leader is the single point of failure then when a producer
sends a message, meaning it can never bypass the leader and write to the
followers in case of leader failure right?
On Thu, Feb 28, 2013 at 8:35 AM
Thanks Jun, great slides!
On Wed, Feb 27, 2013 at 11:52 PM, Jun Rao wrote:
> Hi,
>
> I gave a talk on Kafka replication at ApacheCon yesterday. The slides can
> be found at
> http://www.slideshare.net/junrao/kafka-replication-apachecon2013(it's
> also added to Kafka wiki). I will share the link
Thanks Jun, makes sense.
On Feb 8, 2013 4:00 PM, "Jun Rao" wrote:
> That's right. If you are partitioning by key, that means you insist a
> message has to go to a certain partition, whether it's available or not.
> So, if a partition is not available, we will drop the message for the
> partition
Thanks, makes sense.
On Feb 8, 2013 4:00 PM, "Jun Rao" wrote:
> That's right. If you are partitioning by key, that means you insist a
> message has to go to a certain partition, whether it's available or not.
> So, if a partition is not available, we will drop the message for the
> partition in t
That's right. If you are partitioning by key, that means you insist a
message has to go to a certain partition, whether it's available or not.
So, if a partition is not available, we will drop the message for the
partition in the async mode and consistently throw an exception to the
caller in the s
So if the produces are partitioning by key we have to have replication if
we dont want messages to get lost when partition goes down l right ?
Thanks
On Feb 8, 2013 5:12 AM, "Jun Rao" wrote:
> We have fixed this issue in 0.8. Withreplication factor 1, if the producer
> doesn't care about partitio
We have fixed this issue in 0.8. Withreplication factor 1, if the producer
doesn't care about partitioning by key, messages will be sent to partitions
that are currently available.
Thanks,
Jun
On Thu, Feb 7, 2013 at 3:11 PM, Michal Haris wrote:
> Same here, summary was need as we have a fairly
Same here, summary was need as we have a fairly large ecosystem of multiple
0.7.2 clusters and I am planning to test upgrade to 0.8.
However, one thing creeping at the back of my mind regarding 0.8 is
something i have spotted in one thread few weeks ago namely that the
rebalance behaviour of produ
Same here. Thanks a lot Jun.
Regards,
Vaibhav
On Thu, Feb 7, 2013 at 10:38 AM, Felix GV wrote:
> Thanks Jun!
>
> I hadn't been following the discussions regarding 0.8 and replication for a
> little while and this was a great post to refresh my memory and get up to
> speed on the current replica
Thanks Jun!
I hadn't been following the discussions regarding 0.8 and replication for a
little while and this was a great post to refresh my memory and get up to
speed on the current replication architecture's design.
--
Felix
On Tue, Feb 5, 2013 at 2:21 PM, Jun Rao wrote:
> I just posted the
33 matches
Mail list logo
