I have running zookeeper 3.4.5-cdh5.2.0 cluster on 3 nodes and managing
with cm5.2, Integrated kafka with "
http://www.cloudera.com/content/cloudera/en/developers/home/cloudera-labs/apache-kafka.html";,
The problem is, when I try to create a topic with
*# bin/kafka-topics.sh --create --zookeep
Cloudera Manager utilizes the chroot znode structures so you should connect
with --zookeeper localhost:2181/kafka
Or whatever value CM has set for the chroot path of your installation.
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Sec
Good day,
I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed that
the CPU usage on the broker machines went up by roughly 40%, from ~60% to
~100% and am wondering if anyone else has experienced something similar?
The load average also went up by 2x-3x.
We're running on EC2 and th
Hi,
We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
In one of our application, we want to get all partitions' latest offsets,
so we use getoffsetbefore java API (latest).
We believe at some time, 1 of the partition's latest offset we got is much
smaller than its real latest o
1 additional information we found in the kafka’s application log since the
MAGIC time:
2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
kafka.cluster.Partition - Partition [a.s.3,26] on broker 5: Shrinking
ISR for partition [a.s.3,26] from 5,4 to 5
2014-12-04 09:59:36,728 [kafka-scheduler-2] ERRO
Please subscribe for the group
Ajay
Hi,
I am going to setup Kafka clusters having 3 brokers in Datacenter 1. Topics
can be created time to time. Each topic can have varying partitions mostly
1,10 or 20. Each application might have different partitioning algorithm
that we don't know(let it be hidden from ops team).
We want to setup
Neha,
This seems to return the offset of the next message that will be published. If
I fetch at that offset I will block until a new message is published to that
partition.
I am actually trying to read the contents of the latest message in the
partition, and based on info in the message resubs
The returned latest offset - 1 will be the offset of the last message.
Sorry, should've made it clear in my last email. Let me know if that helps.
On Mon, Dec 8, 2014 at 8:32 AM, Orelowitz, David
wrote:
> Neha,
>
> This seems to return the offset of the next message that will be
> published. If
Neha,
This is not what I am seeing, -1 seems to return the offset of the next
message that will be published to the partition.
If I subtract 1 from the offset, then I get the offset of the last message, and
can fetch that message and read it.
From: Neha Narkhede [mailto:n...@confluent.io]
Se
Hi Madhukar,
>From the same documentation link you referred to -
The source and destination clusters are completely independent entities:
> they can have different numbers of partitions and the offsets will not be
> the same. For this reason the mirror cluster is not really intended as a
> fault-
Thanks for reporting the issue. Would you mind running hprof and sending
the output?
On Mon, Dec 8, 2014 at 1:25 AM, Mathias Söderberg <
mathias.soederb...@gmail.com> wrote:
> Good day,
>
> I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed that
> the CPU usage on the broker mach
I think that A will not be able to become a follower until B becomes a leader.
On Sun, Dec 7, 2014 at 11:07 AM, Xiaoyu Wang wrote:
> On preferred replica election, controller sends LeaderAndIsr requests to
> brokers. Broker will handle the LeaderAndIsr request by either become a
> leader or becom
Xiaoyu,
I think your question is whether the following sequence can happen?
1. A received becomes-leader, stop fetching from B.
2. B received a producer request, and accepts it and append to log.
3. B received becomes-follower, cap its log and start fetching from A, and
hence the message sent on
Helin,
Is there a leader movement just before the get latest offset call? If your
follower is not synced and it then becomes the leader due to some reason,
it will not have the complete partition data.
Guozhang
On Mon, Dec 8, 2014 at 3:02 AM, Helin Xiang wrote:
> 1 additional information we fo
Hi Yury,
Originally the producer behavior under null-key is "random" random, but
later changed to this "periodic" random to reduce the number of sockets on
the server side: imagine if you have n brokers and m producers where m >>>
n, with random random distribution each server will need to maintai
Guozhang, thanks for the super clear answer. Yes, that's the scenario I was
wondering.
On Mon, Dec 8, 2014 at 1:01 PM, Guozhang Wang wrote:
> Xiaoyu,
>
> I think your question is whether the following sequence can happen?
>
> 1. A received becomes-leader, stop fetching from B.
> 2. B received a
Ok, based on all the feedbacks that we have heard, I plan to do the
following.
1. Keep the generic api in KAFKA-1797.
2. Add a new constructor in Producer/Consumer that takes the key and the
value serializer instance.
3. Have KAFKA-1797 reviewed and checked into 0.8.2 and trunk.
This will make it
Thank you Jay. I agree with the issue that you point w.r.t paired
serializers. I also think having mix serialization types is rare. To get
the current behavior, one can simply use a ByteArraySerializer. This is
best understood by talking with many customers and you seem to have done
that. I am conv
Topic deletion doesn't quite work in 0.8.1.1. It's fixed in the upcoming
0.8.2 release.
Thanks,
Jun
On Wed, Dec 3, 2014 at 6:17 PM, Andrew Jorgensen <
ajorgen...@twitter.com.invalid> wrote:
> We are currently running 0.8.1.1, I just double checked. One other thing
> that may be related is I bro
kafka.network.Processor is on the broker. Are you killing the brokers as
well?
Thanks,
Jun
On Thu, Dec 4, 2014 at 5:33 PM, Shannon Lloyd wrote:
> Hi,
>
> I am using the high-level consumer on 0.8.2-beta. I'm attempting to close a
> ConsumerConnector (actually a handful of connectors), but am n
Kafka tolerates 2f failures with 2f+1 replicas by default. What error are
you seeing?
Thanks,
Jun
On Fri, Dec 5, 2014 at 10:01 AM, Ankit Jain
wrote:
> Hi All,
>
> I have two nodes kafka cluster and using replication factor 2. If both the
> node is running, then i am able to push data, but if a
Not explicitly. Some additional background might help. I'm running an
integration test using an embedded Kafka cluster and ZK quorum (ie all in
process). In my @Before method I fire up the cluster. In my @After method I
shut the cluster down. When I call shutdown on the connector during my
test, th
23 matches
Mail list logo
