@Otis thanks for your answer.
Of course it is not about manually watching connection. We already have
monitoring tools in place. But I would expect the kafka client to provide
means for get notified about potential connectivity issues.
Actual behavior is this: the client retries to reconnect inde
@Neha Narkede
Though monitoring the health of Kafka & Zookeeper clusters directly is
useful, it might not be enough.
Consider the following scenario:
You have a client consuming messages. Zookeeper and kafka nodes are
healthy. The monitor says everything is ok. For some reason, a connection
betwe
Yes, I used TopicMetaDataRequest to determine the lead broker for the partition.
I’ve also compared the broker selected by my code with the output from
"bin/kafka-topics.sh --describe[...]" and as far as I can tell, it’s using the
broker which is described as leader for the topic.
/Magnus
On
Hi,
I’m not sure if I’m confusing the concepts here, because I don’t understand
your answer?
I thought that the reason for making a PartitionOffsetRequest was to determine
which offset to use when fetching messages? Also, how is message.max.bytes
related?
I’m using the simple consumer api, a
Neha,
Thanks. I'd still love to know if anyone has used Consul and/or Confd to
manage a cluster.
Casey
From: Neha Narkhede [neha.narkh...@gmail.com]
Sent: Thursday, October 16, 2014 9:54 AM
To: users@kafka.apache.org
Subject: Re: Kafka/Zookeeper deploym
Hi,
The method kafka.api.FetchRequestBuilder.maxWait(maxwait: Int) does not give
the result I would have expected. I have not been able to find any
documentation or clarification on the method’s meaning.
My assumption is: If I send a fetch request for messages after offset X for a
partition wh
Alex,
In the above mentioned case I think monitoring the consumer lag
will be helpful
"kafka.consumer":name="([-.\w]+)-MaxLag",type="ConsumerFetcherManager".
Another approach I used before was to use metrics library to send
consumer stats to ganglia and use nagios alert on those stats, I
Hi Balaji,
You could do a rolling bounce of the brokers to do the in-place upgrade if
your partitions have at least two replicas. After that you may probably
need to rebalance the leaders if they are not balanced any more.
Guozhang
On Wed, Oct 15, 2014 at 10:53 AM, Seshadri, Balaji wrote:
> Hi
At least two including the leader?
On Fri, Oct 17, 2014 at 8:12 AM, Guozhang Wang wrote:
> Hi Balaji,
>
> You could do a rolling bounce of the brokers to do the in-place upgrade if
> your partitions have at least two replicas. After that you may probably
> need to rebalance the leaders if they a
Casey,
Could you describe a little more about how these would help manage a
cluster?
My understanding is that Consul provides service discovery and leader
election. Kafka already uses ZooKeeper for brokers to discover each other
and elect partition leaders. Kafka high-level consumers use ZK to
Yes.
On Fri, Oct 17, 2014 at 8:52 AM, Roger Hoover
wrote:
> At least two including the leader?
>
> On Fri, Oct 17, 2014 at 8:12 AM, Guozhang Wang wrote:
>
> > Hi Balaji,
> >
> > You could do a rolling bounce of the brokers to do the in-place upgrade
> if
> > your partitions have at least two re
Is Kafka supposed to throw exception if topic doesn't exist? It appears
that there is no exception thrown even though no messages are delivered and
there are errors logged in Kafka logs.
Roger,
My understanding of both, beyond what Zookeeper already does, are:
1. Consul can be used to monitor a service and report it's status. This can be
very useful for knowing if a service, such as Zookeeper of Kafka, goes down.
This can be done through a built-in web interface.
2. Confd lev
Thank you, Casey. Both of those features sound pretty useful.
On Fri, Oct 17, 2014 at 10:16 AM, Sybrandy, Casey <
casey.sybra...@six3systems.com> wrote:
> Roger,
>
> My understanding of both, beyond what Zookeeper already does, are:
>
> 1. Consul can be used to monitor a service and report it's
If you have "auto.create.topics.enable" set to "true" (default),
producing to a topic creates it.
Its a bit tricky because the "send" that creates the topic can fail
with "leader not found" or similar issue. retrying few times will
eventually succeed as the topic gets created and the leader gets
e
Hi, everyone,
We are fixing the mbean names in kafka-1482, by adding separate explicit
tags in the name for things like clientId and topic. Another thing that
some people have complained before is that we use quotes in the jmx name.
Should we also just get rid of the quotes as part of kafka-1482?
Thanks! How can I tell if I am using async producer? I thought all the
sends are async in nature
On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira
wrote:
> If you have "auto.create.topics.enable" set to "true" (default),
> producing to a topic creates it.
>
> Its a bit tricky because the "send" that
0.8.1.1 producer is Sync by default, and you can set producer.type to
async if needed.
On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia wrote:
> Thanks! How can I tell if I am using async producer? I thought all the
> sends are async in nature
> On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira
> wro
btw. I got a blog post where I show how I work around the blocking
hasNext() thing.
May be helpful:
http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/
On Thu, Oct 16, 2014 at 12:52 PM, Neha Narkhede wrote:
> Josh,
>
> The consumer's API doesn't allow you to specify
Little confused :) From one of the examples I am using property
request.required.acks=0,
I thought this sets the producer to be async?
On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira
wrote:
> 0.8.1.1 producer is Sync by default, and you can set producer.type to
> async if needed.
>
> On Fri, Oct 1
+1 to get rid of quotes, thanks!
—
Sent from Mailbox
On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao wrote:
> Hi, everyone,
> We are fixing the mbean names in kafka-1482, by adding separate explicit
> tags in the name for things like clientId and topic. Another thing that
> some people have compl
Its using the sync producer without waiting for any broker to acknowledge the
write. This explains the lack of errors you are seeing.
—
Sent from Mailbox
On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia
wrote:
> Little confused :) From one of the examples I am using property
> request.required.
Still don't understand the difference. If it's not waiting for the ack then
doesn't it make async?
On Fri, Oct 17, 2014 at 12:55 PM, wrote:
> Its using the sync producer without waiting for any broker to acknowledge
> the write. This explains the lack of errors you are seeing.
>
> —
> Sent from
Sorry if I'm confusing you :)
Kafka 0.8.1.1 has two producers sync and async. You are using the sync
producer without waiting for acks. I hope this helps?
Regardless, did you check if the partition got created? are you able
to produce messages? are you able to consume them?
Gwen
On Fri, Oct 17,
My understanding of sync is that producer waits on .send until Kafka
receives the message. And async means it just dispatches the message
without any gurantees that message is delivered. Did I get that part right?
On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira wrote:
> Sorry if I'm confusing you :
+1 on getting rid of the quotes.
On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal <
magnus.spang...@deltaprojects.com> wrote:
> +1 to get rid of quotes, thanks!
>
>
>
>
>
>
> —
> Sent from Mailbox
>
> On Fri, Oct 17, 2014 at 8:54 PM, Jun Rao wrote:
>
> > Hi, everyone,
> > We are fixing the mbea
+1 as well for getting rid of the quotes. Isn't it an inheritance of
metrics-core? Anyway, hope Kafka devs will find a way to get rid of these!
Cordialement / Best Regards,
Alexandre VERMEERBERGEN
R&D Cloud Service Supervision Development Director
–––
Office: +33 1 6162 4992
+1 on getting rid of quotes in jmx mbeans.
Thanks,
Raja.
On Fri, Oct 17, 2014 at 4:48 PM, Neha Narkhede
wrote:
> +1 on getting rid of the quotes.
>
> On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal <
> magnus.spang...@deltaprojects.com> wrote:
>
> > +1 to get rid of quotes, thanks!
> >
> >
>
If I understand correctly (and I'll be happy if someone who knows more
will jump in and correct me):
The Sync/Async part is not between the producer and the broker. Its
between you and the producer. The Sync producer takes your message and
immediately contacts the broker, sends the message, either
Thanks for the info. I see there are tons of parameters but is there a
place that lists some important performance specific parameters?
On Fri, Oct 17, 2014 at 2:43 PM, Gwen Shapira wrote:
> If I understand correctly (and I'll be happy if someone who knows more
> will jump in and correct me):
>
>
There's some good advice here: https://kafka.apache.org/081/ops.html
And you may enjoy this post too:
http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
On Fri, Oct 17, 2014 at 5:52 PM, Mohit Anchlia wrote:
> Thanks for the info. I see there are tons of parameters but is the
Hi,
I have a Kafka 0.8.1 cluster. I used the ConsumerOffsetChecker tool to
check the lag of consumer groups. I found that for some partition, the tool
returns negative value for the "lag " column. Is this a known issue that
has been seen before? I find that the negative value prevents the consume
32 matches
Mail list logo
