Hi,
I used the command (bin/zookeeper-shell.sh 0.0.0.0:2181 get /brokers/ids/0) to
get the broker metadata information.
How do I get the list of brokers attached to a zookeeper ?
Is my question clear ?
Thanks
Bala
-Original Message-
From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
I have set log level to DEBUG and saw something strange in the output. For
each topic partition, I see the following pattern:
[2014-06-26 16:00:24,467] ERROR No owner for partition [,0]
(kafka.tools.VerifyConsumerRebalance$)
...
[2014-06-26 16:00:24,469] INFO Owner of partition [,0] is
_-1403049
In ZK shell the following command:
ls /brokers/ids
will give you a list like this:
[0, 1, 2, 3, 4]
where items are broker ids you can further use to issue "get" request to ZK:
get /brokers/ids/
2014-06-26 12:37 GMT+04:00 Balasubramanian Jayaraman <
balasubramanian.jayara...@autodesk.com>:
Hi,
A few questions on timing related issues when certain parts of kafka go
down.
1. If zookeeper goes down, then I bring it back online, do I have to
restart the brokers?
2. If the brokers go down, producers will be erroring out. When the
brokers are back online, do I have to restart the proc
This is a bug in the tool. Please file a bug and attach these error/info
logs to it.
Thanks,
Neha
On Thu, Jun 26, 2014 at 5:24 AM, Yury Ruchin wrote:
> I have set log level to DEBUG and saw something strange in the output. For
> each topic partition, I see the following pattern:
>
> [2014-06-2
I dug some more and it seems like before these errors show up, I see a few
Zookeeper warnings, followed by Kafka errors.
11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient]
(pool-5-thread-1-EventThread) zookeeper state changed (Disconnected)
11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient]
(clojure
1. If zookeeper goes down, then I bring it back online, do I have to
restart the brokers?
You shouldn't have to restart the brokers if the broker side config "
zookeeper.connection.timeout.ms" is tuned such that zookeeper is brought
back online within the configured timeout.
2. If the brokers g
If ssh fails on the broker machine, I'm not why the broker would be in a
state where it maintains it's registration in zookeeper? If so, it will
automatically be marked dead and will not get elected as the leader. Have
you seen a case where such a broker became the leader? Could you elaborate
more
11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient]
(pool-5-thread-1-EventThread) zookeeper state changed (Disconnected)
11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient]
(clojure-agent-send-off-pool-
6-EventThread) zookeeper state changed
(Disconnected)
I wonder why your consumer disconnects from z
Yes I have seen it elected as the leader. The broker process is still up,
but due to some unknown hardware failure, it¹s not working 100%. It is,
however, able to participate in the cluster, and got itself elected as the
leader. I¹m not sure what the hardware problem is though, since I couldn¹t
log
Hello Virendra,
Did you have any producer/consumer clients running during the whole process?
Guozhang
On Wed, Jun 25, 2014 at 11:53 PM, Virendra Pratap Singh <
vpsi...@yahoo-inc.com.invalid> wrote:
> I am aware of lack of programmatic way of deleting topics in kafka 0.8.0.
> So using the sledg
Firstly, I'm not sure this process of deleting topics completely works,
especially in corner cases. Having said that, for #3, you should just be
deleting /brokers/topics/. If producers are sending
data to the Kafka cluster, it will recreate the topics once the brokers
come up.
Thanks,
Neha
On We
Could you file a JIRA for this? We may have to think through this
carefully. Black listing should be done in a way that maintains the
replication factor of the partition. Un blacklisting would then either
increase the replication factor or require carefully removing the replicas
on that broker.
Th
Most of these are available via JMX and others can be available via
zookeeper. I'm not sure why/how you would monitor "messages being deleted
by the broker". In general, monitoring via JMX is preferable to scraping
logs.
Thanks,
Neha
On Mon, Jun 23, 2014 at 11:51 PM, ravi singh wrote:
> Primar
No producers were active. The only producer in our pipeline is a storm
topology and had made it a point that the whole cluster was down and no
topology running.
What I did was nothing short of doing a fresh kafka cluster setup (barring
the fact that I didn¹t wiped the zookeepers as its the same set
I am 100% sure nothing was running. However I am not sure of consumers.
Would that make any difference?
I thought producer message write request could only cause kafka to
initiate topic creation, not consumer read. Would consumer read request
even succeed if the topic metadata is non-existent in zo
Hi All,
Thanks for all your responses.
JMX metrics are there and we do pull the metrics, but I would like to
capture the logs from Kafka lib as well especially WARN, FATAL and ERROR
etc to debug the issue.
To do this, we intercept Log4j logging and send it to Kafka Log Topics, but
I realize
We are in the process of engineering a system that will be using kafka.
The legacy system is using the local file system and a database as the
queue. In terms of scale we process about 35 billion events per day
contained in 15 million files.
I am looking for feedback on a design decision we ar
Hi there,
There have been some internal debates here about how far we can scale
Kafka. Ideally, we'd be able to make it scale to 90 billion events a day.
I've seen somewhere that linked scaled it up to 40 billion events a day.
Has anyone seen a hard plateau in terms of scalability? Does anyone
I think currently we do a little over 200 billion events per day at
LinkedIn, though we are not actually the largest Kafka user any more.
On the whole scaling the volume of messages is actually not that hard in
Kafka. Data is partitioned, and partitions don't really communicate with
each other, so
Hi, I’m having trouble understanding the results from running
kafka-consumer-perf-test. For low number of messages, I see very low
throughput in terms of messages / second. Here is a table of results:
fetch.size
data.consumed.in.MB
MB.sec
data.consumed.in.nMsg
nMsg.sec
1048576
0.0003
0
1
0.014
Using a single Kafka message to contain an application snapshot has the
upside of getting atomicity for free. Either the snapshot will be written
as a whole to Kafka or not. This is poor man's transactionality. Care needs
to be taken to ensure that the message is not too large since that might
caus
I have used 50MB message size and it is not a great idea. First of all you
need to make sure you have these settings in sync:
message.max.bytes
replica.fetch.max.bytes
fetch.message.max.bytes
I had not set the replica fetch setting and didn't realize one of my partitions
was not replicating aft
Thanks for the details Luke.
At what point would you consider a message too big?
Are you using compression?
Bert
On Thursday, June 26, 2014, Luke Forehand <
luke.foreh...@networkedinsights.com> wrote:
> I have used 50MB message size and it is not a great idea. First of all
> you need to make
What would you consider being a message that is “too large”
In April I ran a bunch of tests which I outlined in the following thread
http://grokbase.com/t/kafka/users/145g8k62rf/performance-testing-data-to-share
It includes a google doc link with all the results (its easiest to download
in
Hi folks, I just started a new Meetup specifically for Apache Kafka in NYC
(everyone is welcome of course) http://www.meetup.com/Apache-Kafka-NYC/
For the last couple of years we have been piggy backing talks and the
community with other NYC meetup groups (Storm, Cassandra, Hadoop, etc) and
figure
Hi, I'm trying to stream large message with Kafka into Spark. Generally
this has been working nicely, but I found one message (5.1MB in size) which
is clogging my pipeline up. I have these settings in server.properties:
fetch.message.max.bytes=10485760
replica.fetch.max.bytes=10485760
message.max
Hi Louis,
What are your consumer's config properties?
Guozhang
On Thu, Jun 26, 2014 at 5:54 PM, Louis Clark wrote:
> Hi, I'm trying to stream large message with Kafka into Spark. Generally
> this has been working nicely, but I found one message (5.1MB in size) which
> is clogging my pipeline
in the consumer.properties file, I've got (default?):
zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=100
group.id=test-consumer-group
thanks,
-Louis
On Thu, Jun 26, 2014 at 6:04 PM, Guozhang Wang wrote:
> Hi Louis,
>
> What are your consumer's config properties?
>
> G
Yes, thanks very much Luke - this is very helpful for my plans. I was under
the same impression but it’s always good to have verification, eh?!
On June 26, 2014 at 4:48:03 PM, Bert Corderman (bertc...@gmail.com) wrote:
Thanks for the details Luke.
At what point would you consider a messag
30 matches
Mail list logo
