We sort of have the same situation where our analytics DC is one of the
main producer DC too. If you use Kafka only for analytics it is fine to
produce directly to the analytics cluster from that DC and mirror the rest.
However we also want to be able to run things locally that will consume
local
Neha, thanks for the response.
So the only point of the magic byte is to indicate that the rest of the message
is Avro encoded? I noticed that in Camus a 4 byte int id of the schema is
written instead of the 16 byte SHA. Is this the new preferred way? Which is
compatible with https://issues.ap
Actually, I am not sure if I understand the trouble that you mentioned.
Could you elaborate that a bit more?
Thanks,
Jun
On Wed, Aug 21, 2013 at 12:30 PM, Yu, Libo wrote:
> Hi,
>
> This is from kafka faq:
>
>
> * Each partition is not consumed by more than one consumer
> thread/process in
We define the LinkedIn Kafka message to have a magic byte (indicating Avro
serialization), MD5 header followed by the payload. The Hadoop consumer
reads the MD5, looks up the schema in the repository and deserializes the
message.
Thanks,
Neha
On Wed, Aug 21, 2013 at 8:15 PM, Mark wrote:
> Does
My way is build kafka and install pom in local.
Then set: "Local Maven Repository" at
"file:///Users/jameswu/.m2/repository" in resolvers.
On Wed, Aug 21, 2013 at 5:21 AM, Lorenz Knies wrote:
> hi joe,
>
> thanks for the hint with the apache repo.
> the from directive did not work for me. whil
Does LinkedIn include the SHA of the schema into the header of each Avro
message they write or do they wrap the avro message and prepend the SHA?
In either case, how does the Hadoop consumer know what schema to read?
Thanks for sending this across. One thing that is weird about the thread
dump is that the thread names are not included, so it is somewhat difficult
to read. But just from the stack traces, I don't see the
ReplicaFetcherThreads alive. Can you grep your broker logs to see if the
replica fetcher thre
This log message is marked INFO since it is normal, though I think the
message content can be improved. The broker that does not get elected as
the controller will have this log message that indicates it tried to become
controller but some other broker succeeded instead.
Thanks,
Neha
On Wed, Aug
Yu,
The controller is the broker that has the ActiveControllerCount jmx value
of 1. At any point of time, only one broker in a Kafka cluster should have
a value of 1 for this jmx mbean.
I personally find it very complex to find the replica fetcher thread's lag
for a particular partition that is u
That would be great!
-Jay
On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen wrote:
> Hi Jay,
>
> I'm planning to test run Kafka on Windows in our test environments
> evaluating if it's suitable for production usage.
>
> I can provide feedback with the patch how well it works and if we encounter
>
Hi Jay,
I'm planning to test run Kafka on Windows in our test environments
evaluating if it's suitable for production usage.
I can provide feedback with the patch how well it works and if we encounter
any functional or perf problems.
Tim
On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps wrote:
>
Elizabeth and I have a patch to support our memory mapped offset index
files properly on Windows:
https://issues.apache.org/jira/browse/KAFKA-1008
Question: Do we want this on 0.8 or trunk? I would feel more comfortable
with it in trunk, but that means windows support in 0.8 is known to be
broken
We have 3 brokers in our kafka cluster (1,2,3). Broker 2 somehow is not in isr.
I restarted it and it did not help at all. And we notice in many case we have to
restart the whole cluster to get it back. This is our top priority concern
currently.
Here is the log after the restart:
[2013-08-21 1
No, there isn't, not at the very start when there is no state in
Zookeeper. Once there is state the Kafka team have told me that
rebalancing will not result in any dupes.
However, if there is no state in Zookeeper and your partitions are
empty, simply wait until all consumers have balanced before
Hi team,
I found this line from the log:
[2013-08-21 08:22:55,513] INFO conflict in /controller data: 2 stored data: 3
(kafka.utils.ZkUtils$)
I checked the zookeeper and the data in /controller is 3.
We have 3 brokers in our cluster. The broker ids are 0,1,2 or 1,2,3?
Regards,
Libo
Hi,
This is from kafka faq:
* Each partition is not consumed by more than one consumer thread/process
in each consumer group. This allows to have each process consume in a single
threaded fashion to guarantee ordering to the consumer within the partition (if
we split up a partition of ord
Hi Neha,
Which broker is controller broker and how is it defined?
Regards,
Libo
-Original Message-
From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
Sent: Tuesday, August 20, 2013 10:56 AM
To: users@kafka.apache.org
Subject: Re: How to get broker back to ISR
Once the broker is re
Hi Neha,
Here is the thread dump. The process has been up for almost 7 hours and
Nothing is in the log but numerous " INFO Closing socket connection to "
Thanks
Libo
Deadlock Detection:
No deadlocks found.
Thread 20075: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Com
We never deleted it. Either it was never created or deleted somehow.
Regards,
Libo
-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com]
Sent: Wednesday, August 21, 2013 11:41 AM
To: users@kafka.apache.org
Subject: Re: issue with kafka-preferred-replica-election.sh
Libo,
1 topic.
I don't understand the second question.
Philip
On Aug 21, 2013, at 9:52 AM, Tom Brown wrote:
> Philip,
>
> How many topics per broker (just one?) And what is the read/write profile
> of your setup?
>
> --Tom
>
>
> On Wed, Aug 21, 2013 at 12:24 PM, Philip O'Toole wrote:
>
>> On
Hey Tom,
I tried to expand the FAQ on this, let me know if you feel like it answers
your question:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic%3F
Folks--let me know if I missed anything there. I tried to give both a
simple answer and a mor
Using list-topics command, you can figure out which replicas are not in
ISR. Using the lag jmx (
http://kafka.apache.org/documentation.html#monitoring), you can figure out
if the follower is catching up.
Thanks,
Jun
On Wed, Aug 21, 2013 at 9:47 AM, James Wu wrote:
> Sorry, the link is :
>
> h
It's optional. If you want to access jmx through http, you can include that
jar.
Thanks,
Jun
On Wed, Aug 21, 2013 at 7:39 AM, Yu, Libo wrote:
> Hi team,
>
> I got the message from a broker log:
>
> [2013-08-21 08:22:55,502] INFO Will not load MX4J, mx4j-tools.jar is not
> in th
> e classpath
Added to the 0.8 documentation.
Thanks,
Jun
On Tue, Aug 20, 2013 at 9:22 AM, Jay Kreps wrote:
> Is there any way to channel these many excellent email threads into
> documentation improvements :-)
>
> -Jay
>
>
> On Mon, Aug 19, 2013 at 8:55 PM, Jun Rao wrote:
>
> > We also have a jmx bean th
All per topic configuration properties below have the format of csv (e.g.,
"topic1:value1,topic2:value2"). Updated our website to make it clear.
Thanks,
Jun
On Tue, Aug 20, 2013 at 6:16 AM, Paul Christian
wrote:
> Jun,
>
> For my first example is that syntax correct? I.e.
>
> log.retention.byt
Philip,
How many topics per broker (just one?) And what is the read/write profile
of your setup?
--Tom
On Wed, Aug 21, 2013 at 12:24 PM, Philip O'Toole wrote:
> On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> > I have not been able to find reliable advice regarding how many
> par
Sorry, the link is :
http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/raw/%3CFF142F6B499AE34CAED4D263F6CA32901D34E289%40EXTXMB19.nam.nsroot.net%3E/1
On Thu, Aug 22, 2013 at 12:46 AM, James Wu wrote:
> Thank you Neha.
>
> Actually my situation was more like this topic:
> http://m
Thank you Neha.
Actually my situation was more like this topic:
http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/browser
My broker didn't back to ISR.
I will try to reproduce later.
On Tue, Aug 20, 2013 at 10:55 PM, Neha Narkhede wrote:
> Once the broker is restarted, the contr
Thanks for listing the possible conditions for a stalled consumer, Philip.
I added this to the FAQ -
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F
Frank,
Let us know if the FAQ above helps to understand your issue or can be
improved.
Thanks,
Neha
The high-level consumer will block if a) there are no more messages
available, b) the next message available is larger than the maximum fetch size
you
have specified, or c) your client code simply stops pulling messages from the
iterator (the blocking queue will fill up).
Not sure if this applies
On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> I have not been able to find reliable advice regarding how many partitions
> should exist on a single broker. How many partitions have you used, and
> what kind of throughput have you seen?
We settled on 8 per broker, and usually run 3 b
Hi,
My java consumer client will suddenly stop consuming the stream. While I
enabled the 'TRACE' log, I found that this stop always happened in the end
of a queuechunk consumption - after several times' chunk actually. What
weird is that in log there are not any useful information. It occurred
sud
Could you take a thread dump on that broker and send it across? One of the
possibilities is the replica fetcher thread is somehow dead.
Thanks,
Neha
On Aug 21, 2013 8:00 AM, "Yu, Libo" wrote:
> I checked the log of normal restart. The replication manager should start
> to handle
> leader and isr
Libo,
Just want to clarify, in your case after you created your topic, the
/brokers/topics/my_topic/partitions was never created or it was deleted
somehow?
Guozhang
On Wed, Aug 21, 2013 at 7:25 AM, Yu, Libo wrote:
> For the path /brokers/topics/my_topic/partitions, if you remove partitions
>
I checked the log of normal restart. The replication manager should start to
handle
leader and isr request after the server is up. What may stop it from doing that?
Is it because of missing mx4j-tools.jar?
Regards,
Libo
From: Yu, Libo [ICG-IT]
Sent: Wednesday, August 21, 2013 10:51 AM
To: 'user
Hi team,
We have three kafka brokers in a production cluster. We use replication factor
3 for all topics.
We notice quite frequently one broker is not in isr. Sometimes after it is
restarted, it
will go back to isr. Sometimes even after it is restarted, it will not go back
to isr.
In today's c
Hi team,
I got the message from a broker log:
[2013-08-21 08:22:55,502] INFO Will not load MX4J, mx4j-tools.jar is not in th
e classpath (kafka.utils.Mx4jLoader$)
Should it be in kafka_2.8.0-0.8.0-SNAPSHOT.jar or
kafka-assembly-0.8.0-SNAPSHOT-deps.jar?
Is that a packaging error with sbt? Thank
For the path /brokers/topics/my_topic/partitions, if you remove partitions and
run
kafka-preferred-replica-election.sh, it terminates right away with keeper
exception.
Regards,
Libo
-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com]
Sent: Wednesday, August 21, 2013 12
Say if you have 3 brokers and 2 consumers in a consumer group, you may choose 6.
So when one broker is offline, the load is still well balanced.
If you use 12, when you add a fouth broker, the load is still well balanced.
Regards,
Libo
-Original Message-
From: Tom Brown [mailto:tombrow
I have not been able to find reliable advice regarding how many partitions
should exist on a single broker. How many partitions have you used, and
what kind of throughput have you seen?
Thanks in advance!
--Tom
40 matches
Mail list logo
