Hi all:We have a cluster of 3 brokers(id : 0,1,2). We restart(simply use
stop.sh and start.sh in bin directory) broker 1. The broker started
successfully. However, all the partitions' leader moved to other brokers and no
data were written into broker 2. This is the status of one
topic:Topic:wx_
Thanks Jun. It seems it was an issue with jruby client I was using. Now,
they fixed it.
-Pranay
On Mon, Feb 23, 2015 at 4:57 PM, Jun Rao wrote:
> Did you enable auto offset commit?
>
> Thanks,
>
> Jun
>
> On Tue, Feb 17, 2015 at 4:22 PM, Pranay Agarwal
> wrote:
>
> > Hi,
> >
> > I am trying to
Hi,
There are bunch of metrics available for performance monitoring. These
metrics are can be monitored
by JMX monitoring tool (Jconsole).
https://kafka.apache.org/documentation.html#monitoring.
Some of the available metrics reporters are:
https://cwiki.apache.org/confluence/display/KAFKA/JM
Hi,
Please let me know how to measure the performance metrics while
pushing/consuming the message to/from the topic.
Thanks.
Camus uses the simple consumer, which doesn't have the concept of "consumer
group" in the API (i.e. Camus is responsible for allocating threads to
partitions on its own).
The client-id is hard coded and is "hadoop-etl" in some places (when it
initializes the offsets) and "camus" in other places.
T
Nice :) I like the idea of tying topic name to avro schemas.
I have experience with other people's data, and until now I mostly
recommended:
...
So we end up with things like:
etl.onlineshop.searches.validated
Or if I have my own test dataset that I don't want to share:
users.gshapira.newapp.tes
* ZK was not built for 5K/s writes type of load
* Kafka 0.8.2.0 allows you to commit messages to Kafka rather than ZK. I
believe this is recommended.
* You can also commit batches of messages (i.e. commit every 100 messages).
This will reduce the writes and give you at least once while controlling
Similar pattern for that too. Mostly hovering below.
-Zakee
On Tue, Feb 24, 2015 at 2:43 PM, Jun Rao wrote:
> What about RequestHandlerAvgIdlePercent?
>
> Thanks,
>
> Jun
>
> On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote:
>
> > Hi Jun,
> >
> > With ~100G of data being pushed per hour across 35
HI Jun,
Thanks for info.
Thanks,
Bhavesh
On Tue, Feb 24, 2015 at 2:45 PM, Jun Rao wrote:
> These two metrics are always registered, whether the replica is the leader
> or the follower.
>
> Thanks,
>
> Jun
>
> On Mon, Feb 23, 2015 at 6:40 PM, Bhavesh Mistry <
> mistry.p.bhav...@gmail.com>
> wro
> Could you explain a bit more what you want to achieve through batching?
> Better throughput or atomicity?
Sure! I've assumed that there's per-message atomicity and a per-partition
ordering guarantee with KafkaProducer.send(), but nothing beyond that.
My hopes are to reduce latency from when my
I usually got that exception if
a) there is a firewall issue
b) you are trying to consume messages from another node and have not set
your advertised host name to the public ip address in the kafka server
settings.
On Tue, Feb 24, 2015 at 12:07 PM, alexey yakubovich <
alexe...@yahoo.com.invali
Hmm, anything special with that topic (e.g., did you run any admin
operation on it)? Also, does bouncing the brokers fix the issue?
Thanks,
Jun
On Tue, Feb 10, 2015 at 1:31 PM, Sumit Rangwala
wrote:
> On Thu, Feb 5, 2015 at 12:37 PM, Sumit Rangwala
> wrote:
>
> >
> >
> > On Wed, Feb 4, 2015 a
A messageSet does get written to the local log file atomically. However,
the set may not be fetched atomically by the replica fetcher. So, on
switching the leader of a partition, it's possible that only a subset of
the messages in the set show up in the new leader. The exception is that if
the mess
If a consumer comes from a new consumer group, it will by default consume
from the log end.
In you first scenario, because you started consumer after producing data,
when the consumer starts it will consume from the latest. So you will not
be able to see the messages even though they are in the bro
These two metrics are always registered, whether the replica is the leader
or the follower.
Thanks,
Jun
On Mon, Feb 23, 2015 at 6:40 PM, Bhavesh Mistry
wrote:
> Hi Kafka Team or User Community ,
>
> After leadership election or switch between follower/leader of partition
> for given topic, doe
What about RequestHandlerAvgIdlePercent?
Thanks,
Jun
On Mon, Feb 23, 2015 at 8:47 PM, Zakee wrote:
> Hi Jun,
>
> With ~100G of data being pushed per hour across 35 topics
> (replication-factor 3), the NetworkProcessorAvgIdlePercent is mostly
> showing below 0.5 sometimes when the producers sen
Joe,
It turns out that KAFKA-1724 is already resolved in trunk through another
jira. However, the change in that jira is big. Since KAFKA-1724 seems to
only happen when there is ZK session timeout or the ZK service is not
available, I don't think it's worth fixing the issue in 0.8.2.
Thanks,
Jun
At a fixed number. I think my problem was that I assuming that there
was a single partition in my code ... still working on this.
On Tue, Feb 24, 2015 at 11:06 AM, Zakee wrote:
> Does that count get frozen on a fixed number or any random number?
>
> -Zakee
>
> On Mon, Feb 23, 2015 at 9:48 AM, Stu
Hi,
It is a little long, since I wanted to explain the use case and then ask
questions, so thanks for your attention
Use case:
We have a use case where everything in the queue has to be consumed at
least once. So the consumer has to have "consumed" (saved in some
destination database) the messag
somehow I have to let the consumer run once BEFORE the messages are
produced. then run consumer after messages are produced, then it works:
export topic=ttnew21
sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
--topic $topic --partitions 1 --replication-factor 1
echo "fir
in the following code, I just create a new topic, send some messages onto
the new topic with console producer, then read it with console consumer.
but after the 3rd line, nothing is printed out.
export topic=ttnew2
sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
--topic
That I don't know Jun. I dug a little into the Ruby source yesterday, and
it appeared that we might be affected by the ruby client issuing a topic
metadata request explicitly. I was concerned that we were triggering the
behaviour seen in https://issues.apache.org/jira/browse/KAFKA-1507
However, I'l
nah, it is expected behavior for a synchronous call, it waits and timesout.
Sorry, should have been more specific.
I was really looking for async.
On Tue Feb 24 2015 at 3:56:19 PM Joe Stein wrote:
> Sounds like https://issues.apache.org/jira/browse/KAFKA-1788 maybe
> On Feb 24, 2015 2:28 PM, "S
We have a global namespace hierarchy for topics that is exactly our Avro
namespace with Class Name. The template is basically:
.Core.
..
The up side of this for us is that since the topics are named based on the Avro
schema namespace and type, we can look up the avro schema in the Avro Schema
Hello all,
Just wondering if those with a good amount of experience using Kafka in
production with many streams have converged on any sort of naming convention.
If so would you be willing to share?
Thanks in advance,
Taylor
Sounds like https://issues.apache.org/jira/browse/KAFKA-1788 maybe
On Feb 24, 2015 2:28 PM, "Scott Chapman" wrote:
> Yea, however I don't get async behavior. When kafka is down the log blocks,
> which is kinda nasty to my app.
>
> On Tue Feb 24 2015 at 2:27:09 PM Joe Stein wrote:
>
> > Producer
Xiaoyu,
For 1, I have a patch for 0.8.2 in
https://issues.apache.org/jira/browse/KAFKA-1984. Could you test it out and
see if it fixes your issue?
For 2, I did some local testing. The only issue I saw is that producer can
block on close since there are still unsent messages in the bufferpool.
Thi
I looked at the example camus.properties, it does not specify this. how
could we set the groupId ?
thanks
Yang
Hi, I am learning kafka, ... hard way
I try to run the example given here:
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Whatever I do i I have
java.nio.channels.ClosedChannelException when try to send message
Exception in thread "main" java.nio.channels.Clos
Yea, however I don't get async behavior. When kafka is down the log blocks,
which is kinda nasty to my app.
On Tue Feb 24 2015 at 2:27:09 PM Joe Stein wrote:
> Producer type isn't needed anymore with the new producer so in the the
> logger properties just leave that out in 0.8.2 and it should wo
Interesting, looks like a breaking change from 0.8.1
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala
to 0.8.2
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala
On Tue, Feb 24, 2015 at 2:21 P
Producer type isn't needed anymore with the new producer so in the the
logger properties just leave that out in 0.8.2 and it should work.
On Tue, Feb 24, 2015 at 2:24 PM, Joe Stein wrote:
> Interesting, looks like a breaking change from 0.8.1
> https://github.com/apache/kafka/blob/0.8.1/core/src
I'm including log4j-1.2.17, slf4j-api-1.7.6, slf4j-log4j12-1.6.1,
kafka-clients-0.8.2.0, scala-library-2.11.5, and kafka_2.11-0.8.2.0
(java app)
On Tue Feb 24 2015 at 2:23:40 PM Joe Stein wrote:
> are you including
> https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in
> yo
are you including
https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in
your project?
~ Joe Stein
- - - - - - - - - - - - - - - - -
http://www.stealth.ly
- - - - - - - - - - - - - - - - -
On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman wrote:
> Yea, when I try to set type
and kafka too :)
On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein wrote:
> are you including
> https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in
> your project?
>
> ~ Joe Stein
> - - - - - - - - - - - - - - - - -
>
> http://www.stealth.ly
> - - - - - - - - - - - - - - - - -
>
Does that count get frozen on a fixed number or any random number?
-Zakee
On Mon, Feb 23, 2015 at 9:48 AM, Stuart Reynolds
wrote:
> See SimpleConsumer. getOffsetsBefore
> and the getLastOffset example here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Yea, when I try to set type to async (exactly like the example) I get:
log4j:WARN No such property [producerType] in
kafka.producer.KafkaLog4jAppender.
On Tue Feb 24 2015 at 1:35:54 PM Joe Stein wrote:
> Here is sample log4j.properties
> https://github.com/stealthly/scala-kafka/blob/master/src/t
Here is sample log4j.properties
https://github.com/stealthly/scala-kafka/blob/master/src/test/resources/log4j.properties#L54-L67
I _almost_ have always pulled the class
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala
internal
to private repo a
Hello all,
Just wondering if those with a good amount of experience using Kafka in
production with many streams have converged on any sort of naming
convention. If so would you be willing to share?
Thanks in advance,
Taylor Gautier
Hi, I try to use the example
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example(and
BTW where is 0.8.2 example?).
It does not work, so I make a simplest possible snippet (still not working) out
of coce for findLeader(). Here it is below, and it produces the error:
Err
Jun,
Can you also take a look at the second problem I am having?
> > > I am trying to test how KafkaProducer behaves with topic replication
> > factor
> > > = 1
> > >
> > >1. One broker is offline BEFORE KafkaProducer starts sending
> messages.
> > >Because of the bug I mentioned, KafkaP
Guozhang,
Thank you for the reply, but could you be a little bit more detailed?
When will this new MM with exact mirroring be rolled out? I went to the
following URL to read up on Kafka-1650
https://issues.apache.org/jira/browse/KAFKA-1650
but that issue doesn't appear to be about mirroring, but
Ah, yes. You are right. That's a more obvious bug. Will fix that in
KAFKA-1984.
Thanks,
Jun
On Tue, Feb 24, 2015 at 8:37 AM, Xiaoyu Wang wrote:
> Hi Jun,
>
> If I understand it correctly. the highlighted line is for avoiding
> offline partitions, is it?
>
> for (int i = 0; i < numPartitions; i
Hi Jun,
If I understand it correctly. the highlighted line is for avoiding
offline partitions, is it?
for (int i = 0; i < numPartitions; i++) {
int partition = Utils.abs(counter.getAndIncrement()) % numPartitions;
if (partitions.get(partition).leader() != null) {
return partition;
Hi, Xiaoyu,
1. Could you explain a bit more what the bug is? The code does try to avoid
picking an unavailable partition. There does seem to be an issue when there
are more than one thread producing data to the same producer instance. This
is being tracked in KAFKA-1984. How many producing threads
Jun,
I am trying to test how KafkaProducer behaves with topic replication factor
= 1
1. One broker is offline BEFORE KafkaProducer starts sending messages.
Because of the bug I mentioned, KafkaProducer sends to the offline
partition and hangs forever.
2. One broker goes offline WHILE
Hi Gene,
Looks like you might be running into this
https://issues.apache.org/jira/browse/KAFKA-1758 .
-Harsha
On Tue, Feb 24, 2015, at 07:17 AM, Gene Robichaux wrote:
> About a week ago one of our brokers crashed with a hardware failure. When
> the server restarted the Kafka broker
About a week ago one of our brokers crashed with a hardware failure. When the
server restarted the Kafka broker would not start. The error is listed below. I
tried a couple of time to restart, but no success. The only think that work was
to physically remove the partitions for this broker. I was
48 matches
Mail list logo
