Thanks Jay -- some good ideas there.
I agree strongly that fewer, more solid, non-Java clients are better than many
shallow ones. Interesting that you feel we could do some more work in this
area, as I thought it was well served (even if they have proliferated).
One area I would like see docume
Hi Jiang,
One thing you can try is to set acks=-1, and set the
replica.lag.max.messages properly such that it will not kicks all follower
replicas immediately under your produce load. Then if one of the follower
replica is lagging and the other is not, this one will be dropped out of
ISR and when
Hi Guozhang,
Thank you for considering my suggestions. The security layer sounds like the
right facet to design for these sorts of capabilities. Have you considered a
chained ocap security model for the broker using hash tokens? This would
provide for per-partition read/write capabilities wi
Hi Jun,
I had some discussion with Guozhang about this; just realize it's not post to
the maillist, so I post them again.
Setting replica.lag.max.messages=10^12 is to ensure a follower will not be
removed from ISR because it's behind the leader, so larger value is safer.
10^12 here is used as
One option is to reduce the value of topic.metadata.refresh.interval.ms but
the concern is that may end up sending too many requests to the brokers
causing overhead. I'd suggest you use the new producer under
org.apache.kafka.clients.producer.KafkaProducer that does not have the
problem. It is fair
Sure, I will try to take a snapshot of the data distribution when it
happens next time.
Assuming the topic.metadata.refresh.interval.ms is the concern, how should
we "unstuck" our producers?
The important note from that the documentation seems to suggest that the
metadata refresh will only happen
Does this mean that we should set "auto.leader.rebalance.enable" to true?
I wouldn't recommend that just yet since it is not known to be very stable.
You mentioned that only 2 brokers ever took the traffic and the replication
factor is 2, makes me think that the producer stuck to 1 or few partitio
Basically my thought with getting a separate mailing list was to have
a place specifically to discuss issues around clients. I don't see a
lot of discussion about them on the main list. I thought perhaps this
was because people don't like to ask questions which are about
adjacent projects/code base
Another important part of eco-system could be around the adaptors of
getting data from other systems into Kafka and vice versa. So, for the
ingestion part, this can include things like getting data from mysql,
syslog, apache server log, etc. For the egress part, this can include
putting Kafka data
Yeah that was more or less what I was proposing. I posted my random
ideas in the other thread, let me know what you think.
-Jay
On Fri, Jul 18, 2014 at 3:20 PM, Philip O'Toole
wrote:
> That sounds great -- I do think documentation is always very important (I've
> had some ideas, but that's for
You probably don't need to set replica.lag.max.messages that high. You can
observe the max lag in jmx and set the value to be a bit higher than that.
Thanks,
Jun
On Fri, Jul 18, 2014 at 11:20 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731
LEX -) wrote:
> We tested ack=-1 with replica.lag.max.mes
That sounds great -- I do think documentation is always very important (I've
had some ideas, but that's for another time).
I would be very interested in more ideas around what you think is missing from
the eco-system. That way people get to contribute, but can deepen their
understanding in thei
A question was asked in another thread about what was an effective way
to contribute to the Kafka project for people who weren't very
enthusiastic about writing Java/Scala code.
I wanted to kind of advocate for an area I think is really important
and not as good as it could be--the client ecosyste
hi,
I have the following doubts regarding some kafka config parameters:
For example if I have a Throughput topic with replication factor 1 and a
single partition 0,then i will see the following files under
/tmp/kafka-logs/Throughput_0:
.index
.log
000
Hey Philip,
That is awesome! You actually don't have to write Java or Scala code
to contribute.
There is the usual thing that presentations (which you are already
doing), improving website docs, and general community participation
are all at least as valuable as new code.
However in addition to
We tested ack=-1 with replica.lag.max.messages=1. In this config no
message loss was found.
This is the only config we found to satisfy 1. no message loss and 2. service
keeps available when 1 single broker is down. Are there other configs that can
achieve the same, or stronger cons
Hi Tanguy,
Just updated the wording on this protocol wiki a bit:
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
Hope it is now clearer.
Guozhang
On Thu, Jul 17, 2014 at 11:48 PM, Tanguy tlrx wrote:
> Hi Jay, Hi Guozhang,
>
Hi Rob,
Sorry for the late reply.
If I understand your approach correctly, it requires all brokers to
remember the partition assignment of each consumer in order to decide
whether or not authorizing the fetch request, correct? If we are indeed
going to do such authorization for the security proje
Thanks Joe, I don't see any Out of memory error. Now I get exception when
Producer fetches metadata for a topic
Here is how I created the topic and run producer
pdeshmukh$ bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic secureTopic
Created top
Hi Pramod,
Can you increase KAFKA_HEAP_OPTS to lets say -Xmx1G in the
kafka-console-producer.sh to see if that gets you further along please in
your testing?
Thanks!
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://
Hi Josh,
Will this be due to this cause?
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
?
Guozhang
On Fri, Jul 18, 2014 at 12:45 AM, Josh J wrote:
> > What is the Kafka version you are using?
>
> 0.8.1.1.
>
> > And can you describ
Hello Raja/Joe,
When I turn on security, i still get out of memory error on producer. Is
this something to do with keys? Is there any other way I can connect to
broker?
*producer log*
[2014-07-17 15:38:14,186] ERROR OOME with size 352518400 (kafka.network.
BoundedByteBufferReceive)
java.lang.OutOf
> What is the Kafka version you are using?
0.8.1.1.
> And can you describe the steps to re-produce this issue?
I'm running three kafka brokers using Docker. I was using the docker IP
172.17.42.1 for the kafka host address. It looks like kafka doesn't work
nicely with the same local IP. I ended u
23 matches
Mail list logo
