I recently upgraded some of our kafka clusters to use 0.8.1.1 (from 0.8.0).
It's all looking good so far. One thing I notice though (seems like a
good thing) is that the iostat utilization has gone way down after the
upgrade.
I'm not sure if I know exactly what could could be responsible for thi
Thanks so much for the detailed explanation Jun, it pretty much lines up with
my understanding.
In the case below, if we didn't particularly care about ordering and
re-produced m2, it would then become m5, and in many use cases this would be ok?
Perhaps a more direct question would be, once 0.8
They key point is that we have to keep all replicas consistent with each
other such that no matter which replica a consumer reads from, it always
reads the same data on a given offset. The following is an example.
Suppose we have 3 brokers A, B and C. Let's assume A is the leader and at
some poin
Thanks Jun
Can you explain a little more about what an uncommitted message means? The
messages are in the log so presumably? they have been acked at least by the the
local broker.
I guess I am hoping for some intuition around why 'replaying' the messages in
question would cause bad things.
T
Scott,
The reason for truncation is that the broker that comes back may have some
un-committed messages. Those messages shouldn't be exposed to the consumer
and therefore need to be removed from the log. So, on broker startup, we
first truncate the log to a safe point before which we know all mess
Jiang,
In case 3.1, when C restarts, the protocol is that C can only join ISR if
it has received all messages up to the current high watermark.
For example, let's assume that M is 10. Let's say A, B, C all have messages
at offset 100 and all those messages are committed (therefore high
watermark
Anyone getting this issue. Is it something related to environment or it is
the code. Producer works fine when run with secure=false (no security) mode.
pdeshmukh$ bin/kafka-console-producer.sh --broker-list localhost:9092:true
--topic secureTopic
[2014-07-18 13:12:29,817] WARN Property topic is
Hello Jiang,
If some of your brokers have frequent long GC caused soft failures you can
take a look at the operational wiki page to see if that helps tuning GC
settings.
https://cwiki.apache.org/confluence/display/KAFKA/Operations
Guozhang
On Tue, Jul 22, 2014 at 9:42 AM, Scott Clasen wrote:
Hi Jay,
For sure and a capability protocol adds even more. It's down to cost/benefit,
a nice place to be.
- Rob
> On Jul 22, 2014, at 10:46 AM, Jay Kreps wrote:
>
> Hey Robert,
>
> I think the issue is that you lose much of the simplification if you
> still need to heart beat and you need a
Hey Robert,
I think the issue is that you lose much of the simplification if you
still need to heart beat and you need a new commit notification
message. I think that would make this proposal more complex than the
current one.
-Jay
On Mon, Jul 21, 2014 at 6:44 PM, Robert Withers
wrote:
> Thanks
Ahh, yes that message loss case. I've wondered about that myself.
I guess I dont really understand why truncating messages is ever the right
thing to do. As kafka is an 'at least once' system. (send a message, get
no ack, it still might be on the topic) consumers that care will have to
de-dupe an
kafka-1028 addressed another unclean leader election problem. It prevents a
broker not in ISR from becoming a leader. The problem we are facing is that a
broker in ISR but without complete messages may become a leader. It's also a
kind of unclean leader election, but not the one that kafka-1028
> Interesting, I had missed that. Is it worth updating the documentation to
> make that more explicit, or do other people find it clear enough?
I agree, it's easy to miss, we missed that information too - we noticed it
only few days ago, while we've been using Kafka for weeks and we spent long
hou
13 matches
Mail list logo
