Joel,
Thanks for the comments. I updated the KIP page and added the canary
procedure.
Thanks,
Jiangjie (Becket) Qin
On Wed, Sep 30, 2015 at 6:26 PM, Joel Koshy wrote:
> The Phase 2 2.* sub-steps don't seem to be right. Can you look over
> that carefully? Also, "definitive" - you mean "absolut
The Phase 2 2.* sub-steps don't seem to be right. Can you look over
that carefully? Also, "definitive" - you mean "absolute" i.e., not
relative offsets right?
One more thing that may be worth mentioning is that it is technically
possible to canary the new version format on at most one broker (or
m
The BrokerMetadataRequest API could be used to propagate supported
message.formats from broker to client.
This is currently discussed in the KIP-35 thread.
/Magnus
2015-09-30 1:50 GMT+02:00 Jiangjie Qin :
> Hi Joel and other folks.
>
> I updated the KIP page with the two phase roll out, which
Hi Joel and other folks.
I updated the KIP page with the two phase roll out, which avoids the
conversion for majority of users.
To do that we need to add a message.format.version configuration to broker.
Other than that there is no interface change from the previous proposal.
Please let me know i
Hey Becket,
I do think we need the interim deployment phase, set
message.format.version and down-convert for producer request v2.
Down-conversion for v2 is no worse than what the broker is doing now.
I don't think we want a prolonged phase where we down-convert for
every v1 fetch - in fact I'm les
Hi Joel,
That is a valid concern. And that is actually why we had the
message.format.version before.
My original thinking was:
1. upgrade the broker to support both V1 and V2 for consumer/producer
request.
2. configure broker to store V1 on the disk. (message.format.version = 1)
3. upgrade the co
The upgrade plan works, but the potentially long interim phase of
skipping zero-copy for down-conversion could be problematic especially
for large deployments with large consumer fan-out. It is not only
going to be memory overhead but CPU as well - since you need to
decompress, write absolute offse
Thanks for the explanation, Jay.
Agreed. We have to keep the offset to be the offset of last inner message.
Jiangjie (Becket) Qin
On Mon, Sep 21, 2015 at 6:21 PM, Jay Kreps wrote:
> For (3) I don't think we can change the offset in the outer message from
> what it is today as it is relied upon
Hi Jun,
Thanks a lot for the comments. Please see the inline reply. I will update
the KIP page accordingly.
On Mon, Sep 21, 2015 at 5:17 PM, Jun Rao wrote:
> Jiangjie,
>
> Thanks for the writeup. A few comments below.
>
> 1. We will need to be a bit careful with fetch requests from the follower
For (3) I don't think we can change the offset in the outer message from
what it is today as it is relied upon in the search done in the log layer.
The reason it is the offset of the last message rather than the first is to
make the offset a least upper bound (i.e. the smallest offset >=
fetch_offs
Jiangjie,
Thanks for the writeup. A few comments below.
1. We will need to be a bit careful with fetch requests from the followers.
Basically, as we are doing a rolling upgrade of the brokers, the follower
can't start issuing V2 of the fetch request until the rest of the brokers
are ready to proc
Hi folks,
Thanks a lot for the feedback on KIP-31 - move to use relative offset. (Not
including timestamp and index discussion).
I updated the migration plan section as we discussed on KIP hangout. I
think it is the only concern raised so far. Please let me know if there are
further comments abou
Mayuresh,
I haven't created Jira for KIP-33 yet because it is still under discussion.
I will remove the Jira link.
Thanks,
Jiangjie (Becket) Qin
On Mon, Sep 14, 2015 at 8:15 PM, Mayuresh Gharat wrote:
> I suppose the jira link is different. It points to this jira :
> https://issues.apache.org
I suppose the jira link is different. It points to this jira :
https://issues.apache.org/jira/browse/KAFKA-1
Thanks,
Mayuresh
On Mon, Sep 14, 2015 at 5:13 PM, Jiangjie Qin
wrote:
> I just updated the KIP-33 to explain the indexing on CreateTime and
> LogAppendTime respectively. I also used so
I just updated the KIP-33 to explain the indexing on CreateTime and
LogAppendTime respectively. I also used some use case to compare the two
solutions.
Although this is for KIP-33, but it does give a some insights on whether it
makes sense to have a per message LogAppendTime.
https://cwiki.apache.o
I was trying to send last email before KIP hangout so maybe did not think
it through completely. By the way, the discussion is actually more related
to KIP-33, i.e. whether we should index on CreateTime or LogAppendTime.
(Although it seems all the discussion are still in this mailing thread...)
Thi
Hi Joel,
Good point about rebuilding index. I agree that having a per message
LogAppendTime might be necessary. About time adjustment, the solution
sounds promising, but it might be better to make it as a follow up of the
KIP because it seems a really rare use case.
I have another thought on how
Jay had mentioned the scenario of mirror-maker bootstrap which would
effectively reset the logAppendTimestamps for the bootstrapped data.
If we don't include logAppendTimestamps in each message there is a
similar scenario when rebuilding indexes during recovery. So it seems
it may be worth adding t
Ewen and Jay,
They way I see the LogAppendTime is another format of "offset". It serves
the following purpose:
1. Locate messages not only by position, but also by time. The difference
from offset is timestamp is not unique for all messags.
2. Allow broker to manage messages based on time, e.g. re
Re: MM preserving timestamps: Yes, this was how I interpreted the point in
the KIP and I only raised the issue because it restricts the usefulness of
timestamps anytime MM is involved. I agree it's not a deal breaker, but I
wanted to understand exact impact of the change. Some users seem to want to
Ah, I see, I think I misunderstood about MM, it was called out in the
proposal and I thought you were saying you'd retain the timestamp but I
think you're calling out that you're not. In that case you do have the
opposite problem, right? When you add mirroring for a topic all that data
will have a
> Don't you see all the same issues you see with client-defined timestamp's
> if you let mm control the timestamp as you were proposing? That means time
Actually I don't think that was in the proposal (or was it?). i.e., I
think it was always supposed to be controlled by the broker (and not
MM).
Hey Joel,
Yeah the clock issues would arrive when leadership changed or the time on
the machine changed.
Don't you see all the same issues you see with client-defined timestamp's
if you let mm control the timestamp as you were proposing? That means time
is no longer monotonic. A mixture of mirror
I just wanted to comment on a few points made earlier in this thread:
Concerns on clock skew: at least for the original proposal's scope
(which was more for honoring retention broker-side) this would only be
an issue when spanning leader movements right? i.e., leader migration
latency has to be mu
Hi Jay,
I just changed the KIP title and updated the KIP page.
And yes, we are working on a general version control proposal to make the
protocol migration like this more smooth. I will also create a KIP for that
soon.
Thanks,
Jiangjie (Becket) Qin
On Thu, Sep 10, 2015 at 2:21 PM, Jay Kreps
Great, can we change the name to something related to the change--"KIP-31:
Move to relative offsets in compressed message sets".
Also you had mentioned before you were going to expand on the mechanics of
handling these log format changes, right?
-Jay
On Thu, Sep 10, 2015 at 12:42 PM, Jiangjie Qi
Neha and Jay,
Thanks a lot for the feedback. Good point about splitting the discussion. I
have split the proposal to three KIPs and it does make each discussion more
clear:
KIP-31 - Message format change (Use relative offset)
KIP-32 - Add CreateTime and LogAppendTime to Kafka message
KIP-33 - Buil
Becket,
Nice write-up. Few thoughts -
I'd split up the discussion for simplicity. Note that you can always group
several of these in one patch to reduce the protocol changes people have to
deal with.This is just a suggestion, but I think the following split might
make it easier to tackle the chan
Hey Beckett,
I was proposing splitting up the KIP just for simplicity of discussion. You
can still implement them in one patch. I think otherwise it will be hard to
discuss/vote on them since if you like the offset proposal but not the time
proposal what do you do?
Introducing a second notion of
Hey Todd,
Yeah, totally agree the use case is important. I think there are
potentially also other uses that having access by time opens up too.
-Jay
On Sun, Sep 6, 2015 at 9:54 PM, Todd Palino wrote:
> So, with regards to why you want to search by timestamp, the biggest
> problem I've seen is
Jay,
Thanks for the comments. Yes, there are actually three proposals as you
pointed out.
We will have a separate proposal for (1) - version control mechanism. We
actually thought about whether we want to separate 2 and 3 internally
before creating the KIP. The reason we put 2 and 3 together is i
So, with regards to why you want to search by timestamp, the biggest
problem I've seen is with consumers who want to reset their timestamps to a
specific point, whether it is to replay a certain amount of messages, or to
rewind to before some problem state existed. This happens more often than
anyo
On Sun, Sep 6, 2015 at 4:57 PM, Jay Kreps wrote:
>
> 2. Nobody cares what time it is on the server.
>
This is a good way of summarizing the issue I was trying to get at, from an
app's perspective. Of the 3 stated goals of the KIP, #2 (lot retention) is
reasonably handled by a server-side timesta
One more comment, I think there are really three proposals here:
1. Get a mechanism and policy in place for record format upgrade (we
haven't done this so we don't really have the infra). This is kind of
implicit. I suspect we'll need to do this multiple times in the future so
we should make it eas
The magic byte is used to version message format so we'll need to make sure
that check is in place--I actually don't see it in the current consumer
code which I think is a bug we should fix for the next release (filed
KAFKA-2523). The purpose of that field is so there is a clear check on the
format
On Thu, Sep 3, 2015 at 11:45 PM, Jiangjie Qin
wrote:
> Hey Ewen,
>
> Thanks for the comments and they are really good questions. Please inline
> replies.
>
> On Thu, Sep 3, 2015 at 9:35 PM, Ewen Cheslack-Postava
> wrote:
>
> > A few questions:
> >
> > 1. If we update the producers to only suppor
Hey Ewen,
Thanks for the comments and they are really good questions. Please inline
replies.
On Thu, Sep 3, 2015 at 9:35 PM, Ewen Cheslack-Postava
wrote:
> A few questions:
>
> 1. If we update the producers to only support V1, doesn't that mean people
> get stuck on the current version of the p
Heh, I guess in addition to my wall of text of questions, I should also say
that I think this provides useful functionality and fixes issues that we've
seen a bunch of questions and complaints about, so I'm in favor of a fix
and this looks like a pretty good approach :)
It might also be useful to
A few questions:
1. If we update the producers to only support V1, doesn't that mean people
get stuck on the current version of the producer until they can be sure all
their consumers have been upgraded? Is that going to be a problem for
anyone, and does it potentially keep important fixes/enhance
Hi Guozhang,
I checked the code again. Actually CRC check probably won't fail. The newly
added timestamp field might be treated as keyLength instead, so we are
likely to receive an IllegalArgumentException when try to read the key.
I'll update the KIP.
Thanks,
Jiangjie (Becket) Qin
On Thu, Sep
Hi, Guozhang,
Thanks for reading the KIP. By "old consumer", I meant the
ZookeeperConsumerConnector in trunk now, i.e. without this bug fixed. If we
fix the ZookeeperConsumerConnector then it will throw exception complaining
about the unsupported version when it sees message format V1. What I was
Thanks for the write-up Jiangjie.
One comment about migration plan: "For old consumers, if they see the new
protocol the CRC check will fail"..
Do you mean this bug in the old consumer cannot be fixed in a
backward-compatible way?
Guozhang
On Thu, Sep 3, 2015 at 8:35 AM, Jiangjie Qin
wrote:
Hi,
We just created KIP-31 to propose a message format change in Kafka.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-31+-+Message+format+change+proposal
As a summary, the motivations are:
1. Avoid server side message re-compression
2. Honor time-based log roll and retention
3. Enable of
43 matches
Mail list logo
