Hi folks, During the implementation of the KIP, Guozhang raised another proposal on how to indicate the message timestamp type used by messages.
The difference between current and the new proposal only differs on messages that are a) compressed, and b) using LogAppendTime For compressed messages using LogAppendTime, the timestamps in the current proposal is as below: 1. When a producer produces the messages, it tries to set timestamp to -1 for inner messages if it knows LogAppendTime is used. 2. When a broker receives the messages, it will overwrite the timestamp of inner message to -1 if needed and write server time to the wrapper message. Broker will do re-compression if inner message timestamp is overwritten. 3. When a consumer receives the messages, it will see the inner message timestamp is -1 so the wrapper message timestamp is used. Implementation wise, this proposal requires the producer to set timestamp for inner messages correctly to avoid broker side re-compression. To do that, the short term solution is to let producer infer the timestamp type returned by broker in ProduceResponse and set correct timestamp afterwards. This means the first few batches will still need re-compression on the broker. The long term solution is to have producer get topic configuration during metadata update. The proposed modification is: 1. When a producer produces the messages, it always using create time. 2. When a broker receives the messages, it ignores the inner messages timestamp, but simply set a wrapper message timestamp type attribute bit to 1 and set the timestamp of the wrapper message to server time. (The broker will also set the timesatmp type attribute bit accordingly for non-compressed messages using LogAppendTime). 3. When a consumer receives the messages, it checks timestamp type attribute bit of wrapper message. If it is set to 1, the inner message's timestamp will be ignored and the wrapper message's timestamp will be used. This approach uses an extra attribute bit. The good thing of the modified protocol is consumers will be able to know the timestamp type. And re-compression on broker side is completely avoided no matter what value is sent by the producer. In this approach the inner messages will have wrong timestamps. The changed proposal has gone through some discussion during the KIP hangout as well as the discussion thread. Based on the current consensus, I have updated the KIP wiki with changed proposal and will implement it Please let us know if you have any concern over the changed proposal. Thanks, Jiangjie (Becket) Qin On Thu, Jan 21, 2016 at 10:08 AM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > I updated KIP-32 wiki with Anna's suggestion on ProduceResponse format > change. > > Because it looks everyone also has a +1 on ProduceResponse format change, I > take this update passed as well. > > KIP-32 passed with +6(binding). > > Thanks a lot for everyone's participation and all the comments. > > BTW, the patch for KIP-31 and KIP-32 has been submitted for review in PR > #764. > > Thanks, > > Jiangjie (Becket) Qin > > On Mon, Jan 11, 2016 at 10:48 PM, Becket Qin <becket....@gmail.com> wrote: > > > That makes sense to me, too. I will update the KIP to reflect this. > > Thanks, Anna. > > > > > On Jan 9, 2016, at 9:37 AM, Neha Narkhede <n...@confluent.io> wrote: > > > > > > Anna - Good suggestion. Sounds good to me as well > > > > > > On Fri, Jan 8, 2016 at 2:32 PM, Aditya Auradkar < > > > aaurad...@linkedin.com.invalid> wrote: > > > > > >> Anna, > > >> > > >> That sounds good to me as well. > > >> > > >> Aditya > > >> > > >>> On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira <g...@confluent.io> > > wrote: > > >>> > > >>> Sounds good to me too. Seems pretty easy to add and can be useful for > > >>> producers. > > >>> > > >>>> On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy <jjkosh...@gmail.com> > > wrote: > > >>>> > > >>>> Hi Anna, > > >>>> > > >>>> That sounds good to me - Becket/others any thoughts? > > >>>> > > >>>> Thanks, > > >>>> > > >>>> Joel > > >>>> > > >>>> On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner <a...@confluent.io> > > >> wrote: > > >>>> > > >>>>> Hi Becket and everyone, > > >>>>> > > >>>>> Could we please add the following functionality to this KIP. I > think > > >> it > > >>>>> would be very useful for the broker to return the timestamp in the > > >> ack > > >>> to > > >>>>> the producer (in response: timestamp per partition) and propagate > it > > >>> back > > >>>>> to client in RecordMetadata. This way, if timestamp type is > > >>>> LogAppendTime, > > >>>>> the producer client will see what timestamp was actually set -- and > > >> it > > >>>>> would match the timestamp that consumer sees. Also, returning the > > >>>> timestamp > > >>>>> in RecordMetadata is also useful for timestamp type = CreateTime, > > >> since > > >>>>> timestamp could be also set in KafkaProducer (if client set > timestamp > > >>> in > > >>>>> ProducerRecord to 0). > > >>>>> > > >>>>> Since this requires protocol change as well, it will be better to > > >>>> implement > > >>>>> this as part of KIP-32, rather than proposing a new KIP. > > >>>>> > > >>>>> Thanks, > > >>>>> Anna > > >>>>> > > >>>>> > > >>>>> On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy <jjkosh...@gmail.com> > > >>> wrote: > > >>>>> > > >>>>>> +1 from me > > >>>>>> > > >>>>>> Looking through this thread it seems there was some confusion on > > >> the > > >>>>>> migration discussion. This discussion in fact happened in the > > >> KIP-31 > > >>>>>> discuss thread, not so much in the KIP hangout. There is > > >> considerable > > >>>>>> overlap in discussions between KIP-3[1,2,3] so it makes sense to > > >>>>>> cross-reference all of these. > > >>>>>> > > >>>>>> I'm finding the Apache list archive a little cumbersome to use > > >> (e.g., > > >>>> the > > >>>>>> current link in KIP-31 points to the beginning of September > > >> archives) > > >>>> but > > >>>>>> the emails discussing migration were in October: > > >>> > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > > >>>>>> > > >>>>>> Markmail has a better interface but interestingly it has not > > >> indexed > > >>>> any > > >>>>> of > > >>>>>> the emails from August, September and early October ( > > >> > > > http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward > > >>>>>> ). > > >>>>>> Perhaps KIPs should include a permalink to the first message of > the > > >>>>> DISCUSS > > >>>>>> thread. E.g., > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > > >>>>>> > > >>>>>> Also, just to clarify Jay's comments on the content of KIPs: I > > >> think > > >>>>> having > > >>>>>> a pseudo-code spec/implementation guide is useful (especially for > > >>>>>> client-side KIPs). While the motivation should definitely capture > > >>> “why > > >>>> we > > >>>>>> are doing the KIP” it probably shouldn’t have to exhaustively > > >> capture > > >>>>> “why > > >>>>>> we are doing the KIP *this way*”. i.e., some of the discussions > are > > >>>>>> extremely nuanced and in this case spans multiple KIPs so links to > > >>>> other > > >>>>>> KIPs and the discuss threads and KIP hangout recordings are > perhaps > > >>>>>> sufficient to fill this gap - or maybe a new section that > > >> summarizes > > >>>> the > > >>>>>> discussions. > > >>>>>> > > >>>>>> Thanks, > > >>>>>> > > >>>>>> Joel > > >>>>>> > > >>>>>>> On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao <j...@confluent.io> > wrote: > > >>>>>>> > > >>>>>>> Hi, Jiangjie, > > >>>>>>> > > >>>>>>> 52. Replacing MessageSet with o.a.k.common.record will be ideal. > > >>>>>>> Unfortunately, we use MessageSet in SimpleConsumer, which is part > > >>> of > > >>>>> the > > >>>>>>> public api. Replacing MessageSet with o.a.k.common.record will be > > >>> an > > >>>>>>> incompatible api change. So, we probably should do this after we > > >>>>>> deprecate > > >>>>>>> SimpleConsumer. > > >>>>>>> > > >>>>>>> My original question is actually whether we just bump up magic > > >> byte > > >>>> in > > >>>>>>> Message once to incorporate both the offset and the timestamp > > >>> change. > > >>>>> It > > >>>>>>> seems that the answer is yes. Could you reflect that in the KIP? > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> > > >>>>>>> Jun > > >>>>>>> > > >>>>>>> > > >>>>>>> On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin <becket....@gmail.com > > >>> > > >>>>> wrote: > > >>>>>>> > > >>>>>>>> Thanks a lot for the careful reading, Jun. > > >>>>>>>> Please see inline replies. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>> On Jan 6, 2016, at 3:24 AM, Jun Rao <j...@confluent.io> > > >> wrote: > > >>>>>>>>> > > >>>>>>>>> Jiangjie, > > >>>>>>>>> > > >>>>>>>>> Thanks for the updated KIP. Overall, a +1 on the proposal. A > > >>> few > > >>>>>> minor > > >>>>>>>>> comments on the KIP. > > >>>>>>>>> > > >>>>>>>>> KIP-32: > > >>>>>>>>> 50. 6.c says "The log rolling has to depend on the earliest > > >>>>>> timestamp", > > >>>>>>>>> which is inconsistent with KIP-33. > > >>>>>>>> Corrected. > > >>>>>>>>> > > >>>>>>>>> 51. 8.b "If the time difference threshold is set to 0. The > > >>>>> timestamp > > >>>>>> in > > >>>>>>>> the > > >>>>>>>>> message is equivalent to LogAppendTime." If the time > > >> difference > > >>>> is > > >>>>> 0 > > >>>>>>> and > > >>>>>>>>> CreateTime is used, all messages will likely be rejected in > > >>> this > > >>>>>>>> proposal. > > >>>>>>>>> So, it's not equivalent to LogAppendTime. > > >>>>>>>> Corrected. > > >>>>>>>>> > > >>>>>>>>> 52. Could you include the new value of magic byte in message > > >>>> format > > >>>>>>>> change? > > >>>>>>>>> Also, do we have a single new message format that includes > > >> both > > >>>> the > > >>>>>>>> offset > > >>>>>>>>> change (relative offset for inner messages) and the addition > > >> of > > >>>>>>>> timestamp? > > >>>>>>>> I am actually thinking about this when I am writing the patch. > > >>>>>>>> The timestamp will be added to the o.a.k.common.record.Record > > >> and > > >>>>>>>> Kafka.message.Message. The offset change is in > > >>>>>>>> o.a.k.common.record.MemoryRecords and Kafka.message.MessageSet. > > >>> To > > >>>>>> avoid > > >>>>>>>> unnecessary changes, my current patch did not merge them > > >> together > > >>>> but > > >>>>>>>> simply make sure the version of Record(Message) and > > >>>>>>>> MemoryRecords(MessageSet) matches. > > >>>>>>>> > > >>>>>>>> Currently new clients uses classes in o.a.k.common.record, and > > >>> the > > >>>>>> broker > > >>>>>>>> and old clients uses classes in kafka.message. > > >>>>>>>> I am thinking about doing the followings: > > >>>>>>>> 1. Migrate broker to use o.a.k.common.record. > > >>>>>>>> 2. Add message format V0 and V1 to > > >>> o.a.k.common.protocol.Protocols. > > >>>>>>>> Ideally we should be able to define all the wire protocols in > > >>>>>>>> o.a.k.common.protocol.Protocol. So instead of having Record > > >> class > > >>>> to > > >>>>>>> parse > > >>>>>>>> byte arrays by itself, we can use Schema to parse the records. > > >>>>>>>> > > >>>>>>>> Would that be better? > > >>>>>>>>> > > >>>>>>>>> 53. Could you document the changes in ProducerRequest V2 and > > >>>>>>> FetchRequest > > >>>>>>>>> V2 (and the responses)? > > >>>>>>>> Done. > > >>>>>>>>> > > >>>>>>>>> 54. In migration phase 1, step 2, does internal ApiVersion > > >> mean > > >>>>>>>>> inter.broker.protocol.version? > > >>>>>>>> Yes. > > >>>>>>>>> > > >>>>>>>>> 55. In canary step 2.b, it says "It will only see > > >>>>>>>>> ProduceRequest/FetchRequest V1 from other brokers and > > >>> clietns.". > > >>>>> But > > >>>>>> in > > >>>>>>>>> phase 2, a broker will receive FetchRequest V2 from other > > >>>> brokers. > > >>>>>>>> I meant when we canary a broker in phase 2, there will be only > > >>> one > > >>>>>> broker > > >>>>>>>> entering phase 2, the other brokers will remain at phase 1. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> KIP-33: > > >>>>>>>>> 60. The KIP still says maintaining index at "at minute > > >>>> granularity" > > >>>>>>> even > > >>>>>>>>> though the index interval is configurable now. > > >>>>>>>> Corrected. > > >>>>>>>>> > > >>>>>>>>> 61. In this design, it's possible for a log segment to have > > >> an > > >>>>> empty > > >>>>>>> time > > >>>>>>>>> index. In the worse case, we may have to scan more than the > > >>>> active > > >>>>>>>> segment > > >>>>>>>>> to recover the latest timestamp. > > >>>>>>>> Corrected. > > >>>>>>>>> > > >>>>>>>>> Thanks, > > >>>>>>>>> > > >>>>>>>>> Jun > > >>>>>>>>> > > >>>>>>>>> On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < > > >>>>>>>>> aaurad...@linkedin.com.invalid> wrote: > > >>>>>>>>> > > >>>>>>>>>> Hey Becket/Anna - > > >>>>>>>>>> > > >>>>>>>>>> I have a few comments about the KIP. > > >>>>>>>>>> > > >>>>>>>>>> 1. (Minor) Can we rename the KIP? It's currently "Add > > >>> CreateTime > > >>>>> and > > >>>>>>>>>> LogAppendTime etc..". This is actually the title of the now > > >>>>> rejected > > >>>>>>>> Option > > >>>>>>>>>> 1. > > >>>>>>>>>> 2. (Minor) Can we rename the proposed option? It isn't > > >> really > > >>>>>> "option > > >>>>>>> 4" > > >>>>>>>>>> anymore. > > >>>>>>>>>> 3. I'm not clear on what exactly happens to compressed > > >>> messages > > >>>>>>>>>> when message.timestamp.type=LogAppendTime? Does every batch > > >>> get > > >>>>>>>>>> recompressed because the inner message gets rewritten with > > >> the > > >>>>>> server > > >>>>>>>>>> timestamp? Or does the message set on disk have the > > >> timestamp > > >>>> set > > >>>>> to > > >>>>>>>> -1. In > > >>>>>>>>>> that case, what do we use as timestamp for the message? > > >>>>>>>>>> 4. Do message.timestamp.type and > > >>> max.message.time.difference.ms > > >>>>>> need > > >>>>>>>> to be > > >>>>>>>>>> per-topic configs? It seems that this is really a client > > >>> config > > >>>>>> i.e. a > > >>>>>>>>>> client is the source of timestamps not a topic. It could > > >> also > > >>>> be a > > >>>>>>>>>> broker-level config to keep things simple. > > >>>>>>>>>> 5. The "Proposed Changes" section in the KIP tries to build > > >> a > > >>>>>>> time-based > > >>>>>>>>>> index for query but that is a separate proposal (KIP-33). > > >> Can > > >>> we > > >>>>>> more > > >>>>>>>>>> crisply identify what exactly will change when this KIP (and > > >>> 31) > > >>>>> is > > >>>>>>>>>> implemented? It isn't super clear to me at this point. > > >>>>>>>>>> > > >>>>>>>>>> Aside from that, I think the "Rejected Alternatives" section > > >>> of > > >>>>> the > > >>>>>>> KIP > > >>>>>>>> is > > >>>>>>>>>> excellent. Very good insight into what options were > > >> discussed > > >>>> and > > >>>>>>>> rejected. > > >>>>>>>>>> > > >>>>>>>>>> Aditya > > >>>>>>>>>> > > >>>>>>>>>>> On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin < > > >>>>> becket....@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks Guozhang, Gwen and Neha for the comments. Sorry for > > >>> late > > >>>>>> reply > > >>>>>>>>>>> because I only have occasional gmail access from my > > >> phone... > > >>>>>>>>>>> > > >>>>>>>>>>> I just updated the wiki for KIP-32. > > >>>>>>>>>>> > > >>>>>>>>>>> Gwen, > > >>>>>>>>>>> > > >>>>>>>>>>> Yes, the migration plan is what you described. > > >>>>>>>>>>> > > >>>>>>>>>>> I agree with your comments on the version. > > >>>>>>>>>>> I changed message.format.version to use the release > > >> version. > > >>>>>>>>>>> I did not change the internal version, we can discuss this > > >>> in a > > >>>>>>>> separate > > >>>>>>>>>>> thread. > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> > > >>>>>>>>>>> Jiangjie (Becket) Qin > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>>> On Dec 24, 2015, at 5:38 AM, Guozhang Wang < > > >>>> wangg...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> Also I agree with Gwen that such changes may worth a 0.10 > > >>>>> release > > >>>>>> or > > >>>>>>>>>> even > > >>>>>>>>>>>> 1.0, having it in 0.9.1 would be quite confusing to users. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Guozhang > > >>>>>>>>>>>> > > >>>>>>>>>>>>> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang < > > >>>>>> wangg...@gmail.com > > >>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Becket, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Please let us know once you have updated the wiki page > > >>>>> regarding > > >>>>>>> the > > >>>>>>>>>>>>> migration plan. Thanks! > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Guozhang > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira < > > >>>>>> g...@confluent.io > > >>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Thanks Becket, Anne and Neha for responding to my > > >> concern. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I had an offline discussion with Anne where she helped > > >> me > > >>>>>>> understand > > >>>>>>>>>>> the > > >>>>>>>>>>>>>> migration process. It isn't as bad as it looks in the > > >> KIP > > >>> :) > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> If I understand it correctly, the process (for users) > > >> will > > >>>> be: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 1. Prepare for upgrade (set format.version = 0, > > >>> ApiVersion = > > >>>>>>> 0.9.0) > > >>>>>>>>>>>>>> 2. Rolling upgrade of brokers > > >>>>>>>>>>>>>> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between > > >>>>> brokers > > >>>>>>>> will > > >>>>>>>>>>> use > > >>>>>>>>>>>>>> the new protocol > > >>>>>>>>>>>>>> 4. Start upgrading clients > > >>>>>>>>>>>>>> 5. When "enough" clients are upgraded, bump > > >> format.version > > >>>> to > > >>>>> 1 > > >>>>>>>>>>> (rolling). > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Becket, can you confirm? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Assuming this is the process, I'm +1 on the change. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Reminder to coders and reviewers that pull-requests with > > >>>>>>> user-facing > > >>>>>>>>>>>>>> changes should include documentation changes as well as > > >>> code > > >>>>>>>> changes. > > >>>>>>>>>>>>>> And a polite request to try to be helpful to users on > > >> when > > >>>> to > > >>>>>> use > > >>>>>>>>>>>>>> create-time and when to use log-append-time as > > >>>> configuration - > > >>>>>>> this > > >>>>>>>>>> is > > >>>>>>>>>>> not > > >>>>>>>>>>>>>> a trivial decision. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> A separate point I'm going to raise in a different > > >> thread > > >>> is > > >>>>>> that > > >>>>>>> we > > >>>>>>>>>>> need > > >>>>>>>>>>>>>> to streamline our versions a bit: > > >>>>>>>>>>>>>> 1. I'm afraid that 0.9.0-1 will be confusing to users > > >> who > > >>>> care > > >>>>>>> about > > >>>>>>>>>>>>>> released versions (what if we forget to change it before > > >>> the > > >>>>>>>> release? > > >>>>>>>>>>> Is > > >>>>>>>>>>>>>> it > > >>>>>>>>>>>>>> meaningful enough to someone running off trunk?), we > > >> need > > >>> to > > >>>>>> come > > >>>>>>> up > > >>>>>>>>>>> with > > >>>>>>>>>>>>>> something that will work for both LinkedIn and everyone > > >>>> else. > > >>>>>>>>>>>>>> 2. ApiVersion has real version numbers. > > >>>> message.format.version > > >>>>>> has > > >>>>>>>>>>>>>> sequence > > >>>>>>>>>>>>>> numbers. This makes us look pretty silly :) > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> My version concerns can be addressed separately and > > >> should > > >>>> not > > >>>>>>> hold > > >>>>>>>>>>> back > > >>>>>>>>>>>>>> this KIP. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Gwen > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin < > > >>>>>>> becket....@gmail.com> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Hi Anna, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks for initiating the voting process. I did not > > >> start > > >>>> the > > >>>>>>>> voting > > >>>>>>>>>>>>>>> process because there were still some ongoing > > >> discussion > > >>>> with > > >>>>>> Jun > > >>>>>>>>>>> about > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>> timestamp regarding compressed messages. That is why > > >> the > > >>>> wiki > > >>>>>>> page > > >>>>>>>>>>>>>> hasn't > > >>>>>>>>>>>>>>> reflected the latest conversation as Guozhang pointed > > >>> out. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Like Neha said I think we have reached general > > >> agreement > > >>> on > > >>>>>> this > > >>>>>>>>>> KIP. > > >>>>>>>>>>> So > > >>>>>>>>>>>>>>> it is probably fine to start the KIP voting. At least > > >> we > > >>>> draw > > >>>>>>> more > > >>>>>>>>>>>>>>> attention to the KIP even if there are some new > > >>> discussion > > >>>> to > > >>>>>>> bring > > >>>>>>>>>>> up. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Regarding the upgrade plan, given we decided to > > >> implement > > >>>>>> KIP-31 > > >>>>>>>> and > > >>>>>>>>>>>>>>> KIP-32 in the same patch to avoid change binary > > >> protocol > > >>>>> twice, > > >>>>>>> the > > >>>>>>>>>>>>>> upgrade > > >>>>>>>>>>>>>>> plan was mostly discussed on the discussion thread of > > >>>> KIP-31. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Since the voting has been initiated, I will update the > > >>> wiki > > >>>>>> with > > >>>>>>>>>>> latest > > >>>>>>>>>>>>>>> conversation to avoid further confusion. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> BTW, I actually have started coding work on KIP-31 and > > >>>> KIP-32 > > >>>>>> and > > >>>>>>>>>> will > > >>>>>>>>>>>>>>> focus on the patch before I return from vacation in mid > > >>> Jan > > >>>>>>> because > > >>>>>>>>>> I > > >>>>>>>>>>>>>> have > > >>>>>>>>>>>>>>> no LInkedIn VPN access in China anyway... > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Jiangjie > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On Dec 23, 2015, at 12:31 PM, Anna Povzner < > > >>>>> a...@confluent.io > > >>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Hi Gwen, > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> I just wanted to point out that I just started the > > >> vote. > > >>>>>> Becket > > >>>>>>>>>> wrote > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>> proposal and led the discussions. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> What I understood from reading the discussion thread, > > >>> the > > >>>>>>>> migration > > >>>>>>>>>>>>>> plan > > >>>>>>>>>>>>>>>> was discussed at the KIP meeting, and not much on the > > >>>>> mailing > > >>>>>>> list > > >>>>>>>>>>>>>>> itself. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> My question about the migration plan was same as > > >>> Guozhang > > >>>>>> wrote: > > >>>>>>>>>> The > > >>>>>>>>>>>>>> case > > >>>>>>>>>>>>>>>> when an upgraded broker receives an old producer > > >>> request. > > >>>>> The > > >>>>>>>>>>>>>> proposal is > > >>>>>>>>>>>>>>>> for the broker to fill in the timestamp field with the > > >>>>> current > > >>>>>>>> time > > >>>>>>>>>>> at > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>> broker. Cons: it goes against the definition of > > >>> CreateTime > > >>>>>> type > > >>>>>>> of > > >>>>>>>>>>> the > > >>>>>>>>>>>>>>>> timestamp (we are "over-writing" it at the broker). > > >>> Pros: > > >>>> It > > >>>>>>> looks > > >>>>>>>>>>>>>> like > > >>>>>>>>>>>>>>>> most of the use-cases would actually want that > > >> behavior, > > >>>>>> because > > >>>>>>>>>>>>>>> otherwise > > >>>>>>>>>>>>>>>> timestamp is useless and they will need to support an > > >>> old, > > >>>>>>>>>>>>>> pre-timestamp, > > >>>>>>>>>>>>>>>> behavior. E.g., if we modify log retention policy to > > >> use > > >>>> the > > >>>>>>>>>>>>>> timestamp, > > >>>>>>>>>>>>>>> we > > >>>>>>>>>>>>>>>> would need to support an old implementation (the one > > >>> that > > >>>>> does > > >>>>>>> not > > >>>>>>>>>>> use > > >>>>>>>>>>>>>>>> timestamps in the message). So I actually have a > > >>>> preference > > >>>>>> for > > >>>>>>>> the > > >>>>>>>>>>>>>>>> proposed approach. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>>> Anna > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede < > > >>>>>>>> n...@confluent.io > > >>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Hey Gwen, > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Migration plan wasn't really discussed a ton in the > > >>>>> previous > > >>>>>>>>>>> threads. > > >>>>>>>>>>>>>>> So it > > >>>>>>>>>>>>>>>>> will be great to dive deep and see if there are gaps > > >>>>> there. I > > >>>>>>> had > > >>>>>>>>>>>>>> some > > >>>>>>>>>>>>>>>>> questions, but the details listed on the KIP are > > >> great. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> It is complex, though the plan outlined in the wiki > > >>>>> assumes a > > >>>>>>>> zero > > >>>>>>>>>>>>>>> downtime > > >>>>>>>>>>>>>>>>> upgrade assuming that producers and consumers can't > > >> be > > >>>>>> upgraded > > >>>>>>>> in > > >>>>>>>>>>>>>>> tandem. > > >>>>>>>>>>>>>>>>> This is typical for companies that have a significant > > >>>> Kafka > > >>>>>>>>>>>>>> footprint, > > >>>>>>>>>>>>>>> like > > >>>>>>>>>>>>>>>>> LinkedIn. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>>>> Neha > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira < > > >>>>>>>> g...@confluent.io > > >>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Hi Anna, > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Thanks for the KIP, especially for the details on > > >> all > > >>>> the > > >>>>>>>>>>>>>> alternatives > > >>>>>>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>>> how we arrived at the proposal. Its really great! > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Can you point me at where the migration plan was > > >>>>> discussed? > > >>>>>> It > > >>>>>>>>>>> looks > > >>>>>>>>>>>>>>>>> overly > > >>>>>>>>>>>>>>>>>> complex and I have a bunch of questions, but if > > >> there > > >>>> was > > >>>>> a > > >>>>>>>>>>>>>> discussion, > > >>>>>>>>>>>>>>>>> I'd > > >>>>>>>>>>>>>>>>>> like to read up rather than repeating it :) > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Gwen > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner < > > >>>>>>>>>> a...@confluent.io > > >>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Hi, > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> I am opening the voting thread for KIP-32: Add > > >>>> CreateTime > > >>>>>> and > > >>>>>>>>>>>>>>>>>>> LogAppendTime to Kafka message. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> For reference, here's the KIP wiki: > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> And the mailing list threads: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> September: > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> October: > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> December: > > >> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>>>>>>>> Anna > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>>>> Neha > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> -- > > >>>>>>>>>>>>> -- Guozhang > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> -- > > >>>>>>>>>>>> -- Guozhang > > > > > > > > > > > > -- > > > Thanks, > > > Neha > > >
