Also I agree with Gwen that such changes may worth a 0.10 release or even 1.0, having it in 0.9.1 would be quite confusing to users.
Guozhang On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Becket, > > Please let us know once you have updated the wiki page regarding the > migration plan. Thanks! > > Guozhang > > On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira <g...@confluent.io> wrote: > >> Thanks Becket, Anne and Neha for responding to my concern. >> >> I had an offline discussion with Anne where she helped me understand the >> migration process. It isn't as bad as it looks in the KIP :) >> >> If I understand it correctly, the process (for users) will be: >> >> 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) >> 2. Rolling upgrade of brokers >> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will use >> the new protocol >> 4. Start upgrading clients >> 5. When "enough" clients are upgraded, bump format.version to 1 (rolling). >> >> Becket, can you confirm? >> >> Assuming this is the process, I'm +1 on the change. >> >> Reminder to coders and reviewers that pull-requests with user-facing >> changes should include documentation changes as well as code changes. >> And a polite request to try to be helpful to users on when to use >> create-time and when to use log-append-time as configuration - this is not >> a trivial decision. >> >> A separate point I'm going to raise in a different thread is that we need >> to streamline our versions a bit: >> 1. I'm afraid that 0.9.0-1 will be confusing to users who care about >> released versions (what if we forget to change it before the release? Is >> it >> meaningful enough to someone running off trunk?), we need to come up with >> something that will work for both LinkedIn and everyone else. >> 2. ApiVersion has real version numbers. message.format.version has >> sequence >> numbers. This makes us look pretty silly :) >> >> My version concerns can be addressed separately and should not hold back >> this KIP. >> >> Gwen >> >> >> >> On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin <becket....@gmail.com> >> wrote: >> >> > Hi Anna, >> > >> > Thanks for initiating the voting process. I did not start the voting >> > process because there were still some ongoing discussion with Jun about >> the >> > timestamp regarding compressed messages. That is why the wiki page >> hasn't >> > reflected the latest conversation as Guozhang pointed out. >> > >> > Like Neha said I think we have reached general agreement on this KIP. So >> > it is probably fine to start the KIP voting. At least we draw more >> > attention to the KIP even if there are some new discussion to bring up. >> > >> > Regarding the upgrade plan, given we decided to implement KIP-31 and >> > KIP-32 in the same patch to avoid change binary protocol twice, the >> upgrade >> > plan was mostly discussed on the discussion thread of KIP-31. >> > >> > Since the voting has been initiated, I will update the wiki with latest >> > conversation to avoid further confusion. >> > >> > BTW, I actually have started coding work on KIP-31 and KIP-32 and will >> > focus on the patch before I return from vacation in mid Jan because I >> have >> > no LInkedIn VPN access in China anyway... >> > >> > Thanks, >> > >> > Jiangjie >> > >> > > On Dec 23, 2015, at 12:31 PM, Anna Povzner <a...@confluent.io> wrote: >> > > >> > > Hi Gwen, >> > > >> > > I just wanted to point out that I just started the vote. Becket wrote >> the >> > > proposal and led the discussions. >> > > >> > > What I understood from reading the discussion thread, the migration >> plan >> > > was discussed at the KIP meeting, and not much on the mailing list >> > itself. >> > > >> > > My question about the migration plan was same as Guozhang wrote: The >> case >> > > when an upgraded broker receives an old producer request. The >> proposal is >> > > for the broker to fill in the timestamp field with the current time at >> > the >> > > broker. Cons: it goes against the definition of CreateTime type of the >> > > timestamp (we are "over-writing" it at the broker). Pros: It looks >> like >> > > most of the use-cases would actually want that behavior, because >> > otherwise >> > > timestamp is useless and they will need to support an old, >> pre-timestamp, >> > > behavior. E.g., if we modify log retention policy to use the >> timestamp, >> > we >> > > would need to support an old implementation (the one that does not use >> > > timestamps in the message). So I actually have a preference for the >> > > proposed approach. >> > > >> > > Thanks, >> > > Anna >> > > >> > >> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede <n...@confluent.io> >> > wrote: >> > >> >> > >> Hey Gwen, >> > >> >> > >> Migration plan wasn't really discussed a ton in the previous threads. >> > So it >> > >> will be great to dive deep and see if there are gaps there. I had >> some >> > >> questions, but the details listed on the KIP are great. >> > >> >> > >> It is complex, though the plan outlined in the wiki assumes a zero >> > downtime >> > >> upgrade assuming that producers and consumers can't be upgraded in >> > tandem. >> > >> This is typical for companies that have a significant Kafka >> footprint, >> > like >> > >> LinkedIn. >> > >> >> > >> Thanks, >> > >> Neha >> > >> >> > >>> On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira <g...@confluent.io> >> > wrote: >> > >>> >> > >>> Hi Anna, >> > >>> >> > >>> Thanks for the KIP, especially for the details on all the >> alternatives >> > >> and >> > >>> how we arrived at the proposal. Its really great! >> > >>> >> > >>> Can you point me at where the migration plan was discussed? It looks >> > >> overly >> > >>> complex and I have a bunch of questions, but if there was a >> discussion, >> > >> I'd >> > >>> like to read up rather than repeating it :) >> > >>> >> > >>> Gwen >> > >>> >> > >>>> On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner <a...@confluent.io> >> > >>> wrote: >> > >>> >> > >>>> Hi, >> > >>>> >> > >>>> I am opening the voting thread for KIP-32: Add CreateTime and >> > >>>> LogAppendTime to Kafka message. >> > >>>> >> > >>>> For reference, here's the KIP wiki: >> > >> >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message >> > >>>> >> > >>>> And the mailing list threads: >> > >>>> >> > >>>> September: >> > >> >> > >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E >> > >>>> >> > >>>> October: >> > >> >> > >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E >> > >>>> >> > >>>> December: >> > >> >> > >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E >> > >>>> >> > >>>> >> > >>>> Thanks >> > >>>> Anna >> > >> >> > >> >> > >> >> > >> -- >> > >> Thanks, >> > >> Neha >> > >> >> > >> > > > > -- > -- Guozhang > -- -- Guozhang
