Hi, Thanks for clarification. The KIP you posted is a great piece of information 😊
Regards, Artur -----Original Message----- From: Kamal Chandraprakash [mailto:kamal.chandraprak...@gmail.com] Sent: Friday, May 10, 2019 3:50 PM To: users@kafka.apache.org Subject: Re: Kafka upgrade process details Sabre Email Notification: This email is from an EXTERNAL source. Please use caution when clicking on links or opening attachments from an unknown or suspicious sender.To report a suspected phishing email, browse to: help.sabre.com > Risk & Security > Report a Security Issue ______________________________________________________________________ Hi, In Kafka v2.1.0, the OffsetCommit Request/Response schema version is changed to v4 for the *__consumer_offsets* topic. If you upgrade Kafka to v2.1.0 & higher and changed the inter.broker.protocol version to 2.1, then you cannot revert back to older versions as it doesn't know how to read the *__consumer_offsets *topic with the latest offset message schema. Indeed, if you want to revert back to the older versions, you have to delete the* __consumer_offsets* topic. Note: Once that topic is deleted, all your consumer groups will start to read from the latest/earliest available message as per your `auto.offset.reset` configuration. Please correct me if I am wrong. https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D211-253A-2BRevise-2BExpiration-2BSemantics-2Bof-2BConsumer-2BGroup-2BOffsets&d=DwIFaQ&c=FXJfUb1oWgygD0uNz-ujnA&r=bVI5M04li2b_AW9E6XWAZb5H4NuzOzdzPeKTA_sjdMg&m=FoNa_w6hwhTP2oeFsNP8mygpNhrxoIPqzjssJmXtpsY&s=9nYYkBcasdjeivQfxJyoBXW005zgTFUa8BRAf598l2o&e= Note: The `__consumer_offsets` topic is read by the broker after each restart to load the last commited offset of the consumer groups. So, once you change the `inter.broker.protocol.version` to 2.1 the OffsetCommit schema version changes to v4. On Thu, May 9, 2019 at 4:21 PM Chylek, Artur <artur.chy...@sabre.com> wrote: > Thanks for prompt response. > I am not sure I understand correctly, but I am still confused why > switching inter.broker.protocol.version in the last step would make > the process irreversible. > If we agree that log conversion to a new format is applied when a new > value of log.message.format.version or broker's default one is applied > then according to documentation switching back to old version of > broker will make no harm. So either a broker is able to read a newer > versions of log and dismiss parts he doesn’t understand or the > conversion itself occurs only at certain combination of > inter.broker.protocol.version and log.message.format.version. Or maybe > I am completely wrong with my assumptions. > > Regards, > Artur > > -----Original Message----- > From: M. Manna [mailto:manme...@gmail.com] > Sent: Thursday, May 09, 2019 12:19 PM > To: Kafka Users <users@kafka.apache.org> > Subject: Re: Kafka upgrade process details > > Sabre Email Notification: This email is from an EXTERNAL source. > Please use caution when clicking on links or opening attachments from > an unknown or suspicious sender.To report a suspected phishing email, browse > to: > help.sabre.com > Risk & Security > Report a Security Issue > > ______________________________________________________________________ > Artur, > > > > The upgrade process is such that > > > > 1) You ensure that there is a hard-check on protocol version if not > exists > > already. As you have already mentioned above, in #3 - it's to ensure > that > > min verson for msg formats are being adhered to before upgrade. > > 2) broker protocol version is to ensure that when you do rolling > upgrade > > there is a minimum compatibility between inter-broker comms (if that > makes > > sense). > > > > API versions are there for a specific reason (to ensure compatbility > is > > maintained first before the protocol/msg formats are fixed). Also, it > > ensures that a regular upgrade (e.g. from bugfix->major release) > happens > > seamlessly via rolling restart. If you want to get techincal about it, > you > > can look at kafka Table load and config load in github codebase. > > > > Once you've initiated a rolling restart by setting > > inter.broker.protocl.version and log.messge.format.version there is no > way > > back - but i am happy to be proven wrong as I have only done rolling > > restart successfully (never needed to go back :) ) > > > > > > > > On Thu, 9 May 2019 at 09:54, Chylek, Artur <artur.chy...@sabre.com> wrote: > > > > > Hi, > > > I read the documentation about upgrading Kafka( > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_2 > 1_documentation.html-23upgrade-5F2-5F1-5F0&d=DwIBaQ&c=FXJfUb1oWgygD0uN > z-ujnA&r=bVI5M04li2b_AW9E6XWAZb5H4NuzOzdzPeKTA_sjdMg&m=3g5HRSTfPHijn_j > hhzYIB88jScofj68jB1AmrcMPBC0&s=qiYRoeea_IxCjRGx1Vi7ylwk2vuCiYGGoPhhGYm > iwos&e=) > but I have > > > questions that I believe the documentation doesn't cover. I am > > planning > to > > > upgrade Kafka from 2.0.0 to 2.1.0 and would like to make sure what > > to do > > > when something goes wrong - I am mostly interested in reverting back > > to > the > > > older version. I admit that I am confused about API versions, log > > message > > > versions and inter broker protocol versions. > > > > > > According to the documentation these are the steps I need to follow: > > > > > > 1. Set inter.broker.protocol.version to 2.0.0 > > > 2. I don't have currently log.message.format.version set to > > 2.0.0, so > > > according to documentation I don't need to specify it > > > 3. Upgrade broker code and restart it. This is I am not sure what > > > happens next. Since I don't have log.message.format.version set then > > > broker's default value will be used - I guess 2.1.1. Does it mean that: > > > * New messages that arrive from producers will be saved in a log > > > file with a new format? > > > * Old messages that already exist in log files will be converted > to > > > the new format once the broker starts? > > > * Does inter.broker.protocol.version has anything to do with > > > log.message.format.version? > > > I guess a) and b) are not true since according to the documentation > > up to > > > this point I can revert back to the older version of Kafka. Could > > you > > > confirm? > > > > > > 1. Change inter.broker.protocol.version to 2.1.1 and restart > > brokers - > > > after this is done reverting back is not possible. Could you explain > > what > > > is happening in Kafka so that reverting cannot be done? > > > > > > Regards, > > > Artur > > > > >
