Thanks for the contribution Matt! On Thu, Nov 5, 2015 at 4:21 PM, Matthew Bruce <mbr...@blackberry.com> wrote:
> Hi Grant, > > Creating a patch for this issue actually wasn't a whole lot of work. I've > uploaded one to the JIRA ticket. > > Thanks, > Matt > > -----Original Message----- > From: Matthew Bruce > Sent: Thursday, November 5, 2015 3:52 PM > To: 'dev@kafka.apache.org' <dev@kafka.apache.org> > Subject: RE: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk) > > https://issues.apache.org/jira/browse/KAFKA-2756 > > -----Original Message----- > From: Grant Henke [mailto:ghe...@cloudera.com] > Sent: Thursday, November 5, 2015 3:37 PM > To: dev@kafka.apache.org > Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk) > > Hi Matt, > > Great. Just wanted to be sure of the background. Are you interested in > opening a jira and contributing a patch? If not would you be interested in > testing a patch if I provided one? > > Thank you, > Grant > > On Thu, Nov 5, 2015 at 2:31 PM, Matthew Bruce <mbr...@blackberry.com> > wrote: > > > Hi Grant, > > > > Yes, I have read and followed the Upgrade documentation and I have set > > inter.broker.protocol.version=0.8.2.X and it does not resolve the > > issue - based on the part of the code it's occurring in, it uses the > 'latest' > > version for any API Key, and not the one specified by > > 'inter.broker.protocol.version'. > > > > Matt > > > > -----Original Message----- > > From: Grant Henke [mailto:ghe...@cloudera.com] > > Sent: Thursday, November 5, 2015 3:28 PM > > To: dev@kafka.apache.org > > Subject: Re: Replication Broken between Kafka 0.8.2.1 and 0.9 (trunk) > > > > Hi Matthew, > > > > I have not read into the details of your issues but have done similar > > "rolling" upgrade testing myself. The reason replication breaks is due > > to some wire protocol changes. > > > > Just checking some preliminary things before digging in > > > > - Have you followed the upgrade steps outlined here? > > - https://kafka.apache.org/090/documentation.html#upgrade > > - Does setting inter.broker.protocol.version=0.8.2.X resolve the > issue? > > - Note: you need to unset and restart again after all brokers are > > upgraded. > > > > In the future KIP-35 may help alleviate the manual step of setting the > > inter.broker.protocol.version. You can read more about KIP-35 and > > participate in the discussion/design here: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+ > > protocol+version > > > > Thanks, > > Grant > > > > > > On Thu, Nov 5, 2015 at 2:18 PM, Matthew Bruce <mbr...@blackberry.com> > > wrote: > > > > > Hello Kafka Devs, > > > > > > I've been testing the upgrade procedure between Kafka 0.8.2.1 and > > > Kafka > > > 0.9.0.0 and have been having Replication issues between the two > > > version, and I was wondering if anyone was aware of this issue (I > > > just searched and this seems to be related to KAFKA-2750 raised > yesterday ). > > > > > > I start with 3 brokers running 0.8.2.1 all that contain data (1 > > > topic with > > > 10 partitions), then I shut down one of the brokers, upgrade it to > > > 0.9.0 (making sure to set 'inter.broker.protocol.version=0.8.2.X' in > > > broker.properties). Once the Broker is started I see errors like > > > the > > > following: > > > > > > [2015-11-05 19:13:10,309] WARN [ReplicaFetcherThread-0-182050600], > > > Error in fetch > > > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858 > > <mailto: > > > kafka.server.ReplicaFetcherThread$FetchRequest@6cc18858>. Possible > > cause: > > > org.apache.kafka.common.protocol.types.SchemaException: Error > > > reading field > > > 'responses': Error reading field 'topic': > > > java.nio.BufferUnderflowException > > > (kafka.server.ReplicaFetcherThread) > > > And > > > [2015-11-03 16:55:15,178] WARN [ReplicaFetcherThread-1-182050600], > > > Error in fetch > > > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2 > > <mailto: > > > kafka.server.ReplicaFetcherThread$FetchRequest@224388b2>. Possible > > cause: > > > org.apache.kafka.common.protocol.types.SchemaException: Error > > > reading field > > > 'responses': Error reading field 'partition_responses': Error > > > reading field > > > 'record_set': java.lang.IllegalArgumentException > > > (kafka.server.ReplicaFetcherThread) > > > > > > > > > I've spent some time in the Kafka code, and packet > > > captures/wireshark trying to figure this out, and I believe there is > > > an issue in org.apache.kafka.clients.networkClient.java in the > > > handleCompletedReceives > > > function: > > > When extracting the response body, this function is using > > > ProtoUtils.currentResponseSchema instead of > > > ProtoUtils.ResponseSchema and specifying the API version required by > inter.broker.protocol.version. > > > Struct body = (Struct) > > > ProtoUtils.currentResponseSchema(apiKey).read(receive.payload()); > > > > > > This results in errors when the newer version of a Schema > > > (FETCH_RESPONSE_V1 instead of FETCH_RESPONSE_V0) is applied against > > > the fetch response returned by the 0.8.2.1 broker > > > > > > > > > Thanks, > > > Matthew Bruce > > > mbr...@blackberry.com<mailto:mbr...@blackberry.com> > > > > > > > > > > > > -- > > Grant Henke > > Software Engineer | Cloudera > > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > > > > > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
