I forgot to mention earlier, another way to detect this is to check the logs for the following lines:
[2022-08-25 17:44:05,308] ERROR [Broker id=0] Topic Id in memory: jKTRaM_TSNqoc1233I2aYOQ does not match the topic Id for partition myTopic-0 provided in the request: nI-J1234wQwGiylMfm8k13w. (state.change.logger) -- Divij Vaidya On Thu, Dec 22, 2022 at 11:32 AM Divij Vaidya <divijvaidy...@gmail.com> wrote: > Oh yes! You can run a script which checks that the topic IDs in Zk are > consistent with the ones that are available on partition.metadata. > > An example script would look something like this: > https://gist.github.com/divijvaidya/8a940a3d67bc8b215c95da54f5eff961 > > I will also add this script to the JIRA I linked earlier. Please add a +1 > to the JIRA since it would help measure the impact of this bug and hence, > will help drive prioritization of a fix in the community. > > -- > Divij Vaidya > > On Mon, Dec 19, 2022 at 1:23 PM Atul Kumar (atkumar3) > <atul.ku...@appdynamics.com.invalid> wrote: > >> Hi Divij, >> >> As this issue manifests after the Kafka controller restart, is there a >> way to detect this corrupted Kafka cluster before restarting the controller? >> >> >> >> >> >> Regards, >> Atul >> ________________________________ >> From: Atul Kumar (atkumar3) <atul.ku...@appdynamics.com> >> Sent: 16 December 2022 18:24 >> To: atul.ku...@appdynamics.com.invalid <atul.ku...@appdynamics.com.INVALID>; >> users@kafka.apache.org <users@kafka.apache.org> >> Subject: Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id >> in memory: <> does not match the topic Id for partition <> provided in the >> request: <>. >> >> Hi Divij, >> >> Thanks for the reply! >> >> We were using the Kafka-manager tool (with an older Kafka client) which >> was the root cause of this issue. >> >> >> >> >> Regards, >> Atul >> ________________________________ >> From: Divij Vaidya <divijvaidy...@gmail.com> >> Sent: 08 December 2022 17:55 >> To: atul.ku...@appdynamics.com.invalid <atul.ku...@appdynamics.com >> .invalid> >> Cc: users@kafka.apache.org <users@kafka.apache.org> >> Subject: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in >> memory: <> does not match the topic Id for partition <> provided in the >> request: <>. >> >> Hi Atul >> >> There is a known bug which has similar symptoms that you observed. See: >> https://issues.apache.org/jira/browse/KAFKA-14190 >> The bug manifests when you use a client < 2.8.0 with `--zookeeper` flag. >> It >> could be avoided by either upgrading the client or by using >> `--bootstrap-server` instead of `--zookeeper`. >> >> Does the above explanation match your Kafka setup? >> >> -- >> Divij Vaidya >> >> >> >> On Thu, Dec 8, 2022 at 1:16 PM Atul Kumar (atkumar3) >> <atul.ku...@appdynamics.com.invalid> wrote: >> >> > Hello everyone, >> > >> > We are using Kafka 2.8.1, and recently while rolling restart of one of >> our >> > clusters, we encountered the following issue on the Kafka cluster >> > >> > topic Id in memory: <> does not match the topic Id for partition <> >> > provided in the request: <>. >> > >> > All the produce requests to the impacted topic were failing. We were >> able >> > to fix the issue by deleting partition.metadata from each broker for the >> > impacted topic and doing a rolling restart of the Kafka cluster. We are >> > using the Kafka stream application. >> > >> > The same issue was reported in Jira: >> > https://issues.apache.org/jira/browse/KAFKA-12835 and is marked fixed >> > with version 2.8.1 but we can still see this issue. >> > >> > Any help would be highly appreciated on any fix for this issue. >> > >> > Thanks, >> > Atul Kumar >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > Regards, >> > Atul >> > >> >
