[ https://issues.apache.org/jira/browse/KAFKA-18856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17929906#comment-17929906 ]
Chia-Ping Tsai commented on KAFKA-18856: ---------------------------------------- {quote}it is a confusing error message: "following could not be represented in metadata.version 3.0-IV1: feature flag(s): group.version, transaction.version." {quote} The error printed before "3.0-IV1" (from the empty feature image) seems helpful for users to understand what's happening. {code:java} [2025-02-24 19:19:36,482] ERROR Encountered metadata loading fault: Error loading metadata log record from offset 4 (org.apache.kafka.server.fault.LoggingFaultHandler) java.lang.IllegalArgumentException: No MetadataVersion with feature level 25 at org.apache.kafka.server.common.MetadataVersion.fromFeatureLevel(MetadataVersion.java:592) at org.apache.kafka.image.FeaturesDelta.replay(FeaturesDelta.java:69) at org.apache.kafka.image.MetadataDelta.replay(MetadataDelta.java:310) at org.apache.kafka.image.MetadataDelta.replay(MetadataDelta.java:221) at org.apache.kafka.image.loader.MetadataBatchLoader.replay(MetadataBatchLoader.java:258) at org.apache.kafka.image.loader.MetadataBatchLoader.loadBatch(MetadataBatchLoader.java:136) at org.apache.kafka.image.loader.MetadataLoader.lambda$handleCommit$1(MetadataLoader.java:362) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:132) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:215) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:186) {code} If we want to fix the confused "3.0-IV1", maybe we can backport a part of [https://github.com/apache/kafka/pull/18845] to 3.9 - it changes the MV of empty FeatureImage from "3.0-IV1" to "empty" and add suitable error image in accessing MV of FeatureImage {code:java} [2025-02-24 20:08:13,921] ERROR Encountered metadata loading fault: Unhandled fault in MetadataLoader#handleCommit. Last image offset was 34 (org.apache.kafka.server.fault.LoggingFaultHandler) java.lang.IllegalStateException: Unknown metadata version for FeaturesImage: finalized_eligible.leader.replicas.version(1), finalized_group.version(1), finalized_transaction.version(2), metadataVersion(Optional.empty), zkMigrationState() at org.apache.kafka.image.FeaturesImage.lambda$metadataVersionOrThrow$0(FeaturesImage.java:73) ~[kafka-metadata-4.0.0-SNAPSHOT.jar:?] at java.base/java.util.Optional.orElseThrow(Optional.java:403) ~[?:?] at org.apache.kafka.image.FeaturesImage.metadataVersionOrThrow(FeaturesImage.java:72) ~[kafka-metadata-4.0.0-SNAPSHOT.jar:?] at org.apache.kafka.image.loader.MetadataLoader.maybePublishMetadata(MetadataLoader.java:348) ~[kafka-metadata-4.0.0-SNAPSHOT.jar:?] at org.apache.kafka.image.loader.MetadataBatchLoader.applyDeltaAndUpdate(MetadataBatchLoader.java:275) ~[kafka-metadata-4.0.0-SNAPSHOT.jar:?] at org.apache.kafka.image.loader.MetadataBatchLoader.maybeFlushBatches(MetadataBatchLoader.java:211) ~[kafka-metadata-4.0.0-SNAPSHOT.jar:?] at org.apache.kafka.image.loader.MetadataLoader.lambda$handleCommit$1(MetadataLoader.java:364) ~[kafka-metadata-4.0.0-SNAPSHOT.jar:?] at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:132) ~[kafka-server-common-4.0.0-SNAPSHOT.jar:?] at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:215) ~[kafka-server-common-4.0.0-SNAPSHOT.jar:?] at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:186) ~[kafka-server-common-4.0.0-SNAPSHOT.jar:?] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] {code} > 3.9 broker unable to join dynamic quorum with 4.0 controller > ------------------------------------------------------------ > > Key: KAFKA-18856 > URL: https://issues.apache.org/jira/browse/KAFKA-18856 > Project: Kafka > Issue Type: Bug > Reporter: TengYao Chi > Assignee: TengYao Chi > Priority: Major > > Test step: > * format 4.0 controller /w controller.properties & standalone > ** bin/kafka-storage.sh format --standalone -t "dAtBluZORiuJEKV2uSWB2Q" -c > config/controller.properties > * start 4.0 controller > * init 3.9 broker /w kraft/broker.properties & no-initial-controllers > ** bin/kafka-storage.sh format -t "dAtBluZORiuJEKV2uSWB2Q" --config > config/kraft/broker.properties -N > * start 3.9 broker > error message: > {code:java} > [2025-02-24 23:13:21,271] ERROR Encountered metadata loading fault: Unhandled > error initializing new publishers > (org.apache.kafka.server.fault.LoggingFaultHandler) > org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been > lost because the following could not be represented in metadata.version > 3.0-IV1: feature flag(s): group.version, transaction.version > at > org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) > at > org.apache.kafka.image.FeaturesImage.handleFeatureLevelNotSupported(FeaturesImage.java:108) > at org.apache.kafka.image.FeaturesImage.write(FeaturesImage.java:88) > at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:153) > at > org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:296) > at > org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:267) > at > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:132) > at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:215) > at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:186) > at java.base/java.lang.Thread.run(Thread.java:1583) > ^C[2025-02-24 23:13:21,658] INFO Terminating process due to signal SIGINT > (org.apache.kafka.common.utils.LoggingSignalHandler) > [2025-02-24 23:13:21,660] INFO App info kafka.server for 2 unregistered > (org.apache.kafka.common.utils.AppInfoParser) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)