absolute8511 opened a new issue, #9956: URL: https://github.com/apache/rocketmq/issues/9956
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment Any ### RocketMQ version 5.x ### JDK Version Any ### Describe the Bug ``` "PopReviveService_2" Id=121 cpuUsage=2.93% deltaTime=11ms time=603096ms RUNNABLE at com.alibaba.fastjson.parser.deserializer.FieldDeserializer.setValue(FieldDeserializer.java:167) at com.alibaba.fastjson.parser.deserializer.DefaultFieldDeserializer.parseField(DefaultFieldDeserializer.java:126) at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.parseField(JavaBeanDeserializer.java:1278) at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:893) at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.parseRest(JavaBeanDeserializer.java:1624) at com.alibaba.fastjson.parser.deserializer.FastjsonASMDeserializer_16_PopCheckPoint.deserialze(Unknown Source) at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:287) at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:705) at com.alibaba.fastjson.JSON.parseObject(JSON.java:394) at com.alibaba.fastjson.JSON.parseObject(JSON.java:298) at com.alibaba.fastjson.JSON.parseObject(JSON.java:588) at org.apache.rocketmq.broker.processor.PopReviveService.consumeReviveMessage(PopReviveService.java:366) at org.apache.rocketmq.broker.processor.PopReviveService.run(PopReviveService.java:654) at java.lang.Thread.run(Thread.java:748) "PopReviveService_0" Id=119 cpuUsage=2.82% deltaTime=11ms time=668229ms RUNNABLE at io.opentelemetry.api.internal.ImmutableKeyValuePairs.splitAndMerge(ImmutableKeyValuePairs.java:143) at io.opentelemetry.api.internal.ImmutableKeyValuePairs.splitAndMerge(ImmutableKeyValuePairs.java:143) at io.opentelemetry.api.internal.ImmutableKeyValuePairs.mergeSort(ImmutableKeyValuePairs.java:119) at io.opentelemetry.api.internal.ImmutableKeyValuePairs.sortAndFilter(ImmutableKeyValuePairs.java:110) at io.opentelemetry.api.internal.ImmutableKeyValuePairs.<init>(ImmutableKeyValuePairs.java:51) at io.opentelemetry.api.common.ArrayBackedAttributes.<init>(ArrayBackedAttributes.java:26) at io.opentelemetry.api.common.ArrayBackedAttributes.sortAndFilterToAttributes(ArrayBackedAttributes.java:60) at io.opentelemetry.api.common.ArrayBackedAttributesBuilder.build(ArrayBackedAttributesBuilder.java:31) at org.apache.rocketmq.broker.metrics.PopMetricsManager.incPopReviveGetCount(PopMetricsManager.java:304) at org.apache.rocketmq.broker.metrics.PopMetricsManager.incPopReviveCkGetCount(PopMetricsManager.java:293) at org.apache.rocketmq.broker.processor.PopReviveService.consumeReviveMessage(PopReviveService.java:371) at org.apache.rocketmq.broker.processor.PopReviveService.run(PopReviveService.java:654) at java.lang.Thread.run(Thread.java:748) ``` the PopReviveService keep trying consumeReviveMessage and mergeAndRevive, but the logs showed the newOffset did not changed for a long time. ``` 2025-12-23 14:51:04.631 DEBUG PopReviveService_2 - reviveQueueId=2, scan timeout 2025-12-23 14:51:04.834 DEBUG PopReviveService_5 - reviveQueueId=5, scan timeout 2025-12-23 14:51:04.937 DEBUG PopReviveService_6 - reviveQueueId=6, scan timeout 2025-12-23 14:51:05.031 DEBUG PopReviveService_3 - reviveQueueId=3, scan timeout 2025-12-23 14:51:05.031 DEBUG PopReviveService_1 - reviveQueueId=1, scan timeout 2025-12-23 14:51:05.031 DEBUG PopReviveService_0 - reviveQueueId=0, scan timeout 2025-12-23 14:51:05.031 DEBUG PopReviveService_7 - reviveQueueId=7, scan timeout 2025-12-23 14:51:05.031 DEBUG PopReviveService_4 - reviveQueueId=4, scan timeout 2025-12-23 14:51:05.731 DEBUG PopReviveService_2 - reviveQueueId=2, ck listSize=35742 2025-12-23 14:51:05.731 DEBUG PopReviveService_2 - reviveQueueId=2, 1st ck, startOffset=9131128, reviveOffset=1822555; last ck, startOffset=8831568, reviveOffset=1859338 2025-12-23 14:51:05.731 DEBUG PopReviveService_2 - reviveQueueId=2, revive finish,old offset is 1822554, new offset is 1822554, ckDelay=12499 2025-12-23 14:51:05.835 DEBUG PopReviveService_6 - reviveQueueId=6, ck listSize=33396 2025-12-23 14:51:05.835 DEBUG PopReviveService_6 - reviveQueueId=6, 1st ck, startOffset=2809176, reviveOffset=1821380; last ck, startOffset=8825699, reviveOffset=1855715 2025-12-23 14:51:05.835 DEBUG PopReviveService_6 - reviveQueueId=6, revive finish,old offset is 1821379, new offset is 1821379, ckDelay=12499 2025-12-23 14:51:05.953 INFO PopLongPollingService - pollingMapSize=1,tmpTotalSize=64,atomicTotalSize=64,diffSize=0 2025-12-23 14:51:06.031 DEBUG PopReviveService_4 - reviveQueueId=4, ck listSize=37939 2025-12-23 14:51:06.031 DEBUG PopReviveService_4 - reviveQueueId=4, 1st ck, startOffset=9131160, reviveOffset=1817744; last ck, startOffset=8836843, reviveOffset=1856511 2025-12-23 14:51:06.031 DEBUG PopReviveService_4 - reviveQueueId=4, revive finish,old offset is 1817743, new offset is 1817743, ckDelay=12499 ``` ### Steps to Reproduce 1. using 5x 2. use pop consumer, and change invisible for 10s for every messages 3. keep running for hours to make revive log long enough. ### What Did You Expect to See? the revive log consume offset should be updated ### What Did You See Instead? the revive log consume offset did not update for a long time. ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
