Re: [I] error Caused by: java.util.concurrent.CancellationException: Task was cancelled. [rocketmq]
TickFairy commented on issue #7239: URL: https://github.com/apache/rocketmq/issues/7239#issuecomment-1864142651  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] In Controller mode, messages may be lost due to sharing the same cq offset [rocketmq]
GenerousMan opened a new issue, #7689: URL: https://github.com/apache/rocketmq/issues/7689 ### Before Creating the Bug Report - [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment During my testing of the controller mode, I encountered a scenario that could lead to message loss: 1. The leader controller and master broker are killed simultaneously. 2. Both are restarted at the same time, with the master broker completing recovery and immediately having messages written to it. ### RocketMQ version 5.1.4 ### JDK Version _No response_ ### Describe the Bug Since the master only recovers up to the position of the confirmOffset upon recovery, there may still be some messages that have not been synchronized by the slave. As a result, the data in the cqOffsetTable might be inaccurate. At this time, newly written messages could share the CQ offset with messages after the confirmOffset—this could lead to message loss. ### Steps to Reproduce using openchaos, major-kill. Killing brokers and controllers in the same time. ### What Did You Expect to See? no message lost. ### What Did You See Instead? message lost. ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset [rocketmq]
GenerousMan opened a new pull request, #7690: URL: https://github.com/apache/rocketmq/pull/7690 ### Which Issue(s) This PR Fixes Fixes #7689 Upon restart, it is necessary to compensate for messages after the confirmOffset by moving the cq offset backwards. It must be ensured that these messages remain unconsumable before the confirmOffset is reached. ### Brief Description ### How Did You Test This Change? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: 5292ffcd4d06b52bc1ab32cff70d16f30c2aea49 / juntao refactor the method of compensating for HA. Report URL: https://github.com/apache/rocketmq/actions/runs/7274651068 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: fe9700e1a21d419c08465069cff34bdb76831926 / juntao not modify the imports. Report URL: https://github.com/apache/rocketmq/actions/runs/7274704063 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "E2E test for pull request" failed!
The GitHub Actions job "E2E test for pull request" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: f0a3e933b91c1e5ec964e44073643b7bb8cc5e50 / EvanMi <43713766+eva...@users.noreply.github.com> [ISSUE #7684] Fix iterator.remove() bug (#7682) * bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating * add testcase - Co-authored-by: mipengcheng3 Report URL: https://github.com/apache/rocketmq/actions/runs/7274675159 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "E2E test for pull request" failed!
The GitHub Actions job "E2E test for pull request" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: f0a3e933b91c1e5ec964e44073643b7bb8cc5e50 / EvanMi <43713766+eva...@users.noreply.github.com> [ISSUE #7684] Fix iterator.remove() bug (#7682) * bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating * add testcase - Co-authored-by: mipengcheng3 Report URL: https://github.com/apache/rocketmq/actions/runs/7274726168 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Build and Run Tests by Maven" is working again!
The GitHub Actions job "Build and Run Tests by Maven" on rocketmq.git has succeeded. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: fe9700e1a21d419c08465069cff34bdb76831926 / juntao not modify the imports. Report URL: https://github.com/apache/rocketmq/actions/runs/7274704051 With regards, GitHub Actions via GitBox
Re: [PR] [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset [rocketmq]
RongtongJin commented on PR #7690: URL: https://github.com/apache/rocketmq/pull/7690#issuecomment-1864401574 LGTM～ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset [rocketmq]
RongtongJin commented on code in PR #7690:
URL: https://github.com/apache/rocketmq/pull/7690#discussion_r1432676214
##
store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java:
##
@@ -444,6 +444,7 @@ public void recoverOffsetTable(long minPhyOffset) {
String key = logic.getTopic() + "-" + logic.getQueueId();
long maxOffsetInQueue = logic.getMaxOffsetInQueue();
+log.info("Recover offset table, key: {}, maxOffsetInQueue:{}",
key, maxOffsetInQueue);
Review Comment:
这个日志去掉吧，万一队列太多了打印很多，而且也不是最终的位点
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(rocketmq) branch 4.9.x updated: Bump guava version from 31.1-jre to 32.0.1-jre (#7688)
This is an automated email from the ASF dual-hosted git repository. jinrongtong pushed a commit to branch 4.9.x in repository https://gitbox.apache.org/repos/asf/rocketmq.git The following commit(s) were added to refs/heads/4.9.x by this push: new a79b52991a Bump guava version from 31.1-jre to 32.0.1-jre (#7688) a79b52991a is described below commit a79b52991a49c3179aaadc34effd50f16cff6671 Author: rongtong AuthorDate: Wed Dec 20 20:52:11 2023 +0800 Bump guava version from 31.1-jre to 32.0.1-jre (#7688) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 5634d27174..b99cbc6057 100644 --- a/pom.xml +++ b/pom.xml @@ -108,7 +108,7 @@ 3.20.0-GA 4.2.2 3.4 -31.0.1-jre +32.0.1-jre 0.3.1-alpha 1.2.17 2.0
Re: [PR] [4.9.x] Bump guava version from 31.1-jre to 32.0.1-jre [rocketmq]
RongtongJin merged PR #7688: URL: https://github.com/apache/rocketmq/pull/7688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: 09e3370ffbb7a2cdc5e734674ba0d655768ddbc1 / juntao refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7276421552 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: 23a46a62a4df69ab187268dc726e9a3a7c5e59ed / juntao refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7276453564 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "E2E test for pull request" failed!
The GitHub Actions job "E2E test for pull request" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: f0a3e933b91c1e5ec964e44073643b7bb8cc5e50 / EvanMi <43713766+eva...@users.noreply.github.com> [ISSUE #7684] Fix iterator.remove() bug (#7682) * bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating * add testcase - Co-authored-by: mipengcheng3 Report URL: https://github.com/apache/rocketmq/actions/runs/7276455100 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "E2E test for pull request" failed!
The GitHub Actions job "E2E test for pull request" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: f0a3e933b91c1e5ec964e44073643b7bb8cc5e50 / EvanMi <43713766+eva...@users.noreply.github.com> [ISSUE #7684] Fix iterator.remove() bug (#7682) * bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating * add testcase - Co-authored-by: mipengcheng3 Report URL: https://github.com/apache/rocketmq/actions/runs/7276494531 With regards, GitHub Actions via GitBox
Re: [PR] [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset [rocketmq]
codecov-commenter commented on PR #7690: URL: https://github.com/apache/rocketmq/pull/7690#issuecomment-1864566470 ## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/7690?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report Attention: `11 lines` in your changes are missing coverage. Please review. > Comparison is base [(`f0a3e93`)](https://app.codecov.io/gh/apache/rocketmq/commit/f0a3e933b91c1e5ec964e44073643b7bb8cc5e50?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 43.15% compared to head [(`09e3370`)](https://app.codecov.io/gh/apache/rocketmq/pull/7690?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 43.17%. > :exclamation: Current head 09e3370 differs from pull request most recent head 23a46a6. Consider uploading reports for the commit 23a46a6 to get more accurate results | [Files](https://app.codecov.io/gh/apache/rocketmq/pull/7690?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines | |---|---|---| | [...apache/rocketmq/store/queue/ConsumeQueueStore.java](https://app.codecov.io/gh/apache/rocketmq/pull/7690?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3F1ZXVlL0NvbnN1bWVRdWV1ZVN0b3JlLmphdmE=) | 62.06% | [3 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/7690?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop#7690 +/- ## = + Coverage 43.15% 43.17% +0.01% - Complexity 9801 9806 +5 = Files 1165 1165 Lines 8448584489 +4 Branches 1096910969 = + Hits 3646336480 +17 + Misses 4349143471 -20 - Partials4531 4538 +7 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/rocketmq/pull/7690?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Maven" failed!
The GitHub Actions job "Build and Run Tests by Maven" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: 23a46a62a4df69ab187268dc726e9a3a7c5e59ed / juntao refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7276453562 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Build and Run Tests by Maven" failed!
The GitHub Actions job "Build and Run Tests by Maven" on rocketmq.git has failed. Run started by GitHub user GenerousMan (triggered by GenerousMan). Head commit for run: 09e3370ffbb7a2cdc5e734674ba0d655768ddbc1 / juntao refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7276421551 With regards, GitHub Actions via GitBox
Re: [I] Sending message in asynchronous mode by using python client [rocketmq]
github-actions[bot] commented on issue #5658: URL: https://github.com/apache/rocketmq/issues/5658#issuecomment-1865298736 This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [ISSUE #5710] optimize ServiceThread's waitForRunning logic [rocketmq]
github-actions[bot] commented on PR #5713: URL: https://github.com/apache/rocketmq/pull/5713#issuecomment-1865298700 This PR is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. If you wish not to mark it as stale, please leave a comment in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Timed MessageQueue Refresh Exception [rocketmq-clients]
github-actions[bot] commented on issue #603: URL: https://github.com/apache/rocketmq-clients/issues/603#issuecomment-1865306165 This issue was closed because it has been inactive for 3 days since being marked as stale. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Timed MessageQueue Refresh Exception [rocketmq-clients]
github-actions[bot] closed issue #603: [Bug] Timed MessageQueue Refresh Exception URL: https://github.com/apache/rocketmq-clients/issues/603 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] broker使用hostNetwork: true，但是注册到name-service上的IP还是pod CIDR段的IP [rocketmq-operator]
yylstudy opened a new issue, #202: URL: https://github.com/apache/rocketmq-operator/issues/202 1. Please describe the issue you observed: broker使用hostNetwork: true，但是注册到name-service上的IP还是pod CIDR段的IP broker配置如下  broker启动后ip也为节点IP  查看dashboard上的broker 仍然为pod的CIDR段IP broker日志也是CIDR段的IP  2. Please tell us about your environment: k8s：1.23.6 网络插件:cilium rocketmq-operator：采用helm部署，版本为0.3.0 broker镜像版本apacherocketmq/rocketmq-broker:4.5.0-alpine-operator-0.3.0 请问下为什么会出现这个问题呢，是不是和broker的镜像有关系，谢谢。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] 当MQTT设备数量特别多，消息特别频繁的时候，notify的RPC 因为是每台服务器都通知的，每台服务器接受notify QPS会特别高 [rocketmq-mqtt]
pingww closed issue #212: 当MQTT设备数量特别多，消息特别频繁的时候，notify的RPC 因为是每台服务器都通知的，每台服务器接受notify QPS会特别高 URL: https://github.com/apache/rocketmq-mqtt/issues/212 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] 一直写java.lang.NullPointerException: null日志，每天产生60G日志 [rocketmq-mqtt]
DongyuanPan commented on issue #221: URL: https://github.com/apache/rocketmq-mqtt/issues/221#issuecomment-1865411002 > > 消息无法消费，产生大量的retry消息，然后走到这儿又不被认为是mqtt消息（因为检查user property中没有PROPERTY_ORIGIN_MQTT_TOPIC属性）。 > > 然后，也没有PROPERTY_INNER_MULTI_DISPATCH这个属性，最终event中的topic为空。 另一线程处理event，跑到matchClients这里，TopicUtils.decode(topic)里面未做空判断，直接NPE。 > > 话说回来，消息为啥消费不了？跑的example里面的consumer，依然无法消费。 还有mqttx死活都连不上去，就算用example里面的签名方法处理之后，一样不行。 这个问题应该是RMQ没有开启LMQ这个特性导致的；用example里面的签名方法处理之后应该是需要Base64字符串才能使用在mqttx上。  只需要取前半部分的就可以了，不要输入toCharArray的。mqttx会对字符串再求一次 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] 安装配置好之后使用mqttx测试连接不上 [rocketmq-mqtt]
DongyuanPan commented on issue #216: URL: https://github.com/apache/rocketmq-mqtt/issues/216#issuecomment-1865415192 > > > 奥，那就是没办法用mqttx直接进行收发测试了 > > > > > > 可以的哈，你求出加签后的密码，直接写在mqttx 上就行 > > 右边三种加签后的密码都试了都不行，人麻了  老哥，你这里输入了char array，不要求char array，输入转换前的字符串。mqttx 会再求一次的 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user dingshuangxi888 (triggered by dingshuangxi888). Head commit for run: bc7b507be52ba6228493b07e5f13350ad7b033ee / ShuangxiDing Merge remote-tracking branch 'upstream/develop' into hotfix/fix_pop_born_time Report URL: https://github.com/apache/rocketmq/actions/runs/7283804708 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "E2E test for pull request" failed!
The GitHub Actions job "E2E test for pull request" on rocketmq.git has failed. Run started by GitHub user dingshuangxi888 (triggered by dingshuangxi888). Head commit for run: f0a3e933b91c1e5ec964e44073643b7bb8cc5e50 / EvanMi <43713766+eva...@users.noreply.github.com> [ISSUE #7684] Fix iterator.remove() bug (#7682) * bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating * add testcase - Co-authored-by: mipengcheng3 Report URL: https://github.com/apache/rocketmq/actions/runs/7283825144 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Coverage" failed!
The GitHub Actions job "Coverage" on rocketmq.git has failed. Run started by GitHub user dingshuangxi888 (triggered by dingshuangxi888). Head commit for run: bc7b507be52ba6228493b07e5f13350ad7b033ee / ShuangxiDing Merge remote-tracking branch 'upstream/develop' into hotfix/fix_pop_born_time Report URL: https://github.com/apache/rocketmq/actions/runs/7283804709 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Build and Run Tests by Maven" is working again!
The GitHub Actions job "Build and Run Tests by Maven" on rocketmq.git has succeeded. Run started by GitHub user drpmma (triggered by drpmma). Head commit for run: bd6b4b813bfdab8cead823919ebb36c74010f04b / zhouxiang Optimize rebalance Report URL: https://github.com/apache/rocketmq/actions/runs/7179027878 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user drpmma (triggered by drpmma). Head commit for run: 87c733aeeb74786136e629735b7e09e24aae3a56 / zhouxiang Add return value for sendHeartbeat related method Add sendHeartbeatToBroker Report URL: https://github.com/apache/rocketmq/actions/runs/7178868044 With regards, GitHub Actions via GitBox
(rocketmq) branch develop updated: [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset (#7690)
This is an automated email from the ASF dual-hosted git repository.
jinrongtong pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/rocketmq.git
The following commit(s) were added to refs/heads/develop by this push:
new 2c898c9b31 [ISSUE #7689] In Controller mode, messages may lost due to
sharing the same cq offset (#7690)
2c898c9b31 is described below
commit 2c898c9b31bf195174cf1e3a626a7c61f7576381
Author: Ji Juntao
AuthorDate: Thu Dec 21 14:41:51 2023 +0800
[ISSUE #7689] In Controller mode, messages may lost due to sharing the same
cq offset (#7690)
* fix the reput bug.
* add more logs.
* refactor the method of compensating for HA.
* not modify the imports.
* refactor the log.
* refactor the log.
---
.../rocketmq/store/queue/ConsumeQueueStore.java| 76 --
1 file changed, 40 insertions(+), 36 deletions(-)
diff --git
a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
index 616511b67f..cbe9b4f5ac 100644
--- a/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
+++ b/store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java
@@ -455,48 +455,52 @@ public class ConsumeQueueStore extends
AbstractConsumeQueueStore {
}
// Correct unSubmit consumeOffset
-if (messageStoreConfig.isDuplicationEnable()) {
-SelectMappedBufferResult lastBuffer = null;
-long startReadOffset =
messageStore.getCommitLog().getConfirmOffset() == -1 ? 0 :
messageStore.getCommitLog().getConfirmOffset();
-while ((lastBuffer =
messageStore.selectOneMessageByOffset(startReadOffset)) != null) {
-try {
-if (lastBuffer.getStartOffset() > startReadOffset) {
-startReadOffset = lastBuffer.getStartOffset();
-continue;
-}
+if (messageStoreConfig.isDuplicationEnable() ||
messageStore.getBrokerConfig().isEnableControllerMode()) {
+compensateForHA(cqOffsetTable);
+}
-ByteBuffer bb = lastBuffer.getByteBuffer();
-int magicCode = bb.getInt(bb.position() + 4);
-if (magicCode == CommitLog.BLANK_MAGIC_CODE) {
-startReadOffset += bb.getInt(bb.position());
-continue;
-} else if (magicCode != MessageDecoder.MESSAGE_MAGIC_CODE)
{
-throw new RuntimeException("Unknown magicCode: " +
magicCode);
-}
+this.setTopicQueueTable(cqOffsetTable);
+this.setBatchTopicQueueTable(bcqOffsetTable);
+}
+private void compensateForHA(ConcurrentMap cqOffsetTable) {
+SelectMappedBufferResult lastBuffer = null;
+long startReadOffset = messageStore.getCommitLog().getConfirmOffset()
== -1 ? 0 : messageStore.getCommitLog().getConfirmOffset();
+log.info("Correct unsubmitted offset...StartReadOffset = {}",
startReadOffset);
+while ((lastBuffer =
messageStore.selectOneMessageByOffset(startReadOffset)) != null) {
+try {
+if (lastBuffer.getStartOffset() > startReadOffset) {
+startReadOffset = lastBuffer.getStartOffset();
+continue;
+}
-lastBuffer.getByteBuffer().mark();
-DispatchRequest dispatchRequest =
messageStore.getCommitLog().checkMessageAndReturnSize(lastBuffer.getByteBuffer(),
true, true, true);
-if (!dispatchRequest.isSuccess())
-break;
-lastBuffer.getByteBuffer().reset();
-
-MessageExt msg =
MessageDecoder.decode(lastBuffer.getByteBuffer(), true, false, false, false,
true);
-if (msg == null)
-break;
-
-String key = msg.getTopic() + "-" + msg.getQueueId();
-cqOffsetTable.put(key, msg.getQueueOffset() + 1);
-startReadOffset += msg.getStoreSize();
-} finally {
-if (lastBuffer != null)
-lastBuffer.release();
+ByteBuffer bb = lastBuffer.getByteBuffer();
+int magicCode = bb.getInt(bb.position() + 4);
+if (magicCode == CommitLog.BLANK_MAGIC_CODE) {
+startReadOffset += bb.getInt(bb.position());
+continue;
+} else if (magicCode != MessageDecoder.MESSAGE_MAGIC_CODE) {
+throw new RuntimeException("Unknown magicCode: " +
magicCode);
}
+lastBuffer.getByteBuffer().mark();
+DispatchRequest dispatchRequest =
messageStore.getCommitLog().checkMessageAndReturnSize(la
Re: [PR] [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset [rocketmq]
RongtongJin merged PR #7690: URL: https://github.com/apache/rocketmq/pull/7690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@rocketmq.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GH] (rocketmq): Workflow run "Build and Run Tests by Bazel" failed!
The GitHub Actions job "Build and Run Tests by Bazel" on rocketmq.git has failed. Run started by GitHub user RongtongJin (triggered by RongtongJin). Head commit for run: 2c898c9b31bf195174cf1e3a626a7c61f7576381 / Ji Juntao [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset (#7690) * fix the reput bug. * add more logs. * refactor the method of compensating for HA. * not modify the imports. * refactor the log. * refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7284898070 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "PUSH-CI" failed!
The GitHub Actions job "PUSH-CI" on rocketmq.git has failed. Run started by GitHub user RongtongJin (triggered by RongtongJin). Head commit for run: 2c898c9b31bf195174cf1e3a626a7c61f7576381 / Ji Juntao [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset (#7690) * fix the reput bug. * add more logs. * refactor the method of compensating for HA. * not modify the imports. * refactor the log. * refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7284898063 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Coverage" is working again!
The GitHub Actions job "Coverage" on rocketmq.git has succeeded. Run started by GitHub user RongtongJin (triggered by RongtongJin). Head commit for run: 2c898c9b31bf195174cf1e3a626a7c61f7576381 / Ji Juntao [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset (#7690) * fix the reput bug. * add more logs. * refactor the method of compensating for HA. * not modify the imports. * refactor the log. * refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7284898065 With regards, GitHub Actions via GitBox
[GH] (rocketmq): Workflow run "Build and Run Tests by Maven" failed!
The GitHub Actions job "Build and Run Tests by Maven" on rocketmq.git has failed. Run started by GitHub user RongtongJin (triggered by RongtongJin). Head commit for run: 2c898c9b31bf195174cf1e3a626a7c61f7576381 / Ji Juntao [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset (#7690) * fix the reput bug. * add more logs. * refactor the method of compensating for HA. * not modify the imports. * refactor the log. * refactor the log. Report URL: https://github.com/apache/rocketmq/actions/runs/7284898066 With regards, GitHub Actions via GitBox
